Many newer cubers can get confused and think that they have to solve a 4x4 or a 5x5 in a very similar manner to the 3x3, this causes lots of trouble as it is very difficult and impractical to try to solve the last layer of a 4x4 in a layer by layer method. Speedcubers use variants of a method called reduction where you reduce the puzzle into a 3x3

the first step - solve the centres

the second step - solve the edges (although two aren't solved here)

Once you have solved the centres and edges then the cube should look basically like a 3x3 but there are a few issues:

4x4s do not have fixed centres - on a 3x3 the same colour centres will always be opposite each other but on a 4x4 they can move, to solve a 4x4 you have to make sure the colour scheme is correct, this doesn't apply to 5x5 as while the outer centre pieces can move, there is one fixed centre in the middle of every side.

both 4x4s and 5x5s have OLL parity - this is where one edge is flipped, on 4x4 this isn't easily noticeable until the OLL stage in 3x3 where you are solving the last layer and requires a long algorithm to solve, on 5x5 this is noticeable as soon as all the other edges are solved, it is noticeable here because the middle edge piece will always be correct but the other two outer edge pieces will be flipped, the same algorithm still works though.

4x4 has PLL parity - this is where two edges are swapped, a case which is not possible on 3x3, the algorithm to solve this isn't especially long but it adds about a second to a good speedcubers solve time, this case doesn't happen on 5x5.

So ultimately one could argue that due to the extra parity case and the trouble of putting the centres in the right place, the 4x4 must be harder but ultimately the 5x5 takes longer to solve as there are more pieces to solve, the best speedcubers in the world average around 20-25 seconds on 4x4 and 35-40 seconds on 5x5. There is no reason not to learn to solve both of them, maybe then you can even progress to the 6x6 and 7x7 (and 8x8 and 9x9...)