I've actually found that bottom-up planning for fantasy can be very rewarding. It gives a lot of creative freedom to the aesthetics or mood of the setting before attaching any big elements to it.
Much in the same way that a first draft gives you the pieces to create a well-rounded larger plot, bottom-up lets you create the pieces and then 'will' them into existence by piecing them together with big picture items that tie it all together.
I think the one take-away here, though, is that concept is extremely important. Having a basic idea of what your world will feel like is the best first step.