World building is such an important aspect of creating a magnificent piece of work. Just like with outlining your story, you need to create the world. This is particularly important for fictional tales. Think about it for just a moment. If you are creating a world for your characters to take their adventures in, don't you want to know what that world is like? Some books and movies have done extremely well simply because their world was original and exciting, not because their plot was the most riveting. The more thought out your world, the more people are going to be drawn to it and the more you will be able to use it to interact with your characters. Could you imagine trying to keep track of all the details of every single city and landscape on earth in your head? I don't know how well you did in geography, but even an expert couldn't keep every single detail stored in their memory banks. There is simply too much to store. That is why it is so important to keep track of your world. Think of cartographer for example. Do they just keep everything in their head? No, they map out all the things they have seen. True, much of it is for those who come after, not for themselves, but I would argue that even a seasoned cartographer, could not remember every single detail they have seen. So, let's just not tempt fate. The smallest detail that could be easily forgotten, could be something that would have changed the entire story. If you get it out in front of you, then you won't make mistakes in your writing or forget about great details in your world that you could have used, if they were in the forefront of your mind.
Now, don't misunderstand me. I'm not suggesting you takes years to build your world, no. look at the simple map I posted above. Something like that is a good start, if not a finished product. There may be many parts of the world that your characters never even see, which means you don't have to go too crazy on those areas. However, everywhere your characters go should be intently created and then described. In the end, don't worry too much and let that stop you from writing. Simply create and keep track of the details of your world somewhere other than in you head. It really doesn't even have too be too intensive in the beginning. You can build upon it as you go. Maybe you will write out descriptions, or use apps to build your world, or something else entirely. Whatever the case, I strongly suggest you build your world somewhere you can physically see it, or at the very least, read it.