Why do you think it is so hard to make good fantasy world building?

I personally don’t know why but I always end up getting frustrated making fantasy world building. I always have ideas but they don’t make sense or there’s no way to make them work. What’s your thoughts on this?