If you are drawing to understand what you see not as a flat image but as a figure/object with three dimensional volume and weight, then eventually the understanding will 'click' and you will be able to see and manipulate new volumes and shapes with weight and depth either with realism or design in mind.
I will say that while you are not without any understanding, you seem to still have a ways to go before the moment when it all 'clicks'. Don't give up. Though it's practically a mantra of artistic advice, "keep practicing" is about the only thing you can really do, just make sure that while you're practicing, you're really trying to understand the physicality of what you're drawing as it's very easy to 'practice' without attention and diligence, sometimes losing the forest for the trees.
The only professional illustrator I ever met had been in the industry for over 20 years, and still used refrences. It's not a bad thing. However, the best way to be able to draw without a refrence, is to start simple, and build your way up, and have a good grasp of how things look from every angle prior to starting. So if, for example, you want to draw a person, you need to be intimately familar with how people look, thin and fat, old and young, and how they move, how far an elbow can bend, what knees do when someone kneels etc. You get better at drawing from your imagination, when you know how to draw all these things well off the top of your head. So if you are wanting to draw castles, gather pictures of castles, from every angle and every shape, and draw them. Get a lego castle set build it, and draw it from 3 differnt angles, then put it together differently, and draw that from 3 different angles....THEN try to draw a castle from your imagination. And that process will need to be repeated for anything you want to draw from the imagination, you need to be able to build up the 3D idea of what the object is in your mind, prior to putting it on paper.
By having an understanding on how things are built. You can copy images all you want, but if you don't have any understanding of what it is you are drawing, you might as well be copying abstract art - impossible to re-draw from another angle.
I think if you draw a certain object enough with references, you can draw it from memory. I tried drawing wolf paws some time ago but I couldn't get them to look right until I looked at some photos. So like someone else said, if you draw tons of birds using references from all different angles and perspectives, it would be easier to draw birds from your imagination. Also, fantasy creatures tend to be easier because they don't exist in real life and are a mosh of parts seen in real life.
By remembering what everything is supposed to look like and where it's supposed to go. Now... the world is really complicated, so obviously noticing things like proportions and remembering them is going to take a long time. And you're probably never going to remember how to draw everything from every angle under every lighting condition. But you can certainly get to the point where you can draw basic 3D shapes like cubes and cylinders in perspective and shade them from memory, and then you can use those as mnemonics to help you draw more complicated shapes and light them.
If you want to draw from your head more, learn to construct people and scenes from basic 3D shapes, draw them, then go back and use reference to correct where you screwed up and improve your drawing.
*Darqx phrased it pretty well in the reply below. If you continue drawing from references, you will learn techniques to apply to your drawing when you're not using references. I find that drawing without references also takes much practice; you probably won't start out as well as you want, but keep working at it and drawing from references in the same time, and you'll become more confident at drawing without references.
Another part of it is being very familiar with your subject, I think. If you draw birds from references frequently, for example, it will most likely be easier to start without reference by drawing birds, and the drawing will probably look better than if you had drawn something that you'd only explored a small amount before.
You take the things that you've learned drawing from reality (such as how to block something out, the general proportions of things, how to draw in perspective etc) and apply it to drawing from your imagination. It gets easier with practice.