You know...there might not be anything wrong with that? ^^; I mean, if it works for you, then it works; just because it's not the 'correct' way to do things doesn't mean you need to change it.
I also tend to animate completely from my imagination; I don't bother storyboarding or planning because I usually remember what I want. Right now I'm working on an animation that I first envisioned 4 years ago, and I can still play it back perfectly in my head. ^^ And between that 'footage' and whatever I can come up with on the fly (which usually turns out better anyway), I think I've got it covered.
Scripts are different though...I usually forget dialogue, and I always want the characters to say just the right thing, so if I come up with something good I have to write it down.
But planning out the plot itself is different...for example, here's my 'outline' for a very grand, sprawling fantasy saga:
Granted, this is a story that's lived in my head for a while, and I have written a lot about it (worldbuilding bibles and such) but my actual story-writing process is to just go with the flow and make the characters talk, while figuring out how to get to the next event in the outline (as you can see, I'm writing towards the 'festival' atm).
So it's still pretty loose; that blank space before the last event is to signal to me that I'm expected to come up with more big events to put in between there, whenever I get around to it. :9
TL;DR, my philosophy is 'plan whatever you need to', no matter how little it is. If you feel like you might forget it, or need to see it spelled out in front of you, go ahead and write it down. But if you don't, then don't. If you feel good about just being spontaneous, you can be that. It's totally allowed. ^^