Huh. My answer's no for the exact same reason you said yes. Well, I mean, I can't say my vocabulary's huge, but my brain works in the same way where I can't remember specific words right off the bat. I can't help but feel like I would use one so often even in casual conversations. It's because I know I'd use it as a crutch to my everyday speaking that I avoid using it.
It's actually somewhat fun?? for me to try and remember a word, it's probably more stimulating for me than frustrating. Either way, not being able to find a word... Yeah, that's not as fun. Then I just kinda cave in and finally use a thesaurus, accept defeat and all that