If someone where to ask me on how to start their own website, I'd say stick with Wordpress.com if you don't want to pay anything. It's limited in how much space you can use and a few other limitations, like what @Shanny8 said. But depending on what you are wanting to put up there, you don't need a lot of space to start.
I also brought my domain name on Wordpress.com, but there are many other options out there. I would suggest just getting a .com domain, since the price is mostly stupid cheap (I think $20 on wordpress per year?).
Now, if you were to pay for your website hosting, I would suggest Blue Host.
If you have been using Wordpress.com for a while, using Blue Host is a good move since the two companies have a deal to work with each other. Their support is based in India I believe, but I have still had my problems solved with just a little patience. You'll probably be writing to them anyway, so accent and language barriers should be no problem, since your problems will be clearly stated in writing.
Money wise, going to outside hosting is far, far cheaper then just settling for plans you get on the Wordpress.com site. In most cases, the further ahead you pay, the cheaper it will become.
On the note of money, you get what you pay for, like in most cases. Yes, there are some services that will be more expensive then others and have worse services and others cheaper with better services.
I would say that instead of finding the cheapest hosting site, to find one with:
And I don't want to sugar coat things for you, or anyone else. Website building is a pain in the butt. There will be problems that seem impossible to fix, issues with making your website mobile phone friendly (if you choose a theme that doesn't already have that implemented (ala Comic Press)) and the feeling that you probably made a mistake in creating your website because it has been nothing but trouble.
But you have a community here to help. I would dare say, most of us have a website in one form or anther. I was too afraid of asking the Wordpress forums because they seemed really technical and that scared me away, but here it people will explain it to you in an easier to understand layout.