I spent my childhood speaking English at home, sometimes at school, and in places where English was spoken.
I spoke Spanish at the babysitter's house at my mother's request. She wanted me to learn it alongside English. I also spoke Spanish at school (it was a bilingual school), wherever Spanish was spoken (pretty common in California and rural Idaho alike), and sometimes at home.
I spoke German at family gatherings with my German immigrant grandparents, with the German immigrant neighbors around the corner, at the deli we frequented, with my Sunday School teacher, who was also a German immigrant, and sometimes at home with my dad.
To this day, my siblings and I speak a bizarre mix of all three languages together, leaving new acquaintances unfamiliar with how we were raised struggling to keep up.
And now my kids, thanks to one majoring in Asian studies and both having studied abroad in Japan, now throw Japanese into the mix.
All of us agree that English is the hardest to learn, and it's simply because it's a crazy mixed up language with rules that are constantly broken. At least my siblings and I don't even pretend our family pidgin has rules. Of my three primary languages, I mastered English the last because of all of what we called "jail words."
I tutored a Pakistani student in English, and I had to frequently tell him, "Yes, that's grammatically correct, but you'll likely never hear anyone say it that way."