For me, no it's not hard being a pantser. I tried plotting a few times and it never worked. But, this works for me. The best way for any writer to write is to work the way that works the best for them.
I've had people tell me that pantsers look down on plotters. I've never met a fellow pantser who does that. Frankly anyone who can sit down and pull together people and places and ideas and form a cohesive story, no matter by what method, will always have my respect.
Regarding needing a beginning and an end... being a pantser doesn't mean you don't have a beginning or an ending. I usually know the endings and/or goals of everything I write I just have to find my way there. Sometimes it changes, but not very often, actually, rarely. Sometimes all I have in the beginning is an ending and I have to figure out how to get to it. My two favorite words are "what if."
People seem to think that pantsers don't think about what they're writing at all. We do. We don't sit down for hours planning things and writing them down and making outlines and putting notes on bulletin boards, etc. Sometimes we make notes (otherwise you forget those great ideas you had in the shower) and sometimes you handwrite a scene while having coffee and you're not at your desk. but scenes and ideas flow outside of the time that we're actually writing. I choreograph (physically) all the fight scenes that I write. I may not use all of the choreography but as I write it forms the base and the outcome is already known to me.
I don't worry about arcs or character development or any of that. I just let it flow naturally, let its own rhythm find its way, but it doesn't mean that I'm not cognizant of what's going on or I never think about it when I'm not writing. I've been working like this for so long I don't know any other way to work.
If I start writing things down, like an outline etc. by the time I finish it the story is done for me and I no longer have any interest in working on it. Plotting, for me, kills the creative process. Again, that's how it works for me. I know this because I've tried plotting and outlining and also collaborated with other writers (which was VERY hard). So when I say this works for me I know it does because I have at least tried the other way.
Find the way of working that works best for you and then pursue it. There is no established right or wrong way to work no matter what anyone tells you. The "right" way is the way that works for you.