I do that too and I enjoy drawing female bodies more because they look better to me and are also more interesting to draw. A female body has more interesting curves than a male body. I wouldn´t limit myself because of people questioning it, that doesn´t sound good to me