I don't find it annoying to read, I've read enough manga to be used to it--but I also don't think it should go right to left unless it was natively in a language that read that direction. That said, you can ignore our advice and keep doing you, it's not inherently bad and isn't going to impact you on Tapas. If you want to be published some day though, I would encourage you to go left to right on your next project. The industry won't take you seriously if you're paneling it backwards.
I know you want it to be authentic because it's what you know and what you love. But as a fellow author who is very inspired by manga, I need to tell you: You will never be an authentic mangaka in the way you want to be. You are not from Japan, and neither the Japanese nor the Americans will accept you as one (Even if you're not in the U.S., this applies). I have had friends who know I make a comic that could be considered american manga tell me to my face that they would never buy a manga with an author whose name didn't sound Japanese. You will get that bias from both manga fans and non-manga fans, which puts you at a disadvantage. It's why I always refer to my comic as a comic and not a manga, and why it reads left to right.
Ultimately the decision is up to you, but please think about it more if/when you get serious about selling it. It's something that needs a lot of consideration!