No, it's not annoying- as others have said, just make it clear in your description that that is the way your comic is formatted and there won't be a problem. Tapastic is not a good site for this format sadly, as they (some staff) feel like it's "Confusing" ect. Plus, if you look at the recommended comics by Tapa, I have never once seen a right-to-left format in there. I'd like to be proven wrong on that though! 
However, if you want to make authentic manga, then as you say, right-to-left is the correct way to panel it to be closer to the original Japanese format- so go for it!
That's the joy of self-publishing, do what YOU love, and what YOU prefer, because at the end of the day, it's you that needs to be happy with the work. ^W^
There are still some of us that run right-to-left on the site, myself included, so join us lol. XD