I don't think reading from right to left is annoying, in fact I read a lot of English translated manga in this format because the alternative is flipping the art which I don't feel comfortable with.
However, I think (my opinion etc etc) someone writing in English deliberately making their project read right to left in some attempt at authenticity is a bit pointless and banal. You're kind of saying that Japanese manga reads right to left for some sort of magical reason, rather than it just being an aspect of the Japanese script. It seems a little too far, like you're so enamoured with Manga style that you're assimilating as much as possible from its original culture, even the unnecessary aspects for the language you're writing in. You might as well start putting all your dialogue
t l
h i
i k
s e
for extra "authenticity".