There are no 100% accurate methods for detecting it - the whole purpose of dogwhistling is to provide a cover of plausible deniability. The best you can do is recognize the patterns of common dogwhistles and also know the person expressing them very well, so you can determine "From what I know about this person, do they have a history of holding these sorts of beliefs and hiding them behind language, or do I know they might be entering into unfamiliar territory and adopted this language from elsewhere?"
If you educate yourself on common dogwhistles, what you can do is 1) adjust your own language to avoid common dogwhistles and 2) let others know about this type of language if they hear or even express it (in a way that's more "just to let you know" and less "how dare you?!" - yes, I understand getting this tone through is harder than it sounds) so they can hopefully do step 1.
The most foolproof argument I can think of to almost certainly state that one, specific person is dogwhistling is if they do what Lee Atwater did in my previous comment - confess in an interview that they specifically meant "hurt black people" and not "we should be more discerning with our taxes." And mind you, the interview was anonymous when it was first published in 1981 and Atwater's name wasn't revealed until the 90s. Atwater died in 1991, so he still had a whole decade to continue spreading dogwhistles with plausible deniability.
I understand your desire of "I just want to know if people are engaging in good faith discussion or not" but that's literally a problem even the smartest people in the world haven't solved. It's one of those problems in the world you just have to mitigate by hunkering down and really studying where it comes from and how it works. And it's a very depressing topic to get into and the fact that it's not 100% solved makes it even moreso, so it's perfectly understandable if you get exhausted and just abandon the attempt.
Also, regarding:
Tone of voice will offer you nothing when trying to determine someone's sincerity - just look at any successful salesman.