I'm not using "totem poles." If I were using Totem poles I'd somewhat agree with you, but I'm not.
The word "totem" may have origins with particular groups of people, but so do most words. Actually the concept of totems can be found in a lot of different cultures across the world.
A totem is a person or thing that symbolically represents something else, I doubt anyone is going to get mad at me for using a word that fits. Sure I could call them symbolic objects, but that's literally what totems are.
Maybe if I was using actual "totem poles" then id get where you were coming from. But I feel pretty safe right now considering I'm using a word where it's definition fits. And on top of that, even if the word only had a singular origin, I'd still feel OK using it considering those people aren't the only people to place symbolic meaning on objects, they just happened to come up with a word for it. I'm not including a cultural practice, style, or heritage, I'm using vocabulary, and I'm not even changing the vocabulary. I'm using the word in a situation that fits it's definition.
Should I not call a specific style of braied hair corn rolls because that's a black hair style, even if the hair style is corn rolls?
Should I not call hard shelled layered Mexican food tacos because tacos have Latin origins, even if I'm talking about tacos?
It's not appropriation to use a word if it's definition fits, regardless of where the word originated.
Again if we were talking about Totem poles, it would be different, but were not.
I'm not portraying any actions or cultural ideas, I'm naming something. And even if I were portraying something that was unique to a single group of people, I'd still be doing it correctly because I'm using the word in the right context of its definition.
Ps: I'm only being this long winded because I know this particular line of subject discussion can lead to a lot of heated back and forth, so I feel a need to really cross all my Ts so I'm as clear as possible.