Actually, you absolutely can downvote. If you click upvote (blue) or down (red) the color change indicates that your vote has been saved and counted. The number displayed is only for the total upvotes, and hovering allows you to see who upvoted the comment (similar to recent likes and subscribes).
We don't show the total number of downvotes, nor the list of down-voters, and downvotes are not sutracted directly from the count shown (as that is just the total upvotes). We could change the count displayed to act as a total (upvote = +1 and dowvote = –1), but our intention was to focus on the positive feedback and promoting support within the community. Downvoted comments are pushed down, but we don't show a negative count or red numbers. On a similar note, way back when we experimented with a "dislike" button next to the like button... but instead of useful critique it led mostly to hurt feelings.
I did notice there's a small bug where if you unclick a downvote the color of the counter changes (as if you upvoted), but the count shown is still correct. We'll fix that.
We'll discuss tweaking which count we show (upvotes vs. total vs. two separate counts), but rest assured your votes (up or down) are definitely counted. Sometimes there are bugs, sometimes things might not work the way you'd prefer, but we're never intentionally lying to you or trying to destroy your magic! Sorry you feel that way.