While that's definitely a notable downside, I think the upsides outweigh it.
For one, being able to see upvotes & downvotes seems to have made a lot of people a bit more thoughtful with handing them out. This obviously isn't the case for everyone — there's still a good bit of downvoting people for disagreeing with the hivemind — but I and others have observed that downvote quality is a lot better here on kbin.social, and I think that vote visibility is a big part of that.
It's also just transparency on kbin.social's part. If votes federate, anyone can set up an instance to view your votes or just go to one that shows them. Someone could literally make a website listing downvotes throughout the fediverse, and there's nothing stopping them. Kbin.social is being transparent about the fact that votes on the fediverse can be accessed by the public, and I have no issue with that.