There cannot and should not be balanced and open discussion on this issue. You cannot “polite” your way into finding a way to say that non-binary people are not legitimate.
This is paradox-of-tolerance stuff. Maintaining an inclusive community requires being intolerant of intolerance.
The existence of non-binary people does not hurt you.
Insisting on finding ways to deny that they exist hurts them.