There is no such consensus. Scientifically, “sex” is so much more complicated. Hormones, hormone receptors, gonads, genitalia, secondary sex characteristics, and reproductive gametes, to name just a few of the components of biological sex.

People who are biologically known to be inter-sex (never mind anything about gender or identity) outnumber those with natural red hair.

But all of this is relatively unimportant. Only a doctor would ever need to know most of these things about a person, and only a doctor or a potential sexual partner would need to know the others. There’s no circumstance in which anyone else needs to know any of these details about a person in order to decide how their interactions with that person should go. Especially not anyone who doesn’t have a close personal relationship with the person. Gender is how someone expresses themselves in society, and that’s the only thing that matters in most circumstances.

Do you know what gametes I produce, if any? Do you need to know what gametes I might produce in order to conduct this conversation?

I think I’m managing to convey my meaning to you, and receiving your meaning in return, without knowledge of what’s happening in your abdomen/pelvis.

Whatever concern you have about the terms “sex”, “binary”, and “immutable”, it isn’t scientific.