I've been using gender-neutral restrooms all my life.
Mostly it's been a matter of necessity: growing up as part of a large family in houses that rarely had more than one bathroom, we took turns using said bathroom. Come to think of it, that's been true of every home I've ever been in, no matter whose house it was: the bathrooms are for everyone, regardless of gender. There are times, of course, when it's clear that a bathroom is primarily used by family members of a particular gender (this one's got one bottle of shampoo and a can of shaving cream on the counter, that one is equipped with a full line of skin and hair care products), but even so, there's no question but that the toilet works and is available for anyone who needs it, no matter how that person is plumbed. It must also be noted that these bathrooms were all "one hole" facilities, with just a single toilet making it unlikely there would ever be two adults in the room at the same time.
To be fair, opponents to gender-neutrality in the pee-and-poo-place are generally concerned about public facilities, rather than private homes. As luck would have it, though, I experienced a public gender-neutral bathroom (with multiple stalls!) as early as 1979.
The place was my college dormitory, Lausanne Hall at Willamette University. Most of the dorms on campus were co-ed, but only Lausanne was door-to-door co-ed. Both the second and third floors were equipped with two sets of bathrooms and showers. The first floor, though, had only one set of facilities, most likely because only half the floor had bedrooms, with the other half dedicated to the dining hall and lounge. If the restroom was designated as gender-specific, this would require one gender or the other to climb to the second floor to use its facilities, which would unfairly inconvenience both the first and second floor residents of that gender, since the second floor restrooms had only half the stalls and showers of the one restroom on the first floor. It also meant that whichever gender was using the first floor restroom would be using a room designed for twice the number of users visiting it.
This was, of course, silly. A floor meeting was held, and by unanimous consensus, the first floor agreed to designate their restroom co-ed. For the sake of shower privacy, signs were created to let people know which gender might be showering at any given time, but apart than that, students of both genders moved freely through that room, using it whenever they had need of it. This seemed a sensible, respectful solution to a ridiculous, arbitrary problem.
Imagine my surprise, then, when I first read of a state legislature passing a law requiring that persons use only the restroom for which they are biologically plumbed.
Let's be clear about this: gender-specific restrooms are a relic of the Victorian era. The first law mandating the existence of separate male and female restrooms in public places was passed in Massachusetts in 1887, primarily to address the discomfort men felt at the growing presence of women in the workplace--and to protect frail women from the aggressive leers of brutish men. (That last clause, though ironic on my part, accurately describes gender stereotypes still prevalent as recently as the 1990s.)
Fast-forward to the present, and the increasingly public presence of transgender persons. Trans women (born with male parts, but self-identifying, and often presenting, as women) would rather use the women's room than enter a men's room where they will both be exposed to men's private parts (urinals do not typically have stalls around them) and be subjected to the stares of men upon seeing a person who appears to be a woman in a men's space. The corollary scenario is also disturbing: a person sporting facial hair and dressed as a man entering a women's restroom is certain to raise red flags. Given the ongoing existence of Victorian-era segregated facilities, it makes perfect sense that trans persons of either gender would rather use the restrooms associated with the gender of their identity.
Unfortunately, that makes bigots uncomfortable. And as we saw (and continue to see) in the marriage equality struggle, the primary concern of bigots is preserving systems that oppress the people they hate.
Sex bigots love the Victorian norms of frail women and strong men. There is no place in their world view for same-gender attraction, or for any gender identity other than that which conforms to the organs one was born with between one's thighs. Anything that differs from these norms is perversion, deviance, abomination, and must be forced back into the narrow criteria of the norms. Sex bigots have largely lost the marriage battle, though some are still pressing for the right to discriminate based on religious bigotry. Losing does not go done easily for a bigot of any stripe--witness the continued existence of the Ku Klux Klan--and rather than be chastened by defeats in court and public opinion, sex bigots have simply shifted their hatred to trans issues, particularly where they impact public restrooms.
In prosecuting this phase of the sex war, the bigots have created a straw man: the leering pervert who wants to sneak into a women's room by claiming to be a trans woman, just so he can glimpse some women using the toilet. This threat is patently absurd: as I wrote above, the whole point of trans individuals being able to use restrooms that correspond to their identity is that someone presenting as a man would be completely out of place in a women's restroom, would be spotted immediately, and (unless it was an awkward accident, as happens to us all from time to time) chased out of that space. Add to this the reality that women's rooms have stalls with doors, so that the only place one is likely to actually see genitals out in the open is a men's room--a scenario that doesn't come up in the arguments of male sex bigots because they just don't see women being as perverted as they themselves clearly are. (Remember Mike Huckabee wishing he could've used his "feminine side" when he was in high school to play peeping tom in the girls' locker room?)
The reality is that gender-specific restrooms are an inefficient relic of the nineteenth century. Adding a gender-neutral restroom to a building that already has both male and female restrooms just increases that inefficiency. It makes the most sense to just follow the example of Lausanne Hall's first floor--and of 1990s dramedy Allie McBeal--and have a single, large restroom with enough stalls for everyone.
This will, of course, make the sex bigots uncomfortable. But at some point, we're just going to have to get their minds out of the gutter, and start acting like civilized people who can tolerate the presence of both men and women in the same bathroom.