I don't know how popular it is these days, but The Left Hand of Darkness by Ursula K LeGuin is a classic sci-fi novel where a world is populated by a culture with distinctly nonbinary sexes, to the confusion of a terran visitor. It set a lot of the stage for fantasy since the 70s so it might be kinda dated now, but it's also cool to see the genre develop?
I've been hearing some rad things about Anciliary Justice these days too; I think it's that the main setting is culturally agender? I haven't read it myself, but fingers crossed it's fun like it sounds!