I don't see a problem with it unless your one if the ******s who uses it to grief.
I xray, always have, and it's not a secret. With it you can focus on finishing that building or structure, instead of searching for diamond for hours on end. But when you abuse it, you deserve a ban.
Fly can be disable with a few server commands being disabled.