in my experience, most of times that I had troubles on Linux, which were not happening in windows, were related to difference (substancial) in scheduling for multithreaded applications... and some dangling pointers..
Most GUI are not directly programmed in X, nowadays: you use a toolkit. Sometimes the thread safety is not well suited or, often, task scheduling differences bring at light some architectural flaws.
That is what, in my experience, has stronger incidence on windows/Linux porting and maintenance for cross platform software products. And dependency on 3rd party libraries, of course.
Never had problem with X per se. Nor really OpenGL by itself, too.
I cannot imagine how painful is mac support.
And I'm fine keeping myself and my company ignorant on that.