Likely XO isn't for you.
The big pull of XO is that it gobbles up everything you own. In my case, I have over 80,000 samples in there (just the Battery 3 and 4 libraries - which are both different - add tens of thousands). The joy of it is that it makes it quick and stupidly fun to mix and match stuff, pulling stuff from here there and everywhere. Add in the beat designer, and its more akin to improvised cookery than a strict recipe.
That's how I see XO. It's not the place to forensically get something that's in your head. It's a place to go "something in this sort of vibe" by chucking a search term or criteria at it and then start throwing a lot of things together til you like it.
I love it.
It's amazing how fast and slick it is with 80,000 samples. All my 8 outs are mapped to individual tracks in Cubase so I can add any of my own effects very simply. It lacks for much detailed manipulation of the samples themselves, but with over 80,000 to choose from I don't feel that as much of a constraint. However if you are the type of person who must sculpt every element to be totally unique, then XO isn't the right tool.
I think we all have gone through such things at least once. 
