This was at the top of a google search for 'nested tuplet workaround cubase -dorico' so I checked it out.
(I was working on something where I grew weary of the workaround and of trying to replicate it manually from the score. Took out a trial of Dorico which does it, but trying to do it polyphonically in it was the single worst experience of my life with software. I spent around 4 hours in one bar.)
My findings since way back when include, you may create a couple of types of de facto nested tuplet in the piano roll of Cubendo, 'Quantize Panel': Grid [type], let's go with Triplet 8th note. Obviously 3 per quarter note. With this grid choose your tuplet, EG., '5-let' and save. This is 5 in the time of a triplet 8th note, so the secondary grid with this setting is 15 to a quarter note. Now you have both of these worlds in the sub-grid as well as the regular 16th to a quarter grid.
The other is choose a dotted value as your Grid type (useful most primarily for tuplets in a compound time sig, obv); 5-lets on top of a dotted 8th gets your secondary grid to be 20 in the time of 6 or 30 in the time of 9, etc. So de facto you have 5:4 in a 3, 6, etc time base.
So, you may do such as consider your 4/4 is 12/8 with that (+50%) adjustment tempo-wise and do a tuplet on top of a triplet or onto further dotted values. In effect a third layer.
Not ideal, of course as it means a little bit more thinking.
Also found Muse Score with a show-stopper bug doing this as well. The Logic way is a mite more elegant.
I'm glad I happened on this as I forget how that worked.