Fast TANH aproximation

[Aleksey Vaneev]

Here's "ultra" variant of the tanh appoximation, public domain as well.

Code: Select all

inline double vox_fasttanh_ultra( const double x )
{
	const double ax = fabs( x );
	const double x2 = x * x;
	const double z = x * ( 0.773062670268356 + ax +
		( 0.757118539838817 + 0.0139332362248817 * x2 * x2 ) *
		x2 * ax );

	return( z / ( 0.795956503022967 + fabs( z )));
}

It's in average about 2.5 times more precise than the previous function, only less precise in the X range -0.0083..0.0083, peak relative error against tanh() is 2.783%. The timing is about the same.

It seems that branching on the modern CPU is still quite expensive (as it was always the case in the past), but the math is cheap - you can add a multiplication without measurable performance loss.

[FastTriggerFish]

Didn't know about Eureqa, looks great !

[mystran]

It seems that branching on the modern CPU is still quite expensive (as it was always the case in the past), but the math is cheap - you can add a multiplication without measurable performance loss.

Branching is weird (like most things on modern CPUs). Sometimes you can get away with branches for practically zero effect, while other times you'll take a huge hit... and sometimes it appears to depend on other stuff that's nearby, sometimes in an apparently illogical way.

Among the silly things, I've even found cases where unrolling really short loops can end up with significant performance hits... but then you might change the total loop count a bit (shorter or larger), and suddenly unrolling is faster again. None of it really makes any sense most of the time.

In general my personal experience is that the only workable way to optimize for modern CPUs is to try a large number of variations and profile which ever happens to run the fastest (and micro-profiling parts doesn't really work because of the "depends on what's nearby" thing). Unfortunately even that doesn't always work, because the next system nearby might behave in a different way.

[bootsie]

http://varietyofsound.wordpress.com/201 ... -fraction/

[Aleksey Vaneev]

bootsie, I've used continued fraction approach before, but its not much more efficient than the latest code I've posted above, and while this latest code guarantees no "runaway" at high values (always converges to 1.0), in your continued fractions example you are basically dividing x^7 by x^6 which guarantees runaway if X is high - usual situation in a guitar overdrive.

[FastTriggerFish]

I should try it myself when I get some spare time, but isn't anything that involves the absolute value going to produce a lot more aliasing than tanh since the first derivative isn't defined ?
Plotting the numerical derivative of the above proposed function that uses abs(x) ,it looks 'butt' ugly around zero ( appropriate choice of words here, try it for yourself ) which rings aliasing alarm bells to me.

Another way of saying this could be that the L2 or Linfinity norms ( i.e only looking at the distance of tanh to the function at each point ) might not be the right ones to choose functions that are close to tanh, you might want to choose a norm that also involves the distance of the first derivative ( or more ) too.
Otherwise, if it's much faster but you have to oversample more to make it sound good it may be pointless.

[Aleksey Vaneev]

FastTriggerFish, it all boils down to the deviation from the tanh() function, which is very small. I would say, fabs() and division compensate for each other The function is smooth. Even if you can't prove it analytically, you can always calculate the value range around 0 (e.g. in the X range -1e-9 to 1e9) and see that derivative there is OK.

fabs(), clip() and all other similar functions are pretty much out of the reach of analytical math since these functions can be represented analytically only by a polynomial with infinite number of members, and if you use fewer members than infinite, you get an error. They are best evaluated using computational math.

[mystran]

Another way of saying this could be that the L2 or Linfinity norms ( i.e only looking at the distance of tanh to the function at each point ) might not be the right ones to choose functions that are close to tanh, you might want to choose a norm that also involves the distance of the first derivative ( or more ) too.

Min-maxing is great if you care about the error terms in the function itself. For wave-shaping though, I'd usually care more about the resulting spectrum... as long as the time-domain stuff isn't too badly off.

Anyway, you can get decent (and pretty cheap) approximations by doing a Pade-approx on the truncated Taylor around zero, either with the tanh() directly, or some related function (eg tanh(sqrt(x))/sqrt(x) works surprisingly, but there are other possible choices). You still need some magic (eg abs()-tricks or clipping) to get finite limits at infinity if you need that.

Anyway, personally I've given up the search for the "ideal tanh()" approximation, because something that works in one case isn't necessarily the best for another case. I think I probably have at least half a dozen different approximations for different purposes. YMMV.

[FastTriggerFish]

Yeah actually you can show that the first derivative *is* continuous at zero ( by extension since the right and left limit are the same ) - but the second derivative isn't and has a big discontinuity.
Maybe it doesn't matter at all sound wise and I'm overthinking it, I just tend to think that the smoother the function the less aliasing.

[Aleksey Vaneev]

I think the relative error graph of the approximation tells everything about error in derivative.

[FastTriggerFish]

Min-maxing is great if you care about the error terms in the function itself. For wave-shaping though, I'd usually care more about the resulting spectrum... as long as the time-domain stuff isn't too badly off.

You're right, it makes more sense to optimise directly in the spectrum domain since that's what the ear cares about.

[Aleksey Vaneev]

FastTriggerFish, if the function is smooth, practically linear, no way its second derivative will have any problems, it can be discontinuous, turning into infinite, only in analytical math.

[FastTriggerFish]

I think the relative error graph of the approximation tells everything about error in derivative.

In the limit yes, but not as long as your residual error is not zero - at least it's not as easy to see the issues by just looking at that distance.
For instance the tanh should produce close to zero aliasing if the input is small, but the approximation above will definitely produce some.
Maybe it's inaudible, that I can't say yet.

[Aleksey Vaneev]

Aliasing is not related to the waveshaping function, at all. Aliasing is caused by discrete band-limited processing.

You can regard every X as an independent signal, every constant as constant signal, and every X*X as spectral convolution, fabs() as two-way (positive&negative) clipping with summation. 1/X as again, two-way X^-1 function with summation. Each such two-way function can be expressed as polynomial consisting of X^n spectral convolutions.

So, thinking this way, some functions may negate the effect of other functions, damping their harmonics (you get a bunch of polynomial members all with different coefficients).

I think the first derivative is all you really need to care about, it is where some "undamped" harmonics may arise. I just do not see how second derivative can play a role, I would be greatful if you could describe it.

And, well, the first derivative of the "ultra" function I've posted is not really smooth, I must have missed it.

[FastTriggerFish]

Well think of it this way : a Dirac impulse has full spectrum.
A step function is the integral of a Dirac, and integration is a low pass in the frequency domain ( multiplication by 1/s )
So a step function still has a wide spectrum, but better than Dirac.
Integrate that again, and you get a ramp, which is your Dirac signal now low-passed with a 2 pole.
Integrate once again and you have a function whose second derivative is discontinuous.
It's spectrum is that of Dirac low passed by 3 poles filter.
It's still a full band signal so the spectrum will fold back = aliasing.

Clearly the higher order derivative that is discontinuous the better, but stil.