Crash in MachO ImageLoader

[Urs]

Hi,

does anyone have random crashes on or shortly after

Code: Select all

ImageLoaderMachO::doInitialization

on random user's machines? (Macs, obviously)

We have this super super rarely, and we have pretty much ruled out it's a bug on our side. These users have sent us crash logs from other companies as well, so we know we are not alone.

I would like to connect to other companies and see if there is a common denominator on the user's machines. We have a hunch, but our sample size is too small to act upon it. In other words, if anyone out there has similar issues, I'd love to connect.

If you are a user experiencing this issue, and if you're in contact with any developer about this, it would be great if you could point the developer to this thread, or to me directly.

Thanks,

- Urs

[FigBug]

I seen to recall a lot of weird crashes being caused by users having plugins that used an old version of iLok that corrupted memory.

[Markus Krause]

I remember a couple of issues when I used a png compressor tool to shrink them further. Went back to use Photoshop-only png files. Maybe some corrupt png file or a bug in a library

[Urs]

Thank you,

To clarify: The ImageLoader has nothing to do with graphics. It's about loading the memory image from the drive into the RAM.

The crash happens *before* any code in a plug-in is executed. It happens during the initialisation of static memory in the freshly loaded executable, before the "main" of the bundle is called. It has nothing to do with the order of initalisation, or dependencies of static memory. The symbol accessed during the crash can be anything as simple as 'const int blah = 7;' in the global space.

- U

[mystran]

The crash happens *before* any code in a plug-in is executed.

If we look at ImageLoaderMachO::doInitialization here https://opensource.apple.com/source/dyl ... .auto.html and the two functions it calls (which might not show up in stack trace if they get inlined), we see that the only type of initialization it supports is calling initialization functions stored in the image headers (eg. I think constructors and such would go here) where I believe normally truly static data is simply stored directly in the data segment (at least that's the case with other formats).

Assuming the above code is roughly what ships (which is not unlikely, because you don't usually touch these sorts of things unless you absolutely have to), then there shouldn't be a whole lot here that could go wrong only sporadically, unless something is randomly corrupting the image headers before the function runs, or one of the initializers trashes stack. I think this might be a case where you might want to try to catch it with valgrind (not sure if we works with recent macOS though) or AddressSanitizer (although that won't help if the actual problem is Apple code.. but it could rule out that it's at least not some global constructor).

[audiotools]

A full crash-log would be helpful

but it reminds me of

https://www.fabfilter.com/forum/add-comment?id=6069

[Markus Krause]

The crash happens *before* any code in a plug-in is executed.

I remember such an issue in combination with code signing.

During development I also experienced weird caching issues with AudioUnits running under Rosetta. To me it looked like some kind of race-condition with the translation from x86 code to ARM.

[Urs]

Here's a relevant crash log of ours (hope there's no personal user data...):

Code: Select all

Process:               Live [33887]
Path:                  /Applications/Ableton Live 10 Suite.app/Contents/MacOS/Live
Identifier:            com.ableton.live
Version:               10.1.42 [2021-11-04_537ff2c643] (10.1.42 [2021-11-04_537ff2c643])
Code Type:             X86-64 (Native)
Parent Process:        ??? [1]
Responsible:           Live [33887]
User ID:               501

Date/Time:             2022-06-04 12:15:02.047 -0400
OS Version:            Mac OS X 10.15.7 (19H1824)
Report Version:        12
Bridge OS Version:     6.4 (19P4243)
Anonymous UUID:        19C0B23D-7E08-596C-7AAB-BC5894504174

Sleep/Wake UUID:       545CE373-4112-4516-8641-25E260B6B2B1

Time Awake Since Boot: 850000 seconds
Time Since Wake:       180 seconds

System Integrity Protection: enabled

Crashed Thread:        0  MainThread  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Trace/BPT trap: 5
Termination Reason:    Namespace SIGNAL, Code 0x5
Terminating Process:   exc handler [33887]

Application Specific Information:
/Library/Audio/Plug-Ins/VST3/Satin.vst3/Contents/MacOS/Satin

Thread 0 Crashed:: MainThread  Dispatch queue: com.apple.main-thread
0   com.u-he.Satin.vst3           	0x0000000151bcbf22 0x151ad2000 + 1023778
1   dyld                          	0x00000001146f9353 ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 535
2   dyld                          	0x00000001146f975e ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
3   dyld                          	0x00000001146f417b ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 493
4   dyld                          	0x00000001146f2234 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
5   dyld                          	0x00000001146f22d4 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
6   dyld                          	0x00000001146e3ef2 dyld::runInitializers(ImageLoader*) + 82
7   dyld                          	0x00000001146ee01b dlopen_internal + 609
8   libdyld.dylib                 	0x00007fff699a8d8a dlopen + 171
9   com.apple.CoreFoundation      	0x00007fff2fd648b2 _CFBundleDlfcnLoadBundle + 147
10  com.apple.CoreFoundation      	0x00007fff2fddd5f5 _CFBundleLoadExecutableAndReturnError + 482
11  com.ableton.live              	0x000000010cd7260e 0x10c23c000 + 11757070
12  com.ableton.live              	0x000000010cd716cf 0x10c23c000 + 11753167
13  com.ableton.live              	0x000000010e5115f3 0x10c23c000 + 36525555

The Symbol referenced in our plug-in is something like "const int blah = 7;" in the global space. It is something else in each crash log we got.

[Urs]

A full crash-log would be helpful

but it reminds me of

https://www.fabfilter.com/forum/add-comment?id=6069

This is interesting. In wonder what workaround they achieved!

[mystran]

Oh .. it's a SIGTRAP rather than a regular crash (and upon looking at the FF thread, that seems to be the case there as well). That should only happen if you have a breakpoint (eg. INT3 = 0xCC) or someone's messing with the debug registers.. or I guess uncaught exceptions .. etc. Aggressive copy-protection gone rogue could certainly be one possibility.

ps. This looks similar too: https://forum.cockos.com/archive/index. ... 56650.html and this: https://www.logicprohelp.com/forums/top ... g-session/ and https://www.logicprohelp.com/forums/top ... rtup-1051/ (found the last one searching for "sigtrap pace" for what it's worth)

[Markus Krause]

I seen to recall a lot of weird crashes being caused by users having plugins that used an old version of iLok that corrupted memory.

I can confirm that ilok protected plugins can crash other plugins. They have been notified by me about this for many times but ignore it since a decade. I had to recompile all my stuff. It was very annoying.
ILOK installs a low-level Windows driver with root access to your computer. Several important Windows system runtime libraries (dll files owned by Microsoft!) are modified. This alters the behaviour of Windows itself. Certain Windows functions which like GetThreadContext() can return false values. As a result software from various other companies can crash or do false calculations.
The crash appears only when the software is running at the same time as the ILOK protected plugin. It affects not only plugins, but also other software (Skype for example).

Proof:

//this falsely returns true when an ILOK protected plugin is running at the same time
bool CheckHardwareBreakpoints()
{
unsigned int NumBps = 0;

// This structure is key to the function and is the
// medium for detection and removal
CONTEXT ctx;
ZeroMemory(&ctx, sizeof(CONTEXT));

// The CONTEXT structure is an in/out parameter therefore we have
// to set the flags so Get/SetThreadContext knows what to set or get.
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;

// Get a handle to our thread
HANDLE hThread = GetCurrentThread();

// Get the registers
if(GetThreadContext(hThread, &ctx) == 0)
return false;

// Now we can check for hardware breakpoints, its not
// necessary to check Dr6 and Dr7, however feel free to
if(ctx.Dr0 != 0)
++NumBps;
if(ctx.Dr1 != 0)
++NumBps;
if(ctx.Dr2 != 0)
++NumBps;
if(ctx.Dr3 != 0)
++NumBps;

if (NumBps>0) return true; else return false;
}

[Urs]

in 35 crash logs, each time an iLok protected plug-in is in the list of Binary Images directly over the plug-in that's loaded when MachO ImageLoader crashes.

We're happy to provide those crash logs, stripped off personal date.

[Markus Krause]

The fact that really sucks is that iLok crashes the software of other non-iLok developers. This makes customers think that iLok works as expected and your products are blamed for their technical issues.