Need VST host help

[DougCox]

I need help with using CreateDialogIndirectParam in my VST host program, PianoRollComposer, a Windows program written in C with Visual Studio 2008.

I can use DialogBoxIndirectParam with no problem. But it's a modal dialog box, and I have to close it to use the VST plugin.

HOWEVER, in the code snippets below, with the lines for using DialogBoxIndirectParam commented out,
when I try using CreateDialogIndirectParam with any plugin, the VST plugin window appears, but then stays stuck in limbo, with the hour-glass symbol staying
(WM_INITDIALOG is called, but WM_KEYDOWN and WM_COMMAND can't be gotten to).

What am I doing wrong?
(the entire source code is at http://jdmcox.com/PianoRollComposer.cpp)

Code: Select all

//this is in the main thread:
struct MyDLGTEMPLATE: DLGTEMPLATE {
	MyDLGTEMPLATE () { memset (this, 0, sizeof (*this)); };
	WORD ext[3];
};
MyDLGTEMPLATE dt;

// this is in VSTthread:
dt.style = WS_POPUP|WS_VISIBLE|WS_CAPTION|WS_BORDER|WS_SYSMENU;
//DialogBoxIndirectParam(hModule[plugin], (LPCDLGTEMPLATE)&dt, NULL, (DLGPROC)EditorProc, (LPARAM)effect[plugin]);
hwndDialog[plugin] = CreateDialogIndirectParam((HINSTANCE)hModule[plugin], (LPCDLGTEMPLATE)&dt, NULL, (DLGPROC)EditorProc, (LPARAM)effect[plugin]);

//this is in the main thread:
LRESULT CALLBACK EditorProc(HWND hwndedit, UINT msg, WPARAM wParam, LPARAM lParam)
{
	static AEffect* theEffect;
	static ERect* eRect;
	static RECT editorRect;

	switch(msg)
	{
	case WM_INITDIALOG:
		theEffect = (AEffect*)lParam; // effect[plugin]
		SetWindowText (hwndedit, TEXT("    *** CLOSE THIS TO USE PLUG-IN ***")); // necessary with DialogBoxIndirectParam
		if (theEffect) {
			editoropen = TRUE;
			theEffect->dispatcher(theEffect, effEditOpen, 0, 0, hwndedit, 0);
			eRect = 0;
			theEffect->dispatcher(theEffect, effEditGetRect, 0, 0, &eRect, 0);
			if (eRect)
				MoveWindow(hwndedit, eRect->left, eRect->top, eRect->right - eRect->left, eRect->bottom - eRect->top+Title, TRUE);
		}
		break;

	case WM_KEYDOWN: // to refresh EastWest's window
		if (eRect) {
			editorRect.top = eRect->top;
			editorRect.bottom = eRect->bottom;
			editorRect.left = eRect->left;
			editorRect.right = eRect->right;
			InvalidateRect(hwndedit, &editorRect, FALSE);
		}
		break;

	case WM_COMMAND:
		switch (LOWORD(wParam))
		{
		case IDOK:
		case IDCANCEL:
		case WM_CLOSE:
			editoropen = FALSE;
			if (theEffect)
				theEffect->dispatcher (theEffect, effEditClose, 0, 0, 0, 0);
//			EndDialog (hwndedit, TRUE);
//			return TRUE;
			DestroyWindow(hwndDialog[plugin]);
			hwndDialog[plugin] = NULL;
			break;
		}
		break;
	}
//	return 0;
	return DefWindowProc(hwndedit, msg, wParam, lParam);
}

[aciddose]

Are you calling the dispatcher functions with the window handle? If you only call it with your own handle it'll filter only messages for your window. The dialog messages will never be processed, except for the init message which is passed immediately as the window is created.

Code: Select all

while (PeekMessage(&msg, (HWND)my_window_handle, 0, 0, PM_REMOVE) || PeekMessage(&msg, 0, WM_TIMER, WM_TIMER, PM_REMOVE)) {
	// translate wm_keydown to wm_char
	TranslateMessage(&msg);
	DispatchMessage(&msg);
}

while (PeekMessage(&msg, (HWND)dialog_window_handle, 0, 0, PM_REMOVE)) {
	// translate wm_keydown to wm_char
	TranslateMessage(&msg);
	DispatchMessage(&msg);
}

I don't see any dispatch for the vst dialog messages in wWinMain, although you do manually filter for other dialogs. I'm not so sure this is the best way to handle filtering messages, you may want to let the get/peek message function do this for you instead and use multiple loops and a list of windows you need to process for.

For example:

Code: Select all

// sorry if this code doesn't actually work, just fudging this in so you get the idea
typedef std::list<HWND> window_handle_list_t;
for (window_handle_list::iterator i = my_window_handle_list.begin(); i; i++) {
	while (PeekMessage(&msg, (HWND)i->window_handle, 0, 0, PM_REMOVE)) {
		// translate wm_keydown to wm_char
		TranslateMessage(&msg);
		DispatchMessage(&msg);		
	}
}

[DougCox]

That sounded like a good idea, but when I added this:

Code: Select all

int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPWSTR lpCmdLine, int nCmdShow)
{
	.....
	while (GetMessage(&msg, NULL, 0, 0))
	{
                .....
		if (hwndDialog[plugin] && (IsDialogMessage(hwndDialog[plugin], &msg)))
			x=x; // put a breakpoint here
		else {
			TranslateMessage(&msg);
			DispatchMessage(&msg);
		}
	}
	return (int)msg.wParam;
}

when I ran it in debug mode, it never stopped at the breakpoint I set at x=x;

[aciddose]

http://msdn.microsoft.com/en-us/library ... deless_box

Unfortunately I'm not familiar enough with this stuff to say "oh, you just need to _______". You'll have to read the *gasp* documentation. (read the red word in an evil dracula voice.)

[DougCox]

Thanks for your help, aciddose! Your first suggestion has finally sent me in the right direction. (I read that Modeless Box section, as well as Charles Petzold's Programming Windows (Fifth Edition) -- but it's been a long time since then -- I'll re-read it). You were right that a modeless dialog box gets its messages through the main window's GetMessage loop (I'd forgotten). So I'll have to take that line

Code: Select all

hwndDialog[plugin] = CreateDialogIndirectParam(hModule[plugin], (LPCDLGTEMPLATE)&dt, NULL, (DLGPROC)EditorProc, (LPARAM)effect[plugin]);

out of the VSTthread, and put it into the main thread.

[DougCox]

I ended up putting a while (PeekMessage(...)) in the vstThread, and it works nicely, although I have to clean up a couple of things.

[aciddose]

Be careful if you're trying to direct messages between threads. Not all messages are thread-safe.

Why are you using a separate "vstthread" anyway? Is it really necessary?

Perhaps it is if you want to prevent a plugin from locking your main thread, but just keep in mind that it might make things more complicated. Be on the lookout for message queue issues.

[DougCox]

I use a VST thread because I presumed that multi-core CPUs would automatically use a separate core for each thread (instance of a plugin). My program works fine with EWQL's Hollywood instruments, which use a LOT of resources. But thanks for the advice about message queue issues and thread-safe issues.

[JCJR]

I am rather ignorant, so take with a grain of salt. Maybe in some environments gui can be handled from any thread, but there are window/control components and gui frameworks that are only safe from the main thread. Including the ones I've used.

So I handle plugin window gui on the main thread. Various plugins probably use every conceivable set of custom controls ever written, so even if some visual controls might be thread-safe, it doesn't seem reasonable to assume any generic plugin one might select, would use completely thread-safe controls and visual components.

I don't know how you are handling audio, but if you are set up with asio or some other driver model where the audio calls callbacks in your code to ask for output audio buffers, or to deliver input buffers-- In that case the audio part of your code, and the plugin process methods you call from your audio callback, are operating on a different thread, owned by the audio driver or the audio components in the system or whatever. So in that case you get modest multithreading of audio playback or recording "for free" without having to create a discrete thread.

I've found that quite a lot of processing can happen on the audio thread before things bog down and get glitchy, though a very busy multitrack audio program might be forced to spawn additional audio threads. I figure it is needless complication to make new threads as long as playback can keep up "on demand" in the audio callbacks.

If you use a windows multimedia timer mechanism in your program, then as best I can tell windows implements such timers as threads nowadays. Similarly, the legacy timer routines on mac are apparently implemented as threads nowadays.

So even if you don't spawn your own threads, if you use a callback-type audio driver model and a multimedia timer, you are probably working at least three threads anyway.

Most modern plugin api's seem to separate gui code from audio processing code, almost like it is two dlls in the same wrapper. So the plugin writers expect that the gui will most likely get called from possibly different threads than the audio.

Though maybe there are environments where gui is safe from multiple threads, I stay vigilant to only do gui on the main thread. Sometimes it could get tricky-- Maybe there is a function which writes to the screen, but that function might get accidentally called from the audio thread (called deep in a chain of calls from an audio callback), or accidentally called from a timer handler.

So for instance if a time or audio related action wants to redraw an on-screen indicator, the timer or audio code needs to either set a flag or send a message that the main thread will notice later, and respond to the state change by updating the screen. So that the main thread is the only place gui interaction can happen.

[DougCox]

JCJR, thanks for your excellent comments!
Maybe I'll work a paradigm shift on my program and put all gui routines in the main thread. Not only because of your comments but because I'm still having very odd problems getting the modeless plugin editor dialog box to work right. Almost no one is using my program besides me, and I'm fine with a modal plugin editor dialog box. But I've done a lot of programming just to see if I could do it, and I guess I'll keep on.