I've been programming hot code reloading into a project of mine in a simlar way that Casey does in Handmade Hero and I've come across some crazy behaviour that I can't explain...

Essentially I have run into a case where the call to LoadLibrary fails with

ERROR_DLL_INIT_FAILED
    1114 (0x45A)
    A dynamic link library (DLL) initialization routine failed.

This happens if I simply comment out a specific function call in the DLL codebase. Additionally, adding a dummy variable declaration fixes the problem!?

I am very confused...

Video demo

Did you try loading the dll in an empty application with just the LoadLibrary call to see if it works ?

Could you share a minimal reproduction case or the full code ?

Do you have lockfile implemented like in HH? If not, then most likely what is happening is that linker starts creating new dll file and your hot reload code notices that new file is there and tries to reload it, but linking process is still in progress - it has not finished to fully produce dll file. So loading it will fail, because partially written dll file has incorrect format. You can easily test this by adding something like Sleep(2000) before LoadLibrary call. If that success, then this is the case.

Yeah even trying to load the dll in an empty application doesn't work. Here's some test code that just tries to get a handle to the dll with LoadLibrary and that's it. It fails for me but if I uncomment line 326 in modaw.cpp and then recompile it will load successfully. Curiously also just deleting any random code you like in modaw.cpp seems to have the same effect.

LoadLibraryBug.zip

I don't think this is the case because I'm not even trying to recompile the dll while the program is running. It just wont even load the dll on startup.

This happens because you did not implement DllMainCRTStartup function properly. It must return TRUE for dll loading to succeed. You are not returning anything, so random garbage is returned (whatever value is in rax register) - most likely 0. And 0 is FALSE which signals OS that dll initialization failed and it must unload it.

For more information what this function must do (and what is its proper signature) is here: https://docs.microsoft.com/en-us/windows/win32/dlls/dllmain

Ah amazing, thank you! Is the prototype for DllMainCRTStartup identical to DllMain? i.e. I should have something like this?

extern "C" int
_DllMainCRTStartup(HINSTANCE Instance, DWORD Reason, LPVOID Reserved)
{
  switch(Reason) 
  { 
    case DLL_PROCESS_ATTACH:
    {
    } break;

    case DLL_THREAD_ATTACH:
    {
    } break;

    case DLL_THREAD_DETACH:
    {
    } break;

    case DLL_PROCESS_DETACH:
    {
    } break;
  }
  return TRUE;
}

Yes, that's correct. Add WINAPI in case you want to have code compatible win 32-bit code. No need for those switch statements if you don't need to do different things based on process/thread start/end.