I searched the forum but I couldn't find and answer. I don't have the source code so I can't check.
I'm implementing something similar to the HM profiler and I have something very similar to the HM hot code reloading.
In my profiling macro I use the compiler macros
to identify the profiled section.
When I gather the data I keep the pointer to those strings. When I recompile and it reloads sometimes the addresses of those strings change and I crash when I access them.
I was wondering if the HM profiler is compatible with the hot code reloading. (or runtime compiled c++) (<- this is for SEO ;P )
Thinking about a possible solution I cannot imagine one without string searching, string comparing and string copying. Even a simple hash function could be heavy in some profiling situations, and searching for a string implies cache dirtying. :\
I'm implementing something similar to the HM profiler and I have something very similar to the HM hot code reloading.
In my profiling macro I use the compiler macros
1 | __FILE__ __LINE__ __PRETTY_FUNCTION__ |
When I gather the data I keep the pointer to those strings. When I recompile and it reloads sometimes the addresses of those strings change and I crash when I access them.
I was wondering if the HM profiler is compatible with the hot code reloading. (or runtime compiled c++) (<- this is for SEO ;P )
Thinking about a possible solution I cannot imagine one without string searching, string comparing and string copying. Even a simple hash function could be heavy in some profiling situations, and searching for a string implies cache dirtying. :\
Edited by Gianluca Alloisio
on
IIRC, we do the string copy during collation, thus getting the benefit of persisting across hot code reloading, but avoiding the problem of inserting nasty copy code into the middle of something that is being profiled.
- Casey
- Casey
Hmm. String copying is made atomic? Unfortunate timing could catch the copying during dl loading.
I also asked to Doug Binks (the RCC++ guy):
I also asked to Doug Binks (the RCC++ guy):
HM Tweet embedding
@theGiallo My RCC++ impl. keeps original exe and all dll's active, so you can reference old ptrs if you want.
— Doug Binks (@dougbinks) 4 august 2016
No, it cannot. We control the timing (obviously), so we only reload the DLL after the debug collation has completed.
- Casey
- Casey
Thou art a wiseman.
---------------------------
Disclaimer: it's 4am, I'm sleepy thus I don't 100% endorse my mind.
The problem I see is that after you collate and before you reload, other threads could push new values, with e.g. __FILE__. Then you would reload and when collating next time pointers could be invalid. I don't see a simple solution not involving copying at record time or making it a "critical section" together with the dl loading and doing a bit of collation at its beginning.
I've tried the "leave all dl loaded" solutions and it seems to work.
I output the dl with a random name and I write it on a file. The program then checks for modification time of that file and reads it to get the dl name and loads the dl.
I delete all the dl if when compiling there is not the process running (although in Linux I could delete every time).
I feel you weighted up this solution but went for the "copy" one. Do you recall the reasons, if was so?
---------------------------
Disclaimer: it's 4am, I'm sleepy thus I don't 100% endorse my mind.
The problem I see is that after you collate and before you reload, other threads could push new values, with e.g. __FILE__. Then you would reload and when collating next time pointers could be invalid. I don't see a simple solution not involving copying at record time or making it a "critical section" together with the dl loading and doing a bit of collation at its beginning.
I've tried the "leave all dl loaded" solutions and it seems to work.
I output the dl with a random name and I write it on a file. The program then checks for modification time of that file and reads it to get the dl name and loads the dl.
I delete all the dl if when compiling there is not the process running (although in Linux I could delete every time).
I feel you weighted up this solution but went for the "copy" one. Do you recall the reasons, if was so?
Edited by Gianluca Alloisio
on
Reason: I don't know the policy about multiposting here, so I'm editing
No, that is not how the system works. We wait for all threads to finish, then we turn off debug event recording, then we collate. The next time a DLL of the game is successfully loaded, we turn debug event recording back on. So there is no way for any threads to write to the debug log during the changeover. This is important for other reasons, namely that you don't want debug information to be interrupted in the middle of a block, etc.
- Casey
- Casey
Oh, I see. That works. But you have to stop all the threads during collation.
I have a single-writer single-reader non-locking-queue for each thread for the section profiling data. I also have a big "FRAME" section on thread 0, containing a whole single frame. I collate the other threads' data up to the end time of the thread 0 "FRAME" section. If there is a not-ended section I just set the end time to 0. And yes, I collate in chunks, one per "FRAME".
Aside from these differences, do you see any drawbacks keeping all the dls open?
I have a single-writer single-reader non-locking-queue for each thread for the section profiling data. I also have a big "FRAME" section on thread 0, containing a whole single frame. I collate the other threads' data up to the end time of the thread 0 "FRAME" section. If there is a not-ended section I just set the end time to 0. And yes, I collate in chunks, one per "FRAME".
Aside from these differences, do you see any drawbacks keeping all the dls open?
Sorry to keep replying with "no", but no, you do not stop all threads during collation! You only stop all threads when you are going to reload the DLL. If you're not going to reload, you just let collation happen concurrently with the other threads.
- Casey
- Casey
Edited by Casey Muratori
on
Oh, ok! I understood you were doing that :D
Oh and...
Oh and...
theGiallo
Aside from these differences, do you see any drawbacks keeping all the dls open?
Edited by Gianluca Alloisio
on
I'm not sure if there are downsides to keeping the DLL open. It feels a little bit like a faux solution to me for most things (like function pointers), because although it allows those sorts of things to still work, you're calling old code, which isn't the goal of hot reloading. So I think you really could _only_ use this as something to keep strings around for debug purposes, and at that point, it seems like it should be fine, since they don't really hamper the behavior of the program...
However, I suppose I would be concerned about what might happen if you don't unload the DLL in that you might never know if you were secretly using globals or function pointers which are now stale, and instead of crashing things just sort of subtly don't work - I don't know that I want that risk, so I would say there might still be a reason to force-unload the DLL so you can ensure you're always properly running with new code, not perhaps running with partial old code and not knowing.
- Casey
However, I suppose I would be concerned about what might happen if you don't unload the DLL in that you might never know if you were secretly using globals or function pointers which are now stale, and instead of crashing things just sort of subtly don't work - I don't know that I want that risk, so I would say there might still be a reason to force-unload the DLL so you can ensure you're always properly running with new code, not perhaps running with partial old code and not knowing.
- Casey
Yes, now I'm using old dls only for strings.
About the second concern I'm not really sure. :\
Hmm. One problem could be if you store a pointer to a global or a global pointer and then change the dl. The new global could be different and you could modify it, not seeing the new value because watching at the old memory. This is true also for pointers to functions in the dl.
Were you thinking at that? Avoiding that or resetting the values after a dl reload could solve the problem (apart from not adhering to that guideline :P).
About the second concern I'm not really sure. :\
Hmm. One problem could be if you store a pointer to a global or a global pointer and then change the dl. The new global could be different and you could modify it, not seeing the new value because watching at the old memory. This is true also for pointers to functions in the dl.
Were you thinking at that? Avoiding that or resetting the values after a dl reload could solve the problem (apart from not adhering to that guideline :P).