Handmade Hero » Forums » Code » Memory mapping .hmi files and interpreting MSDN
gilman
2 posts
#10305 Memory mapping .hmi files and interpreting MSDN
8 months, 2 weeks ago

Way back in Week 5 Casey was trying to improve the speed of initializing input recording by Memory Mapping the .hmi file into memory so that we could just do a (faster) MemCopy call to save the game state, however I realized that the main reason that the initial save of the game state was not sped up by the Memory Mapping was that we did not actually use it :p by the end of the episode we got all "piggy" and mapped 4 .hmi files into memory and then used the file handle into the file on the HD to append input data to the end.
So I decided to dig into it more (mainly here https://msdn.microsoft.com/en-us/...ws/desktop/aa366537(v=vs.85).aspx) and found that windows garauntees that Memory Mappings are coherent and identical between all Mapped Views into the file and the file itself (well it says that they're not necessarily coherent when using WriteFile ... but why else does it block for so long?) - so when we used the file handle to seek to the end of the file the call was blocked until Windows was able to flush the game state to disk

So yay problem understood/solved except it also turned out that Memory mapped files also do not contain any way to easily append to the backing file, you cannot use the Handle from CreateFileMapping() in the same way as a regular File Handle and MapViewOfFile() cannot map past the file size initialized with CreateFileMapping() (except when the end of the file doesn't use up all of the page size) So in the end I had to keep track of how far I had written/read in the file myself and come up with my own strategy for expanding the size of the file to make space for new input - this is where MSDN is fairly vague as to what it allows you to do - which was to call CreateFileMapping() again with a larger size and Map a new view of the file into memory.
MSDN makes it sound like you have to close all open Mapped Views before you can call CreateFileMapping() on the same file handle but the process still works if you resize the file first - and with the tests I ran I could even see changes made in the View mapped using the second FileMapping handle, in the original view mapped using the first FileMapping handle)

Here's how my code ended up

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
internal void win32_endRecordingInput(win32_state *state)
{
	state->inputRecordingIndex = 0;
	state->currentBuffer->writtenSize = state->currentRecordSize;
	state->currentBuffer = 0;
}

internal void win32_beginInputRecording(win32_state *state, int index)
{
	win32_replay_buffer *buffer = win32_getReplayBuffer(state, index);

	if(!buffer->initialized)
	{
		win32_getInputFileLocation(state, index, sizeof(buffer->fileName), buffer->fileName);
		buffer->mappedFile = CreateFileA(buffer->fileName, GENERIC_READ|GENERIC_WRITE, FILE_SHARE_WRITE, 0, CREATE_ALWAYS, 0, 0);

		//NOTE: Memory mapping does not allow you to extend the size of
		//the file, you have to recreate the file mapping with a larger
		//size and then map a new view of the file
		LARGE_INTEGER size;
		size.QuadPart = state->totalSize + Kilobytes(4); //avoid the mandatory resizing for the first inputs
		buffer->memoryMap = CreateFileMapping(buffer->mappedFile, 0, PAGE_READWRITE, size.HighPart, size.LowPart, 0);
		buffer->memory = MapViewOfFile(buffer->memoryMap, FILE_MAP_ALL_ACCESS, 0, 0, size.QuadPart);
		buffer->fileSize = size.QuadPart;
		buffer->initialized = 1;
		if(!buffer->memory)
		{
			//TODO: Logging
		}
	}

	if(buffer->memory)
	{
		state->inputRecordingIndex = index;
		state->currentBuffer = buffer;
		CopyMemory(buffer->memory, state->gameMemoryBlock, state->totalSize);
		state->currentRecordSize = state->totalSize;
	}
}

internal void win32_beginInputPlayback(win32_state *state, int index)
{
	win32_replay_buffer *buffer = win32_getReplayBuffer(state, index);
	if(buffer->initialized)
	{
		if(buffer->memory)
		{
			state->inputPlayingIndex = index;
			CopyMemory(state->gameMemoryBlock, buffer->memory, state->totalSize);
			state->currentRecordSize = state->totalSize;
			state->currentBuffer = buffer;
		}
	}
	else
	{
		//TODO: Logging
	}
}

internal void win32_endInputPlayback(win32_state *state)
{
	state->inputPlayingIndex = 0;
	state->currentBuffer = 0;
}

internal void win32_recordInput(win32_state *state, game_input *newInput)
{
	if(state->currentBuffer)
	{
		uint32_t bytesToWrite = sizeof(*newInput);
		if(state->currentRecordSize+bytesToWrite >= state->currentBuffer->fileSize)
		{
			state->currentBuffer->fileSize += Kilobytes(4);

			/*
			 *
			 * NOTE: This seems to work with just creating a new FileMapping
			 * mapping a new view then unmapping the old view instead of first
			 * closing the old handle, everything works but it seems to go
			 * counter to what CreateFileMapping says on MSDN (as I understand
			 * it) which is that if an existing object exists that one is
			 * returned (with its old file size instead of the new size
			 * requested)
			 * 
			 * Doing it this way doesn't seem to make any difference in the
			 * loading/memcopy times (dirty pages of closed memory mapped files
			 * are lazily flushed to disk) so it should be safer
			*/
			//TODO: maybe try holding 2 memory mappings? 1 to the state and 1 to the input
			//gotchas: windows might not close the memoryMap Handle if it has mapped views & need to map offsets on page boundaries
#if 0
			void *oldMem = state->currentBuffer->memory;
			state->currentBuffer->memoryMap = CreateFileMapping(state->currentBuffer->mappedFile,
                                                                            0, PAGE_READWRITE,
                                                                            (state->currentBuffer->fileSize >> 32),
                                                                            state->currentBuffer->fileSize & 0xFFFFFFFF, 0);
			state->currentBuffer->memory = MapViewOfFile(state->currentBuffer->memoryMap, FILE_MAP_ALL_ACCESS,
                                                                     0, 0, state->currentBuffer->fileSize);
			UnmapViewOfFile(oldMem);
#else
			void *oldMem = state->currentBuffer->memory;
			UnmapViewOfFile(oldMem);
			CloseHandle(state->currentBuffer->memoryMap);
			state->currentBuffer->memoryMap = CreateFileMapping(state->currentBuffer->mappedFile,
                                                                            0, PAGE_READWRITE,
                                                                            (state->currentBuffer->fileSize >> 32),
                                                                            state->currentBuffer->fileSize & 0xFFFFFFFF, 0);
			state->currentBuffer->memory = MapViewOfFile(state->currentBuffer->memoryMap, FILE_MAP_ALL_ACCESS,
                                                                     0, 0, state->currentBuffer->fileSize);
#endif
		}

		void *writePos = (void *)((char *)state->currentBuffer->memory+state->currentRecordSize);
		CopyMemory(writePos, (void *)newInput, bytesToWrite);
		state->currentRecordSize += bytesToWrite;
	}
}

internal void win32_playbackInput(win32_state *state, game_input *newInput)
{
	if(state->currentBuffer)
	{
		uint32_t bytesToRead = sizeof(*newInput);
		if(state->currentRecordSize >= state->currentBuffer->writtenSize)
		{
			int index = state->inputPlayingIndex;
			win32_endInputPlayback(state);
			win32_beginInputPlayback(state, index);
		}
		void *readPos = (void *)((char *)state->currentBuffer->memory+state->currentRecordSize);
		CopyMemory((void *)newInput, readPos, bytesToRead);
		state->currentRecordSize += bytesToRead;
	}
}


So in the end I got good performance out of doing it this way with ~2 sec pause the first time and ~1 sec thereafter, and I get to keep everything all together in 1 file :D but I'm still not sure if maybe I'm just interpreting MSDN wrong in regards to expanding the file size or not, maybe I'm not reading between the lines enough?