Hey there,
It's been lots of fun following along with this video series. As someone who's been coding a long while in the web development sphere as well as writing games, it's great to get a fresh perspective on development process along with modern game dev techniques. Thanks a bunch Casey - I'm recommending your series everywhere I go!
So a question: I'm new to the memory pooling approach, and I'm interested in how serialisation might work.
With my previous engine I had to manually walk through all the pointers in a complex hierarchy: it was slow, error-prone and took ages to code and maintain. With a memory pool, it seems like we can mostly save arrays of entities etc direct to disk.
Assuming we aren't fixing the memory base addresses in production, the only thing we'd need to do besides dumping and reloading most of the memory is to fix any pointers. As long as we keep pointers out of the main structures, and only use indexes, in theory we could simply store the old base address we were given, then do some pointer arithmetic to move all the pointers over to the new scheme.
Anything I'm missing? Are there other 'standard' ways of doing this out there?
It's been lots of fun following along with this video series. As someone who's been coding a long while in the web development sphere as well as writing games, it's great to get a fresh perspective on development process along with modern game dev techniques. Thanks a bunch Casey - I'm recommending your series everywhere I go!
So a question: I'm new to the memory pooling approach, and I'm interested in how serialisation might work.
With my previous engine I had to manually walk through all the pointers in a complex hierarchy: it was slow, error-prone and took ages to code and maintain. With a memory pool, it seems like we can mostly save arrays of entities etc direct to disk.
Assuming we aren't fixing the memory base addresses in production, the only thing we'd need to do besides dumping and reloading most of the memory is to fix any pointers. As long as we keep pointers out of the main structures, and only use indexes, in theory we could simply store the old base address we were given, then do some pointer arithmetic to move all the pointers over to the new scheme.
Anything I'm missing? Are there other 'standard' ways of doing this out there?
Edited by Chris Parsons
on
Reason: remove duplicate sig
Yes, that's pretty much you will need to do - adjust pointers to serialized stuff, if you keep any. Or simply don't store any pointers to serialized data, just indices, then doing fread/fwrite will be everything you need to do.
I know people don't like C++ much here, but if you want then for serializing pointers you could use this helper template:
This will allow to write code like this:
C++ will automatically resolve pointer from relative offset. And if you can guarantee that you are placing objects closer together you can create Pointer16 with int16_t offset, saving 2 bytes. Even just by using Pointer32 you will save 4 bytes per pointer on 64-bit architecture.
And you can easily fread/fread whole memory block that stores Blob's as array of bytes.
And because pointers are typically aligned at least at 4 or even 8 bytes, you can increase distance 4 or 8 types by always using 0 as lowest bits of offset and shifting it up 2 or 3 bits.
And if you are OK using MSVC specifc syntax you can look into __based keyword, but I recomend against using it and staying portable.
I know people don't like C++ much here, but if you want then for serializing pointers you could use this helper template:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | template <class T>
struct Pointer32
{
int32_t offset;
T* operator ()
{
return (T*)((uint8_t*)this + offset));
}
Pointer32& operator = (T* other)
{
ptrdiff_t diff = (uint8_t*)other - (uint8_t*)this;
assert(diff>= INT_MIN && diff <= INT_MAX);
offset= (int32_t)diff;
return *this;
}
};
|
This will allow to write code like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 | struct Blob
{
float x;
float y;
Pointer32<Blob> next;
};
Blob* blob = ...; // unserialize
blob->x = 1;
blob->next->x = 2; // here we reference other object by relative offset
Blob* otherBlob = ...;
blob->Next = otherBlob; // change pointer
|
C++ will automatically resolve pointer from relative offset. And if you can guarantee that you are placing objects closer together you can create Pointer16 with int16_t offset, saving 2 bytes. Even just by using Pointer32 you will save 4 bytes per pointer on 64-bit architecture.
And you can easily fread/fread whole memory block that stores Blob's as array of bytes.
And because pointers are typically aligned at least at 4 or even 8 bytes, you can increase distance 4 or 8 types by always using 0 as lowest bits of offset and shifting it up 2 or 3 bits.
And if you are OK using MSVC specifc syntax you can look into __based keyword, but I recomend against using it and staying portable.
mmozeiko
Or simply don't store any pointers to serialized data, just indices, then doing fread/fwrite will be everything you need to do.
One obvious advantage of this is that the same data works on both 32-bit and 64-bit platforms (assuming the same endianness). The size of a "pointer" depends only on the data, not the platform, and the data is known in advance.
chrismdp
With my previous engine I had to manually walk through all the pointers in a complex hierarchy [...]
Of course, from a data-oriented development point of view, the very words "complex hierarchy" are a red flag.
mmozeiko
Or simply don't store any pointers to serialized data, just indices, then doing fread/fwrite will be everything you need to do.
That sounds like the best approach to me - I like your templated pointer class idea though. I guess one way to do it without templates would be to simply store the memory offset from the current class to the next one, and do the addition in a helper method. It would be a shame having to walk through all the lists on deserialise to fix all the pointers, so I'm tempted to take the offset storage approach.
Pseudonym73
Of course, from a data-oriented development point of view, the very words "complex hierarchy" are a red flag.
Yes, quite. I realise that now :)
Thanks for the ideas both!
Edited by Chris Parsons
on