What are some serialization strategies in order to save data to the disk? Obviously just dumping the data to disk is easy, but I'd like something a little more robust. I'd need something that can handle rearranging the members of the struct, and also adding new members, and still be able to load old data from disk. I feel like this might be a metaprogramming/code generation problem?
Edit: I guess a better way to phrase this would be, how can I 'version' my structs if I plan to save and reload them over time and changes. I'd like to be able to work on the game (i.e. creating levels, etc) without having to worry about having to redo everything if I decide to add a field to my 'level' structure (this is not a 'procedural' style game). Also writing a converter every time I make a change feels wrong somehow.
Edit: I guess a better way to phrase this would be, how can I 'version' my structs if I plan to save and reload them over time and changes. I'd like to be able to work on the game (i.e. creating levels, etc) without having to worry about having to redo everything if I decide to add a field to my 'level' structure (this is not a 'procedural' style game). Also writing a converter every time I make a change feels wrong somehow.
Edited by Daniel Moore
on
This problem sounds like a metaprogramming problem.
I know libraries are not that encouraged in this community. But I would suggest you take a look at Protobuf: https://developers.google.com/protocol-buffers/
It is a simple library that tackles exactly your problem. Adding new fields to existing data structures does not break the serialization process. The implementation is quite simple on an abstract level. Every member of a struct is assigned a unique number in that struct. This number is used as a key when loading or storing data. If you add a new member, old data can still be deserialized. In that case the new member will be zero-initialized. If you remove a member, this member will be ignored while parsing old data.
You can use that same strategy to implement your own serialization. Keep in mind that every field must have a unique number. That means even if you remove a member you cannot use that number again.
If you want to do metaprogramming yourself, you could use a format like this:
The SERIALIZE() marks the struct as serializable. The ID(num) sets the unique number for the members of the struct. In this example the float member "Unused" was commented out to show that you cannot use the ID 2 anymore after it has been used by a previous version of the struct.
As you may have noticed this aproach does not require a version number for the whole struct. As long as you do not reuse field numbers and zero-initialisation is okay for not set members this method should be both backwards and forwards compatible.
I know libraries are not that encouraged in this community. But I would suggest you take a look at Protobuf: https://developers.google.com/protocol-buffers/
It is a simple library that tackles exactly your problem. Adding new fields to existing data structures does not break the serialization process. The implementation is quite simple on an abstract level. Every member of a struct is assigned a unique number in that struct. This number is used as a key when loading or storing data. If you add a new member, old data can still be deserialized. In that case the new member will be zero-initialized. If you remove a member, this member will be ignored while parsing old data.
You can use that same strategy to implement your own serialization. Keep in mind that every field must have a unique number. That means even if you remove a member you cannot use that number again.
If you want to do metaprogramming yourself, you could use a format like this:
1 2 3 4 5 6 7 | struct data SERIALIZE()
{
int ID; ID(1)
//float Unused; ID(2)
float X; ID(3)
float Y; ID(4)
};
|
The SERIALIZE() marks the struct as serializable. The ID(num) sets the unique number for the members of the struct. In this example the float member "Unused" was commented out to show that you cannot use the ID 2 anymore after it has been used by a previous version of the struct.
As you may have noticed this aproach does not require a version number for the whole struct. As long as you do not reuse field numbers and zero-initialisation is okay for not set members this method should be both backwards and forwards compatible.
Ok, this is great, it sounds like you understood my problem perfectly. I was trying to think of a way to have the metaprogram create the annotation of the members of the struct, but that's probably my inexperience with metaprogramming. It makes much more sense to just give the metaprogram a little bit of data to help it along.
I will look through protobuf, but this could be a fun programming challenge. I did have something partially working that was similar, where it saved a header on the file that described the structure as it was in the file, but I started getting discouraged trying to serialize structs within structs.
I will look through protobuf, but this could be a fun programming challenge. I did have something partially working that was similar, where it saved a header on the file that described the structure as it was in the file, but I started getting discouraged trying to serialize structs within structs.