I've fantasized about doing this with The Sims (all of 4+ of them in one fancy new engine) for a long time, and there's plenty of source code and documentation available from mod tools last I looked.
As far as a more graphically intensive performance challenge goes, you could look into the Mass Effect series. ME3Explorer (the suite of mod tools that despite it's name actually supports all three games to varying degrees) is open source, and their wiki looks like it has at least good entry-level overviews of the formats involved (see
http://me3explorer.wikia.com/wiki/Textures_in_Mass_Effect for example). You might need to look at their actual code for the details though, for all I know it could be an incomprehensible mess. But it's still an active project so you could probably get answers to any questions you had if you ask them. As a bonus, if you can master these asset formats you'll have one foot in the door to supporting innumerable other Unreal Engine games.
Any 3d Elder Scrolls or Fallout game is gonna have decent documentation buried somewhere under the thousands of of higher level modding tutorials, like here for example:
http://www.uesp.net/wiki/Tes5Mod:File_Formats
Really though, google whatever games you'd be most exited about doing this to plus "file formats" or "mod tools" and see what you can find.