About a year ago I published a few articles on my experiments with C++ reflection system. It extracts all the type information from PDB files. Nice thing – it’s all automatic and requires none to very little source code changes. One thing I didn’t like was problem with adding metadata to our types/fields. It’s mostly for editing purposes, but wouldn’t it be nice to have at least a help/description string for fields? I tried to overcome it by adding special comments in C++ headers and parsing it myself. It works, but isn’t terribly efficient/flexible. It probably could be extended without that much work, but I still tried to think of a better solution. Luckily, one of main ideas behind the whole system is that all the information is loaded from external file. Once it’s generated, it should be quite easy to add new data. I toyed with the following options:
- type structure loaded from PDB, metadata defined by user in another file (custom format),
- data definition language for type structures & metadata,
- PDB + visual editor in C# (with possibility of adding metadata).
They all have advantages & disadvantages and version #1 would probably be the easiest/quickest to code, I decided to go with version #2 (DDL) however, as it looked the most interesting (ah, the luxury of home coding).
At first, I wanted to use some existing standard to describe my types. I knew I did not want to go with XML, but I had heard lots of good things about JSON, so wanted to give it a shot. As it turned out – yes, it’s more concise & readable than XML, but then again, it’s like saying an elephant is smaller than a whale. I quickly tried some very basic definitions and it was waaay too much typing, way too much noise for my taste. Had to drop my initial JSON+Python plan. Instead, I decided to create my own format + whip up a simple Perl parser. Here’s sample type definition (pretty much same as types I used for my previous PDB test):
type Vector3d float x [-1, 1] "X coordinate" float y "Y coordinate" float z "Z coordinate" end type Color float r [-1, 1] "R" float g [-1, 1] float b [-1, 1] "B" end type Entity Vector3d position "entity position" uint32 i "counter" int8 c "char" end type SuperBar vtable uint32 i float* p "ptr" bool b "b" int8 s "some char" Color color "Color of something" SuperBar* psb "Other SuperBar" vector<int32> v "Vector of ints" end type Bar : SuperBar enum TestEnum FIRST = 0, SECOND, LAST = 10 end float f uint8 c uint16 shortArray "array of shorts" Vector3d position "Position" Vector3d* porient "Orientation" SuperBar sb SuperBar** pb TestEnum en "Some enum" end
As you can see – it’s pretty close to C/C++ code, but I don’t require semicolons and use slightly different keywords (e.g. I don’t care if it’s a class or structure). Now, this system obviously can’t handle virtual functions (no obvious way to get addresses), the vtable definition is only to preserve compatibility with my test data. This definition file is now parsed by Perl script and result is binary file with type description. It’s almost exactly the same as one generated by PDB reader. The difference is, for every field there’s optional field edit info structure (containing help string + bounds).
Here’s a sample screenshot of a small C# application I coded to test my editor metadata:
Next step would be generating corresponding C/C++ header from our DDL (quite trivial as formats are so similar). Problem is – it’s still additional file that needs to be maintained, PDB solution feels more automatic. Those two could be easily combined – approach #1 mentioned before (Perl loads binary generated by PDB reflector and ‘decorates’ it with metadata). Another nice thing is that metadata is completely optional, it can be easily stripped from the final version of the game, the only cost is pointer per field (=4 bytes). It’d also be very easy to add possibility of actually editing the metadata from game editor (so that you can update field descriptions, modify their limits or flags etc). Storing all information externally really gives plenty of possibilities to update/edit them as we please.
Here you can download my test package containing new reflection reader, Perl script and sample DD file. Obviously, the script is very simple, it handles my test scenario and most of situations I encounter during my experiments, but it’d probably require serious modifications before employing in a professional environment (mostly because it’s been only tested with MSVC, as mentioned in comments for previous articles, GCC may give slightly different layout).