Reflection in C++ – metadata

About a year ago I published a few articles on my experiments with C++ reflection system. It extracts all the type information from PDB files. Nice thing - it’s all automatic and requires none to very little source code changes. One thing I didn’t like was problem with adding metadata to our types/fields. It’s mostly for editing purposes, but wouldn’t it be nice to have at least a help/description string for fields? I tried to overcome it by adding special comments in C++ headers and parsing it myself. It works, but isn’t terribly efficient/flexible. It probably could be extended without that much work, but I still tried to think of a better solution. Luckily, one of main ideas behind the whole system is that all the information is loaded from external file. Once it’s generated, it should be quite easy to add new data. I toyed with the following options:

• type structure loaded from PDB, metadata defined by user in another file (custom format),

• data definition language for type structures & metadata,

• PDB + visual editor in C# (with possibility of adding metadata).

They all have advantages & disadvantages and version #1 would probably be the easiest/quickest to code, I decided to go with version #2 (DDL) however, as it looked the most interesting (ah, the luxury of home coding).

At first, I wanted to use some existing standard to describe my types. I knew I did not want to go with XML, but I had heard lots of good things about JSON, so wanted to give it a shot. As it turned out - yes, it’s more concise & readable than XML, but then again, it’s like saying an elephant is smaller than a whale. I quickly tried some very basic definitions and it was waaay too much typing, way too much noise for my taste. Had to drop my initial JSON+Python plan. Instead, I decided to create my own format + whip up a simple Perl parser. Here’s sample type definition (pretty much same as types I used for my previous PDB test):

type Vector3d
  float x [-1, 1] "X coordinate"
  float y "Y coordinate"
  float z "Z coordinate"
end 

type Color
	float r [-1, 1] "R"
	float g [-1, 1]
	float b [-1, 1] "B"
end

type Entity
	Vector3d position "entity position"
	uint32 i "counter"
	int8 c "char"
end

type SuperBar
  vtable

  uint32			i
  float*			p "ptr"
  bool	  			b "b"
  int8				s "some char"
  Color	  		color "Color of something"
  SuperBar*			psb "Other SuperBar"
  vector<int32>		v "Vector of ints"
end

type Bar : SuperBar
  enum TestEnum
	FIRST = 0,
	SECOND,
	LAST = 10
  end

  float		f
  uint8		c
  uint16		shortArray[10] "array of shorts"
  Vector3d		position "Position"
  Vector3d*		porient "Orientation"
  SuperBar		sb
  SuperBar**	pb
  TestEnum 	en "Some enum"
end

As you can see - it’s pretty close to C/C++ code, but I don’t require semicolons and use slightly different keywords (e.g. I don’t care if it’s a class or structure). Now, this system obviously can’t handle virtual functions (no obvious way to get addresses), the vtable definition is only to preserve compatibility with my test data. This definition file is now parsed by Perl script and result is binary file with type description. It’s almost exactly the same as one generated by PDB reader. The difference is, for every field there’s optional field edit info structure (containing help string + bounds).

Here’s a sample screenshot of a small C# application I coded to test my editor metadata:

As you can see - there’s a ‘help’ status text in the bottom of InstanceEditor window.

Next step would be generating corresponding C/C++ header from our DDL (quite trivial as formats are so similar). Problem is - it’s still additional file that needs to be maintained, PDB solution feels more automatic. Those two could be easily combined - approach #1 mentioned before (Perl loads binary generated by PDB reflector and ‘decorates’ it with metadata). Another nice thing is that metadata is completely optional, it can be easily stripped from the final version of the game, the only cost is pointer per field (=4 bytes). It’d also be very easy to add possibility of actually editing the metadata from game editor (so that you can update field descriptions, modify their limits or flags etc). Storing all information externally really gives plenty of possibilities to update/edit them as we please.

Here you can download my test package containing new reflection reader, Perl script and sample DD file. Obviously, the script is very simple, it handles my test scenario and most of situations I encounter during my experiments, but it’d probably require serious modifications before employing in a professional environment (mostly because it’s been only tested with MSVC, as mentioned in comments for previous articles, GCC may give slightly different layout).

Old comments

NA 2011-01-27 09:29:44

there are some standard methods like encapsulated function ptr to setter/getter.
Using third party tools to parse the code is not fine. IMO.
I always though the game industry used C++/CLI for reflection, because its mostly used for editors…the reflection will not be shipped with the game.

admin 2011-01-26 01:12:50

I’d gladly do that if there were any ‘standard C++ methods’ for reflection. PDB approach is one of the least intrusive I’ve seen. DDL approach is widely used in the gamedev industry, in many “real applications” (==games). That’s my area of interest, I’m not that concerned about other types of apps.

NA 2011-01-25 10:45:17

why are you not trying to implement reflection with standard c++ methods?
So everybody can use it?
You approach ist interesting, but not useable for “real applications”.

admin 2011-01-28 02:09:01

It varies, sometimes it’s editor, sometimes it’s in game as well. One popular approach is to use some macros/template magic, in which case reflection info is part of executable, so it might be not so easy to ship without it. Some parts of reflection, that could be used for load-in-place for example, are pretty useful for final builds as well. From my experience it seems like pre-processing external DDL files is widely accepted in the industry.

BGB 2011-08-12 02:35:36

I am wondering if your PDB reading tool is available for others to use.
If not, I understand.

admin 2011-08-12 02:59:44

It sure is, see my previous articles (links at the very top)