Corsix.org

Above: Company of Heroes: Opposing Fronts running with a prototype overlay rendered on top.
Download video of the overlay in action.

The Europe in Ruins mod for Company of Heroes always presents interesting challenges. At the moment, I'm prototyping an in-game UI for the mod, as administering your army inside the game should be alot more user friendly than doing it on a website. If you're interested in how it's done then carry on reading, otherwise be happy that EiR might one day use an in-game UI.

1. Firstly, a special mod DLL is specify in the EiR.module file. This DLL behaves exactly like the default CoH mod DLL, except it replaces the Lua loadfile function with one that can load DLLs as well as files. This mod DLL is a minimal jailbreak; with it Lua/SCAR can load whatever DLLs they want to, without having to hack or patch any CoH code. The replacement it performs is very simple; the first 5 bytes of the original loadfile function are replaced with a jump to the custom loadfile function.
2. A command typed into the console (dofile [[eir.lua]]) then loads a DLL containing the overlay code along with some auxiliary helper code written in Lua. The overlay DLL patches two things: the DirectXDevice9's EndScene function, and the PeekMessage function.
3. For the EndScene function, it calls Spooge.dll to get a Spooge Device pointer, and then takes the DirectXDevice pointer from the Spooge Device, and then gets a pointer to the DirectXDevice's virtual function pointer table. In this table, it replaces the pointer to the EndScene function with a pointer to our own version of the function (after first recording the address of the original function).
4. For the PeekMessage function, it finds where PeekMessage is called in Platform.dll and replaces those 5 bytes with a call to our own version of the function. The overlay DLL can now draw things on top of CoH by drawing in the custom EndScene function, and can intercept input events by filtering the output from PeekMessage. It exposes to Lua/SCAR simple constructs for drawing and for rendering.
5. The aforementioned Lua auxiliary helper code abstracts the primitive constructs provided by the DLL into a simple API for creating/manipulating windows and other GUI elements like buttons, text boxes and dropdown lists (although only window and button are done at the moment).

Hopefully someday this will come together to allow a full in-game UI for EiR, but at the moment it is still very much in the prototyping stage. Stay tuned!

Edit:
Some people have mentioned that it looks like the Steam in-game UI. As a clarification, this does not use or depend upon any part of Steam. I have been using the Steam UI as a look & feel to aim for (at least for my prototyping), but this just means that the 9 images that are used to stitch together a window graphic are the same colours and whatnot as Steam uses. Again, this will not need Steam, does not use Steam and is not associated with Steam.