I know, its been a while, but good things take their time.
We've hit some papers about detecting selfdecrypted shellcodes in network streams
and as nobody wanted to share the code, we wrote our own little x86 cpu emulation, and as reinventing existing wheels is really boring, we threw in api hooking for shellcodes run on our own cpu emulation.
The project is called libemu and the plan is to provide a cpu & memory emulation as c library for use in honeypots or ids systems as snort, as you might guess, we are not done yet, but -even though some unwritten internal policy is not to talk about new things in the open public- we thought the current date might be a good choice to spread the word.
Whats working so far is, detecting the GetPC code, emulating shellcodes on the cpu, hooking calls to windows dll's, we're missing the backwards traversal to detect required instructions infront of the GetPC, but we're working on it.
Here are two pictures of shellcode execution flows, they are really large, so don't panic if you box starts swapping:
The pictures were created using our own emulation library and graphviz.
update The Agobot CSend graph is really large, make sure you have 100mb ram free before you open it.