I know, its been a while, but good things take their time. We've hit some papers about detecting selfdecrypted shellcodes in network streams

• Advanced Computer Networks Polymorphic Shellcode (Detection)
  
• Analyzing Network Traffic To Detect Self-Decrypting Exploit Code
• Defending against Polymorphic Attacks: Recent Results and Open Questions
• Detecting Network-based Obfuscated Code Injection Attacks Using Sandboxing
• Hybrid Engine for Polymorphic Shellcode Detection
• Network-Level Polymorphic Shellcode Detection Using Emulation

and as nobody wanted to share the code, we wrote our own little x86 cpu emulation, and as reinventing existing wheels is really boring, we threw in api hooking for shellcodes run on our own cpu emulation. The project is called libemu and the plan is to provide a cpu & memory emulation as c library for use in honeypots or ids systems as snort, as you might guess, we are not done yet, but -even though some unwritten internal policy is not to talk about new things in the open public- we thought the current date might be a good choice to spread the word. Whats working so far is, detecting the GetPC code, emulating shellcodes on the cpu, hooking calls to windows dll's, we're missing the backwards traversal to detect required instructions infront of the GetPC, but we're working on it.

Here are two pictures of shellcode execution flows, they are really large, so don't panic if you box starts swapping:

• linkbot bind filetransferr (bremen) (3385 x 6875)
• Metasploit Pex encoded bindshell (4307 x 15323)
• update Agobot CSend (halle) (5363 x 22139)

The pictures were created using our own emulation library and graphviz.
update The Agobot CSend graph is really large, make sure you have 100mb ram free before you open it.