Spent some time on writing a libemu module for nepenthes last months, turned out to be rather difficult as nepenthes is a single threaded program and shellcode emulation is slow and may require creating new processes. Installing software with a broken Makefile ran rm -rf / (yes, as root) on my system, therefore this effort got lost anyway.
honeytrap is more likely to get working shellcode emulation than nepenthes, is multiple processes structure fits exactly the needs.
But even if you do not have to worry about creating (sub)processes, emulating shellcodes is not easy.
Most shellcode is written as fire&forget, if the shellcode works, be glad, if it does not work, do not care about the attacked host.
For example relying on returnvalues from system calls to create arguments on the stack, without verification.
In this case, the shellcode relied on the returnvalue of connect, and used it to create the length parameter for the recv syscall.
Click this link to enter the libemu homepage, download your copy, and enjoy the first open source shellcode detection engine using emulation.
It has been a lot of work, it took a lot of time and it is not complete yet, libemu based detection modules for honeytrap, nepenthes and snort are still todo.
In the meantime you can enjoy the great shellcode detection commandline utility sctest to detect and profile shellcodes in suspicious dumps and create graphs of it.
The Internet Stormcenter recently published a call for packets and asked for a shellcode analysis.
We received the dump from the incidents handler William Salusky and here is the libemu result:
the shellcodes graph
The shellcode spawns a command prompt and connects this shell to 18.104.22.168 10000.
While evaluating the problems of shellcode emulation providing proxied api hooks, we came to the conclusion it might be damn dangerous as one could write shellcode which scanning for vulnerable systems to spread.
Just when this was said, we rememberd a worm which sent (and still sends) shellcode to scan for vulnerable system and infect them, the good old sqlslammer, a single udp packet which created a lot of havoc in 2003.
As the worm is plain assembler, and just some hundred bytes long, we decided to give it a ride on libemu.
After faking three IAT entries it worked, and here is the result:
(click for large version)
For those who want to print the graph, here is a vertical version.
Playing with libemu, I stumbled about the missing hook for SetUnhandledExceptionFilter, creating the hook was easy, but for some reason it still did not work when using EXITFUNC=SEH in Metasploit.
The code in question is:
68F08A045F push dword 0x5f048af0
53 push ebx
FFD6 call esi
FFD0 call eax
As I wanted to plot the callgraphs posted previously, I had to shring the graphs and change the orientation from up→down to left→right.
As the left→right graphs got way too large (more than 60000×1800), I had to optimize the graph structure and here is the result:
(click to resize)
Minor differences between the graphs are the result of bugfixing.
I know, its been a while, but good things take their time.
We've hit some papers about detecting selfdecrypted shellcodes in network streams
and as nobody wanted to share the code, we wrote our own little x86 cpu emulation, and as reinventing existing wheels is really boring, we threw in api hooking for shellcodes run on our own cpu emulation.
The project is called libemu and the plan is to provide a cpu & memory emulation as c library for use in honeypots or ids systems as snort, as you might guess, we are not done yet, but -even though some unwritten internal policy is not to talk about new things in the open public- we thought the current date might be a good choice to spread the word.
Whats working so far is, detecting the GetPC code, emulating shellcodes on the cpu, hooking calls to windows dll's, we're missing the backwards traversal to detect required instructions infront of the GetPC, but we're working on it.
Here are two pictures of shellcode execution flows, they are really large, so don't panic if you box starts swapping:
The pictures were created using our own emulation library and graphviz.
update The Agobot CSend graph is really large, make sure you have 100mb ram free before you open it.