The Case of the Rebooting Workstation
Posted by William Diaz on June 1, 2012
This was initially described as a log off each time the user opened Outlook. This was the first time I heard of Outlook logging someone off their system when it was opened. This sparked my curiosity, of course, so I asked the techs working on this to leave it alone until I had a chance to look after hours. I started by connecting remotely to the Windows Event Viewer for the problem workstation to see if anything obvious stood out. After about a minute, the MMC console became hung and I could no longer browse events. I thought maybe the workstation became disconnected from the network, so I waited and tried again a few minutes later. I resumed browsing the event logs … only to get disconnected again. Logging in via RDP or VNC was also a no go, as I was getting disconnected after about 2 minutes, barely enough time to get pass the initial desktop and application loading and analyze what was happening. My next approach was to query the workstation for all the running processes via PsList from SysInternals (using the Front End for PsTools); maybe something might stand out and allude to what was happening:
There were no odd or otherwise suspicious processes running but at least I knew why I was being disconnected: savedump.exe was a sign that the workstation was recovering from a bug check, the infamous blue screen of death. Knowing that, I UNC’d to C:\Windows\MiniDump on the problem workstation and saw that workstation had been bug checking since that afternoon:
I quickly configured the workstation for a kernel dump via remote registry1 (mini dumps are not always enough), waited for it to crash, copied the MEMORY.DMP to my lab workstation that had Windbg from the Windows Debugging Tools installed and opened it:
Loading User Symbols Loading unloaded module list Use !analyze -v to get detailed debugging information. BugCheck C, {0, 0, 0, 0} *** ERROR: Module load completed but symbols could not be loaded for usbhc.sys Followup: MachineOwner |
If you know Windows and have seen your fair share of kernel dumps, the problem is already alluded to when the dump was opened and the the following was output upon WinDbg trying to find symbols (yes, symbols properly configured) and timestamp information for a system file named usbhc.sys:
*** WARNING: Unable to verify timestamp for usbhc.sys
*** ERROR: Module load completed but symbols could not be loaded for usbhc.sys
If it’s not that easy, then basic crash analysis is as simple as clicking the command hyperlink !analyze –v, confirming this may be the problem module:
DEFAULT_BUCKET_ID: DRIVER_FAULT BUGCHECK_STR: 0xC PROCESS_NAME: System LAST_CONTROL_TRANSFER: from 804fab4b to 804f9f1e STACK_TEXT: STACK_COMMAND: kb FOLLOWUP_IP: SYMBOL_STACK_INDEX: 2 SYMBOL_NAME: usbhc+6684 FOLLOWUP_NAME: MachineOwner MODULE_NAME: usbhc IMAGE_NAME: usbhc.sys DEBUG_FLR_IMAGE_TIMESTAMP: 4fbbf736 FAILURE_BUCKET_ID: 0xC_usbhc+6684 BUCKET_ID: 0xC_usbhc+6684 Followup: MachineOwner |
It seems this module is running out of System process (i.e. kernel-mode). The stack reads from bottom up so after a couple thread start functions, usbhc.sys is encountered and then the bug check ensues, crashing the workstation. At first, I am a bit disappointed because I know there are a number of usb drivers in the system32\drivers folder for Windows and I am thinking that this is not going to be a straight forward crash analysis since we might be dealing with a generic system driver for the usb subsystem. However, the fact that there is no symbol information for what is trying to pass itself off as a system file leaves me suspicious. This is further confirmed by querying the the sys file properties in the dump.
0: kd> lmvm usbhc |
Normally, Windows system files or legit third party system drivers are going to provide some details, like the company and description information. For example, looking at the main USB driver for Windows XP:
0: kd> lmvm usbd |
Right-clicking the suspicious usbhc.sys file doesn’t reveal any details about it either:
Finally, I compare this against the XP lab system I am working from and see there is no such file; the Internet also fails to turn up any information, a sign that this might be a recent piece of malware for which no definition or signatures exist yet. Convinced this is the culprit, I delete the file remotely from the workstation after it recovers from another crash, issue a reboot, and afterwards no more bug checks.
The next part was find out how it was starting. As I mentioned earlier, I did not see any suspicious processes running so it was unlikely running at startup via another malware process. That process had already done its work and would need to be located and examined later. Remember, it had already made itself into the kernel via the System process as the dump pointed out. Looking with Autoruns reveals it was being started as a service:
Next, I wanted to see how usbhc.sys found its way into the system. Likely it was the payload delivered by means of an executable that had run earlier. Malware exes are not too difficult to locate. They often reside in the user profile temp folder or some other sub folder within and have what seem like randomly generated alpha-numeric names. Sure enough, within C:\Documents and Settings\username\Local Settings\Temp, there were a few suspicious exes, all with the same creation date:
I focused on the larger exe with the icon, K8h0pp.exe and dropped Strings on the workstation. I wanted to see if I could gather some purpose from this exe by dumping any printable ANSI or Unicode characters. Sure enough it looks like this is the package that deliverers the usbhc.sys payload:
In fact, you can see here that K8h0pp.exe calls on sc.exe, the Windows service creator utility, to create usbhs.sys as a system service on the workstation. Further down, the missing file properties that made it easy to determine it was not a valid system driver:
You might also notice that it requires administrative access in order to run and demonstrates why your users should not be part of the local administrator account. Trying to execute this exe as a Power User, for example, would have failed.
I did the same for the other randomly named .exe files I found. They’re all the same size so likely they all contained the same strings. I am not sure what to make of these but you can see that they are html:
Copied into Notepad, saved as html and opened with IE:
Perhaps this is part of the malware logging process when it tries to connect to the URL indicated in the message, http://nopwired.info/?…
There also seems to be legit dll involved from Daemon Tools, a CD\DVD emulator and image utility, which seems to have been renamed from DTCommonRes.dll to cleset.dll:
I’m not sure what role it might play, but CD\DVD emulators are often used to circumvent copy protection schemes.
As of the first encounter on 5/25/2012, there were no reports of either K8h0pp.exe or usbhc.sys in the wild yet. Later in the weekend, Microsoft Security Essentials scan picked it up while scanning the folder that I dropped it into on my home system. I uploaded both to www.virustotal.com and as of today (5/28/2012) K8h0pp.exe is described as payload of Blackhole exploit kit at hxxp://idyllicpowerups.co.in/w.php?f=01438&e=2, and first seen by VirusTotal on 2012-05-26 09:22:40 UTC. Usbhc.sys was not seen until I uploaded it and is described as a rootkit, but so far only 4 of the 41 major anti-virus suites has picked it up.
The last step was to isolate the exe on a virtual machine2 to see what it does to the system beside creating a service. The tool of choice here is Process Monitor. I’m only interested in seeing what files are being written to the system as well as what reg keys are being created or modified. The quickest way to do this is to use the Tools > Count Occurrence feature and focus on Operations.
First WriteFile:
A single file, or payload, is extracted from the executable, usbhc.sys. This makes sense since K8h0pp.exe is 32 KB and its payload, usbhc.sys, is 31 KB. The rest is just code that leverages sc.exe to create usbhc.sys as a system service and its accompanying keys and values:
This is only a basic analysis that focuses on the workstation. Obviously, there is more at work here and the intent was not to crash the workstation. That was happening because some unknown variable existed on the user’s workstation or is already part of the images we deploy. The blackhole exploit is obviously meant to get control of the system at some later point to either set it up as botnet or remotely access the system for purposes of stealthy information theft. In most malware cases I come across, usually the offender is scareware (fake anti-virus) of some kind. These can usually easily be detected and removed because they want to make their presence known so that you go off and purchase a worthless product. On the other hand, the kind of threat here wants to remain hidden and these you have to worry about because rootkits can easily hide themselves from prying eyes once they get into a system. Even experienced malware fighters can find it hard or impossible to detect them. Because of this, don’t assume 100% success in removal, especially if its new and still running around undetected in the wild. And because 100% removal is not guaranteed, you are better off isolating the infected system from the network and simply reimaging it to be on the safe side.
1 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl. Change CrashDumpEnabled value to 2.
2 Don’t assume a virtual environment will unravel the complete internals of more advanced malware. It might be aware of the virtual environment and avoid running so as to avoid revealing how it works.
marek1712 said
My friend brought his friend’s computer to me today with the same issue.
Well – I’ve taken quicker route: checked the minidump cause, file’s properties. Detailed properties == none, then rename to .bak.
It’s quite strange that MSE doesn’t sound the alarm even after manual scan of the usbhc.sys file.
William Diaz said
New file properties perhaps? MSE definitions not up to date? Since encountering, I came across this http://artemonsecurity.blogspot.com/2012/07/investigation-interesting-kernel-mode.html#!/2012/07/investigation-interesting-kernel-mode.html so it looks like MSE has a definition for it.