Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

The Case of the Rebooting Workstation

Posted by William Diaz on June 1, 2012


This was initially described as a log off each time the user opened Outlook. This was the first time I heard of Outlook logging someone off their system when it was opened. This sparked my curiosity, of course, so I asked the techs working on this to leave it alone until I had a chance to look after hours. I started by connecting remotely to the Windows Event Viewer for the problem workstation to see if anything obvious stood out. After about a minute, the MMC console became hung and I could no longer browse events. I thought maybe the workstation became disconnected from the network, so I waited and tried again a few minutes later. I resumed browsing the event logs … only to get disconnected again. Logging in via RDP or VNC was also a no go, as I was getting disconnected after about 2 minutes, barely enough time to get pass the initial desktop and application loading and analyze what was happening. My next approach was to query the workstation for all the running processes via PsList from SysInternals (using the Front End for PsTools); maybe something might stand out and allude to what was happening:

image

There were no odd or otherwise suspicious processes running but at least I knew why I was being disconnected: savedump.exe was a sign that the workstation was recovering from a bug check, the infamous blue screen of death. Knowing that, I UNC’d to C:\Windows\MiniDump on the problem workstation and saw that workstation had been bug checking since that afternoon:

image

I quickly configured the workstation for a kernel dump via remote registry1 (mini dumps are not always enough), waited for it to crash, copied the MEMORY.DMP to my lab workstation that had Windbg from the Windows Debugging Tools installed and opened it:

Loading User Symbols

Loading unloaded module list
…………
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck C, {0, 0, 0, 0}

*** ERROR: Module load completed but symbols could not be loaded for usbhc.sys
Probably caused by : usbhc.sys ( usbhc+6684 )

Followup: MachineOwner

 

If you know Windows and have seen your fair share of kernel dumps, the problem is already alluded to when the dump was opened and the the following was output upon WinDbg trying to find symbols (yes, symbols properly configured) and timestamp information for a system file named usbhc.sys:

*** WARNING: Unable to verify timestamp for usbhc.sys
*** ERROR: Module load completed but symbols could not be loaded for usbhc.sys

If it’s not that easy, then basic crash analysis is as simple as clicking the command hyperlink !analyze –v, confirming this may be the problem module:

DEFAULT_BUCKET_ID:  DRIVER_FAULT

BUGCHECK_STR:  0xC

PROCESS_NAME:  System

LAST_CONTROL_TRANSFER:  from 804fab4b to 804f9f1e

STACK_TEXT: 
aa5b5d18 804fab4b 0000000c 00000000 aa5b5d88 nt!KeBugCheck+0x14
aa5b5d54 aa3bd684 00000042 898d9ef8 00000001 nt!KeWaitForMultipleObjects+0x37
WARNING: Stack unwind information not available. Following frames may be wrong.
aa5b5dac 805cffa8 00000000 00000000 00000000 usbhc+0x6684
aa5b5ddc 8054615e aa3bd6f0 00000000 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND:  kb

FOLLOWUP_IP:
usbhc+6684
aa3bd684 8bf8            mov     edi,eax

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  usbhc+6684

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: usbhc

IMAGE_NAME:  usbhc.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4fbbf736

FAILURE_BUCKET_ID:  0xC_usbhc+6684

BUCKET_ID:  0xC_usbhc+6684

Followup: MachineOwner

 

It seems this module is running out of System process (i.e. kernel-mode). The stack reads from bottom up so after a couple thread start functions, usbhc.sys is encountered and then the bug check ensues, crashing the workstation. At first, I am a bit disappointed because I know there are a number of usb drivers in the system32\drivers folder for Windows and I am thinking that this is not going to be a straight forward crash analysis since we might be dealing with a generic system driver for the usb subsystem. However, the fact that there is no symbol information for what is trying to pass itself off as a system file leaves me suspicious. This is further confirmed by querying the the sys file properties in the dump.

0: kd> lmvm usbhc
start    end        module name
aa3b7000 aa3e0000   usbhc      (no symbols)          
    Loaded symbol image file: usbhc.sys
    Image path: \??\C:\WINDOWS\system32\drivers\usbhc.sys
    Image name: usbhc.sys
    Timestamp:        Tue May 22 16:29:42 2012 (4FBBF736)
    CheckSum:         0000E5A4
    ImageSize:        00029000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

 

Normally, Windows system files or legit third party system drivers are going to provide some details, like the company and description information. For example, looking at the main USB driver for Windows XP:

0: kd> lmvm usbd
start    end        module name
ba5c6000 ba5c7280   USBD       (deferred)            
    Image path: \SystemRoot\system32\DRIVERS\USBD.SYS
    Image name: USBD.SYS
    Timestamp:        Fri Aug 17 17:02:58 2001 (3B7D8682)
    CheckSum:         000040AF
    ImageSize:        00001280
    File version:     5.1.2600.0
    Product version:  5.1.2600.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     usbd.sys
    OriginalFilename: usbd.sys
    ProductVersion:   5.1.2600.0
    FileVersion:      5.1.2600.0 (XPClient.010817-1148)
    FileDescription:  Universal Serial Bus Driver
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

 

Right-clicking the suspicious usbhc.sys file doesn’t reveal any details about it either:

image

Finally, I compare this against the XP lab system I am working from and see there is no such file; the Internet also fails to turn up any information, a sign that this might be a recent piece of malware for which no definition or signatures exist yet. Convinced this is the culprit, I delete the file remotely from the workstation after it recovers from another crash, issue a reboot, and afterwards no more bug checks.

The next part was find out how it was starting. As I mentioned earlier, I did not see any suspicious processes running so it was unlikely running at startup via another malware process. That process had already done its work and would need to be located and examined later. Remember, it had already made itself into the kernel via the System process as the dump pointed out. Looking with Autoruns reveals it was being started as a service:

image

Next, I wanted to see how usbhc.sys found its way into the system. Likely it was the payload delivered by means of an executable that had run earlier. Malware exes are not too difficult to locate. They often reside in the user profile temp folder or some other sub folder within and have what seem like randomly generated alpha-numeric names. Sure enough, within C:\Documents and Settings\username\Local Settings\Temp, there were a few suspicious exes, all with the same creation date:

image

I focused on the larger exe with the icon, K8h0pp.exe and dropped Strings on the workstation. I wanted to see if I could gather some purpose from this exe by dumping any printable ANSI or Unicode characters. Sure enough it looks like this is the package that deliverers the usbhc.sys payload:

image

In fact, you can see here that K8h0pp.exe calls on sc.exe, the Windows service creator utility, to create usbhs.sys as a system service on the workstation. Further down, the missing file properties that made it easy to determine it was not a valid system driver:

image

You might also notice that it requires administrative access in order to run and demonstrates why your users should not be part of the local administrator account. Trying to execute this exe as a Power User, for example, would have failed.

I did the same for the other randomly named .exe files I found. They’re all the same size so likely they all contained the same strings. I am not sure what to make of these but you can see that they are html:

image

Copied into Notepad, saved as html and opened with IE:

image

Perhaps this is part of the malware logging process when it tries to connect to the URL indicated in the message, http://nopwired.info/?…

There also seems to be legit dll involved from Daemon Tools, a CD\DVD emulator and image utility, which seems to have been renamed from DTCommonRes.dll to cleset.dll:

image

I’m not sure what role it might play, but CD\DVD emulators are often used to circumvent copy protection schemes.

As of the first encounter on 5/25/2012, there were no reports of either K8h0pp.exe or usbhc.sys in the wild yet. Later in the weekend, Microsoft Security Essentials scan picked it up while scanning the folder that I dropped it into on my home system. I uploaded both to www.virustotal.com and as of today (5/28/2012) K8h0pp.exe is described as payload of Blackhole exploit kit at hxxp://idyllicpowerups.co.in/w.php?f=01438&e=2, and first seen by VirusTotal on 2012-05-26 09:22:40 UTC. Usbhc.sys was not seen until I uploaded it and is described as a rootkit, but so far only 4 of the 41 major anti-virus suites has picked it up.

The last step was to isolate the exe on a virtual machine2 to see what it does to the system beside creating a service. The tool of choice here is Process Monitor. I’m only interested in seeing what files are being written to the system as well as what reg keys are being created or modified. The quickest way to do this is to use the Tools > Count Occurrence feature and focus on Operations.

First WriteFile:

image

A single file, or payload, is extracted from the executable, usbhc.sys. This makes sense since K8h0pp.exe is 32 KB and its payload, usbhc.sys, is 31 KB. The rest is just code that leverages sc.exe to create usbhc.sys as a system service and its accompanying keys and values:

image

image

This is only a basic analysis that focuses on the workstation. Obviously, there is more at work here and the intent was not to crash the workstation. That was happening because some unknown variable existed on the user’s workstation or is already part of the images we deploy. The blackhole exploit is obviously meant to get control of the system at some later point to either set it up as botnet or remotely access the system for purposes of stealthy information theft. In most malware cases I come across, usually the offender is scareware (fake anti-virus) of some kind. These can usually easily be detected and removed because they want to make their presence known so that you go off and purchase a worthless product. On the other hand, the kind of threat here wants to remain hidden and these you have to worry about because rootkits can easily hide themselves from prying eyes once they get into a system. Even experienced malware fighters can find it hard or impossible to detect them. Because of this, don’t assume 100% success in removal, especially if its new and still running around undetected in the wild. And because 100% removal is not guaranteed, you are better off isolating the infected system from the network and simply reimaging it to be on the safe side.


1 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl. Change CrashDumpEnabled value to 2.

2 Don’t assume a virtual environment will unravel the complete internals of more advanced malware. It might be aware of the virtual environment and avoid running so as to avoid revealing how it works.

Advertisements

2 Responses to “The Case of the Rebooting Workstation”

  1. marek1712 said

    My friend brought his friend’s computer to me today with the same issue.
    Well – I’ve taken quicker route: checked the minidump cause, file’s properties. Detailed properties == none, then rename to .bak.
    It’s quite strange that MSE doesn’t sound the alarm even after manual scan of the usbhc.sys file.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: