Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Help! Everything Is Crashing

Posted by William Diaz on July 25, 2012


This is an XP workstation so likely the post-mortem default debugger is capturing the exception. I UNC-navigate to \\computername\Documents and Settings\All Users\Application Data\Microsoft\DrWatson. I grab both the drwtsn32.log and user.dmp files. They have recent time stamps of the day before which means that they were likely created as a result of the issue the user was experiencing. I start by examining the log file, starting from the bottom working my way up. The user’s initial complaint was the IE was crashing when going to various websites. I expected to find iexplore.exe process crashing in the log. A few searches in the text file later, I find IE crashing on that day:

Application exception occurred:
        App: C:\Program Files\Internet Explorer\iexplore.exe (pid=6828)
        When: 7/24/2012 @ 11:28:13.701
        Exception number: c0000005 (access violation)

 

Changing my search now down, I look for FAULT:

FAULT ->00001360 ??               ???
Error 0x00000001
        00001362 ??               ???
        00001364 ??               ???
        00001366 ??               ???
        00001368 ??               ???
        0000136a ??               ???
        0000136c ??               ???
        0000136e ??               ???
        00001370 ??               ???
        00001372 ??               ???
        00001374 ??               ???

*—-> Stack Back Trace <—-*
WARNING: Stack unwind information not available. Following frames may be wrong.
ChildEBP RetAddr  Args to Child             
041dfd30 0108cc7c 00cc0048 041dfdac 041dfd68 0x1360
041dfd40 00c848d2 00cc0048 03fa6394 00000064 0x108cc7c
041dfd68 3d942088 00cc0048 03fa6158 00000064 mqtgonce+0x48d2
041dfeb0 3d94732e 00000064 041dfed0 00000008 WININET!CommitUrlCacheEntryA+0x1100
041dfee0 77f69598 00000000 04ab7de0 77f6957b WININET!InternetLockRequestFile+0xfa8
041dfef8 7c92796d 04ab7de0 7c97e460 04ab7d80 SHLWAPI!Ordinal120+0xbf
041dff40 7c9279ab 77f6957b 04ab7de0 00000000 ntdll!RtlGUIDFromString+0x283
041dff60 7c927a6d 00000000 04ab7de0 04ab7d80 ntdll!RtlGUIDFromString+0x2c1
041dff74 7c927a44 7c927991 00000000 04ab7de0 ntdll!RtlGUIDFromString+0x383
041dffb4 7c80b729 00000000 00000000 0309d5d4 ntdll!RtlGUIDFromString+0x35a
041dffec 00000000 7c910250 00000000 00000000 kernel32!GetModuleFileNameA+0x1ba

 

There is a component  that stands out because it’s the last thing IE hits before it crashes and I don’t recognize it. Interestingly, this wasn’t the last application that crashed, though, on this workstation. I confirmed with the user after she elaborated that the whole system was “acting up”. As I search down again for application and FAULT, I see a couple other crashes but no sign of the responsible culprit in the crashing thread:

Application exception occurred:
        App: C:\WINDOWS\system32\msiexec.exe (pid=4588)
        When: 7/24/2012 @ 14:50:49.185
        Exception number: c0000005 (access violation)

FAULT ->715b9e59 ??               ???
Error 0x00000001
        715b9e5b ??               ???
        715b9e5d ??               ???
        715b9e5f ??               ???
        715b9e61 ??               ???
        715b9e63 ??               ???
        715b9e65 ??               ???
        715b9e67 ??               ???
        715b9e69 ??               ???
        715b9e6b ??               ???
        715b9e6d ??               ???

*—-> Stack Back Trace <—-*
WARNING: Stack unwind information not available. Following frames may be wrong.
ChildEBP RetAddr  Args to Child             
00b9ffb4 7c80b729 00a90000 77dd6a4e 0013ffb0 0x715b9e59
00b9ffec 00000000 715b9e59 00a90000 00000000 kernel32!GetModuleFileNameA+0x1ba

Application exception occurred:
        App: C:\WINDOWS\system32\cmd.exe (pid=5444)
        When: 7/24/2012 @ 14:51:31.593
        Exception number: c0000005 (access violation)

FAULT ->715b9e59 ??               ???
Error 0x00000001
        715b9e5b ??               ???
        715b9e5d ??               ???
        715b9e5f ??               ???
        715b9e61 ??               ???
        715b9e63 ??               ???
        715b9e65 ??               ???
        715b9e67 ??               ???
        715b9e69 ??               ???
        715b9e6b ??               ???
        715b9e6d ??               ???

*—-> Stack Back Trace <—-*
WARNING: Stack unwind information not available. Following frames may be wrong.
ChildEBP RetAddr  Args to Child             
00a4ffb4 7c80b729 00940000 00140000 0013ffb0 0x715b9e59
00a4ffec 00000000 715b9e59 00940000 00000000 kernel32!GetModuleFileNameA+0x1ba

 
However, other thread stacks outside of the crashing thread may contain traces of the suspicious component; for example, before the fault:
 

*—-> Stack Back Trace <—-*
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\kernel32.dll –
WARNING: Stack unwind information not available. Following frames may be wrong.
*** WARNING: Unable to verify checksum for C:\WINDOWS\system32\mqtgonce.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\mqtgonce.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\WINDOWS\system32\cmd.exe
ChildEBP RetAddr  Args to Child             
0013f498 7c802455 00000064 00000000 0013f7a0 ntdll!KiFastSystemCallRet
0013f4a8 00ab1273 00000064 00ac4a88 00000001 kernel32!Sleep+0xf
0013f7a0 00ab104b 00000000 4ad02bb4 4ad34400 mqtgonce+0x1273
0013f7b8 4ad031c5 00164450 00162558 00000000 mqtgonce+0x104b
0013f918 4ad02d98 00160720 001685c0 00000000 cmd+0x31c5
0013f94c 4ad02df6 00160720 00000000 00000000 cmd+0x2d98
0013f964 4ad05f20 00160720 00000000 4ad250a0 cmd+0x2df6
0013fb94 4ad013eb 00160720 4ad250a0 00160720 cmd+0x5f20
0013fbd8 4ad06930 00000002 00000001 0015ec48 cmd+0x13eb
0013fc00 4ad06b37 0015f0e0 00153b38 00153b38 cmd+0x6930
0013fc30 4ad07c7e 00153b38 0015ec48 00000000 cmd+0x6b37
0013fc54 4ad02df6 00153b38 00000000 00000000 cmd+0x7c7e
0013fc6c 4ad05f20 00153b38 00000000 00153b38 cmd+0x2df6
0013fe9c 4ad013eb 00153b38 00153b38 00000001 cmd+0x5f20
0013fee0 4ad040d6 00000000 00000001 00000000 cmd+0x13eb
0013ff44 4ad05154 00000006 00033a38 00032e48 cmd+0x40d6
0013ffc0 7c817077 001335d0 00000000 7ffd6000 cmd+0x5154
0013fff0 00000000 4ad05046 00000000 78746341 kernel32!RegisterWaitForInputIdle+0x49

The last dump should also contain clues. Opening the user.dmp with WinDbg and running !analyze –v doesn’t offer much as the stack text is blank and there are no other threads to examine:

STACK_TEXT: 
00000000 00000000 cmd.exe+0x0

0:001> k
ChildEBP RetAddr 
WARNING: Frame IP not in any known module. Following frames may be wrong.
00a4ffb4 7c80b729 0x715b9e59
00a4ffec 00000000 kernel32!BaseThreadStart+0x37

 

Lets see what modules are loaded:

0:001> lm
start    end        module name
00880000 008fa000   CtxSbxHook T (no symbols)          
00ab0000 00ac8000   mqtgonce   (export symbols)       mqtgonce.dll
00ad0000 00ad9000   normaliz T (no symbols)          
10000000 1003e000   mfaphook T (no symbols)          
3d930000 3da16000   wininet  T (pdb symbols)          c:\symbols\wininet.pdb\48E6BE480D2F4000872B105E21351D432\wininet.pdb
3dfd0000 3e1bb000   iertutil T (pdb symbols)          c:\symbols\iertutil.pdb\E5092D04500045A2B7AC326075F153F82\iertutil.pdb
4ad00000 4ad61000   cmd      # (pdb symbols)          c:\symbols\cmd.pdb\6CEF5CCCAD3C4359BE4BFE642184D33E2\cmd.pdb
76390000 763ad000   imm32      (pdb symbols)          c:\symbols\imm32.pdb\F7A5B5DB13324153B57AAF340C77EA512\imm32.pdb
76bf0000 76bfb000   psapi      (pdb symbols)          c:\symbols\psapi.pdb\B9875A55C874489384EA8FB805322C312\psapi.pdb
77120000 771ab000   oleaut32   (pdb symbols)          c:\symbols\oleaut32.pdb\E04ECB48CAED47B2958C3D2C1094E23F2\oleaut32.pdb
773d0000 774d3000   comctl32   (pdb symbols)          c:\symbols\MicrosoftWindowsCommon-Controls-6.0.2600.6028-comctl32.pdb\E882C2C890724D598449E20A4FE6F07C1\MicrosoftWindowsCommon-Controls-6.0.2600.6028-comctl32.pdb
774e0000 7761e000   ole32      (pdb symbols)          c:\symbols\ole32.pdbE73207536D64E9C9FB83C682ED9E5852\ole32.pdb
77b40000 77b62000   apphelp    (pdb symbols)          c:\symbols\apphelp.pdb\67A8FA70E1A74574815714DA307E21BB2\apphelp.pdb
77c00000 77c08000   version    (pdb symbols)          c:\symbols\version.pdb\EA3D1BD3FE65475C8449C8D8B00722962\version.pdb
77c10000 77c68000   msvcrt     (pdb symbols)          c:\symbols\msvcrt.pdb\7BCF30D8C91B4F1B85FA4E55896250111\msvcrt.pdb
77dd0000 77e6b000   advapi32   (pdb symbols)          c:\symbols\advapi32.pdb\F759D3F1C6614313B07C84BC33F02E4D2\advapi32.pdb
77e70000 77f03000   rpcrt4     (pdb symbols)          c:\symbols\rpcrt4.pdb\1A465C67828242F28A8C70E3B9D5C4772\rpcrt4.pdb
77f10000 77f59000   gdi32    # (pdb symbols)          c:\symbols\gdi32.pdb\372C0F0E08FB456EAB7B4CB2B53E27952\gdi32.pdb
77f60000 77fd6000   shlwapi  T (pdb symbols)          c:\symbols\shlwapi.pdb\483E8894476B412DABC2FBA7F470E39A2\shlwapi.pdb
77fe0000 77ff1000   secur32  # (pdb symbols)          c:\symbols\secur32.pdb\7867B3F28B5C41CE847895E3FC013DC52\secur32.pdb
78130000 78263000   urlmon   T (pdb symbols)          c:\symbols\urlmon.pdb\C2AA815B46CE49E582F6E3AC302189992\urlmon.pdb
7c800000 7c8f6000   kernel32   (pdb symbols)          c:\symbols\kernel32.pdb72FF0EB54D24DFAAE9D13885486EE092\kernel32.pdb
7c900000 7c9b2000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\CEFC0863B1F84130A11E0F54180CD21A2\ntdll.pdb
7e410000 7e4a1000   user32     (pdb symbols)          c:\symbols\user32.pdb\D18A41B74E7F458CAAAC1847E2D8BF022\user32.pdb

 

There’s that DLL again. I can’t but find a couple vague references to it on the Internet that show up in a couple Firefox bug reports. I ask the user if she ever had this browser installed. She never has used anything but IE. Lets see if it has any properties that allude to its purpose:

0:001> lmvm mqtgonce
start    end        module name
00ab0000 00ac8000   mqtgonce   (export symbols)       mqtgonce.dll
    Loaded symbol image file: mqtgonce.dll
    Image path: C:\WINDOWS\system32\mqtgonce.dll
    Image name: mqtgonce.dll
    Timestamp:        Mon Jul 23 10:08:40 2012 (500D5AE8)
    CheckSum:         0001A34C
    ImageSize:        00018000
    File version:     6.2.3.7
    Product version:  6.2.3.7
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0
    CompanyName:      FTIP003106976 Air Contractors Engineering Ltd
    ProductName:      Blind, master parallel introduced.
    InternalName:     breathing.dll
    OriginalFilename: breathing.dll
    ProductVersion:   6,2,3,7
    FileVersion:      6,2,3,7
    FileDescription:  Blind, master parallel introduced.
    LegalCopyright:   Copyright (C) 2010

 

This doesn’t look like a legit DLL. The company name doesn’t allude to where it comes from and the Product-File description name doesn’t mean anything. It has a timestamp that coincides with the day of the user complaints. This is beginning to look like a some piece of malware. To confirm, I run Process Explorer on the user’s workstation and do a search for the suspicious DLL. Sure enough, it has, in fact, injected itself into all of the user processes:

image

The next step was to find out how it was starting itself. DLLs themselves cannot run like program exe’s on their own. They need to leverage rundll32.exe. Common places to look for malware startups are the Run keys in HKLM and HKCU, but these turned up nothing. Searching for the actual DLL in the registry, though, did reveal how it was getting injected into the user processes:

image

The DLL loads by creating an entry to its location in HKLM\System\CurrentControlSet\Control\Session Manager\AppCertDlls. This is a less popular but well known key that malware is known to take advantage of, and all processes started by the user will load whatever DLL specified in this key. It also demonstrates why sometimes you need to manually search the registry instead exclusively relying on malware utilities to find them. Even Autoruns would have missed this because it does not search this key.

The intent was not to crash user mode apps. Likely, the other part of the package, an exe, was spotted and dealt with by the anti-virus suite on the workstation, orphaning this DLL. Without its other component, the DLL couldn’t continue on its own and, as a result, processes that were injected were crashing. Resolving was a simple matter of removing the string pointing to the DLL from the registry, rebooting the workstation, and then deleting the DLL.

As for the DLL itself, in addition to code, it is filled with random phrases or words. Here is an excerpt extracted using Strings:

SNAGHTMLc1289d6_thumb[2]

You usually see these kinds of random words and phrases in spam type emails that use this technique to fool spam filters, known as Bayesian poisoning. I wonder if the user was initially hit via an by an email she opened. Upon asking her, she did mention that she was reading her personal web mail and receives an odd message from someone she knew but that, upon asking them, they were not aware of sending the message to her. If that is then likely the sender’s email account has been compromised and was/is being used by a spambot to distribute emails that contain scripts that are designed to drop scareware type malware on the user’s system.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: