Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

The Case of the Random Internet Explorer Crashes

Posted by William Diaz on July 2, 2010


Some time ago we started to see a rash of complaints where Internet Explorer was crashing while visiting certain sites. Even though we were still largely a mix of IE6 and IE7, I had no problem reproducing the crash with IE8. The error message was rather vague (they often are) and didn’t offer any details as to what was the cause or how to troubleshoot: "IEXPLORE.EXE – Application Error – The Instruction at … referenced memory at … The memory could not be ‘read’":

In other cases, the error encountered was different but still just as vague: “Microsoft Visual C++ Runtime Library – Runtime Error! … R6025 – pure virtual function call"
 
Initial troubleshooting first meant finding a site that would consistently crash IE, and I soon I noticed a trend: web pages that were heavy on Adobe Flash content seemed to be part of the problem, mostly online news websites, and specifically nytimes.com virtually all of the time. With that, working around the issue was rather simple, avoid the problem sites or disable the Adobe Flash component in IE (in IE this is known as the Shockwave Flash Object). Simple but not practical. Try browsing the web with Flash disabled or telling users in our NY office not to visit nytimes. Another approach was to update to latest Adobe Flash Player offering. When this didn’t work, I tried several older versions, but to no avail.

Early in the troubleshooting process I always peer into the Windows Event Viewer to look for clues before going deeper. It was here where I noticed that a network component/protocol driver might be involved, specifically the Windows Proxy Client. Each time IE crashed, the following event was logged: “Application [iexplore.exe]. Authentication was rejected by the WinSock Proxy Server. Check the credentials for the account that is used for the process."

Additionally, I noticed in the system tray that the Windows ISA Firewall Client would indicate a failed authentication:
 
This was my first time troubleshooting a firewall service and admittedly I didn’t know where to begin. Out of desperation, I tried resetting the winsock catalog (cmd > netsh winsock reset catalog). Amazingly, the problem disappeared. And soon I would discover why: a couple days later the test workstation I was using for this project was not able to send software faxes. The fax software relied on the Firewall Client for ISA to transmit. When you run a reset of the winsock catalog, you remove the Base Service Providers that are not already included with Windows, i.e. it ended up removing the BSP for the Microsoft Firewall Client for ISA (otherwise known as the Microsoft Firewall Client Service Provider). If the FWC icon is present in the System Tray, there is sometimes a visual queue that your FWC is no longer functioning properly by the presence of a yellow exclamation over the icon. Since our workstations cannot do without the FWC, I reinstalled it, and with BSPs for the FWC once again in place, the crashing behavior in IE returned1.

My next approach was to see if I could isolate the component or driver that was causing IE to crash. I went to C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson and opened the drwatson32.log file. The file is written to each time the Dr. Watson (otherwise known as the Windows XP post mortem debugger) captures the output of a user-mode crash. User-mode is just another term for an application or service. The system state, various application threads and stacks information are logged here as plain text and can be quickly analyzed to gather important clues on what module was involved in the crash. The log provides a good means of tracking a history of persistent problems and spotting common traits among crashing processes. Newest information is written to the end of the log, and thread and stack information is read from the bottom to the top. There is a lot of information here that looks cryptic, but once you know what you are looking for its easy enough to identify what you are looking for. See this MS TechNet blog for more details on how to interpret these logs.

To start, check to see if the crash was captured: from the bottom of the log Ctrl+F and search for the word application going up. iexplore.exe is the crashing process. Verify by the time stamp that this coincides with the time of crash. You can also see the type of exception, e.g. an access violation in this case, where a program tries to access a part of memory it does not have access to (these are the most common types).

Now search for FAULT going down (check the Upper Case option). Ideally, the FAULT is where we encounter the problem module (usually a .dll or .sys file). If symbols are available, you will see the function in the stack of the problem thread, otherwise you will encounter some hexadecimal string in its place. In the case here, we are dealing with a security component and/or a component that is not part of the Windows out-of-the-box experience so it doesn’t surprise me that the function is not available (you also encounter this with 3rd party modules for which symbols are not usually available):

 
After pouring through several drwtsn32.logs, the common pattern in all the crashes was the presence of wspwsp. A little research reveals this is the WinSockProxy WS2.0 provider DLL that belongs to the Windows ISA firewall client. I was a little surprised by this because statistically 3rd party drivers are usually the cause, not Microsoft. This, coupled with the Event Viewer logs I saw earlier, meant that the likely culprit was, in fact, the FWC.

The first, and most obvious, path was to see if the FWC needed updating. The version of the FWC we were running was already a few years old and since then a new FWC for ISA had been released. I removed the old FWC on a few workstations, installed the latest offering from Microsoft, and the crashing behavior no longer occurred. Eventually, the latest FWC offering was pushed out to all the workstations and we no longer saw the issue.

In addition to the drwtsn32.log, the post mortem debugger generates a user mode dump in the form of a file called user.dmp2 in the same location. Unlike the plain text log, it requires WinDbg from the Windows Debugging Tools to be opened. In short, WinDbg employs heuristics to attempt to identify the cause of a user mode crash. Upon opening the dump, use the !analyze –v command (!=BANG) to start an analysis:

STACK_TEXT: 
0d65fec0 7c901046 00bfe334 5560817a 14bfe334 ntdll!RtlpWaitForCriticalSection+0x8c
0d65fec8 5560817a 14bfe334 14bfe328 5560944d ntdll!RtlEnterCriticalSection+0x46
WARNING: Stack unwind information not available. Following frames may be wrong.
0d65fef8 556072ae 14bfe328 1121e378 14a3f6b8 WSPWSP+0x817a
0d65ff60 71ab4010 71ab4043 71ac4098 1121e470 WSPWSP+0x72ae
0d65ff80 5560838f 00000004 0d55eb78 1121e470 ws2_32!WSACleanup+0x23
0d65ffb4 7c80b729 1121e400 00000004 0d55eb78 WSPWSP+0x838f
0d65ffec 00000000 5560831c 1121e470 00000000 kernel32!BaseThreadStart+0x37

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  WSPWSP+817a

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: WSPWSP

IMAGE_NAME:  WSPWSP.DLL

DEBUG_FLR_IMAGE_TIMESTAMP:  408b776a

STACK_COMMAND:  ~18s; .ecxr ; kb

You can see how the stack is similar to the drwtsn32.log. Clicking the module name hyperlink performs the lmvm command, which provides the details for the module:

0:018> lmvm WSPWSP
start    end        module name
55600000 5561d000   WSPWSP   T (no symbols)          
    Loaded symbol image file: WSPWSP.DLL
    Image path: C:\Program Files\Microsoft Firewall Client\WSPWSP.DLL
    Image name: WSPWSP.DLL
    Timestamp:        Sun Apr 25 04:31:38 2004 (408B776A)
    CheckSum:         000223E8
    ImageSize:        0001D000
    File version:     3.0.1200.365
    Product version:  3.0.1200.365
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Since !analyze –v is heuristics based, it might be worthwhile examining the problem thread in detail for clues to 3rd party modules as well or for more symbolic stack information. Start by dumping all the threads using the ~*kb command and find the thread with module identified heuristically. In this case, thread 33 looks like a match:

0:033> kb
ChildEBP RetAddr  Args to Child             
1105efc8 7c90df5a 7c91b24b 00000c2c 00000000 ntdll!KiFastSystemCallRet
1105efcc 7c91b24b 00000c2c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
1105f054 7c901046 00df3b5c 556096f5 00df3b5c ntdll!RtlpWaitForCriticalSection+0x132
1105f05c 556096f5 00df3b5c fffffffe 06067bc8 ntdll!RtlEnterCriticalSection+0x46
WARNING: Stack unwind information not available. Following frames may be wrong.
1105f090 556106e5 00df3b30 06067bc8 00d83bd0 WSPWSP+0x96f5
1105f12c 71ab32b0 00d83bd0 00000000 1105f1e4 WSPWSP+0x106e5
1105f144 71ab3290 00cd6598 001b0500 00000000 ws2_32!NSPROVIDER::NSPLookupServiceNext+0x17
1105f160 71ab325a 00cd65f0 00000000 1105f1e4 ws2_32!NSPROVIDERSTATE::LookupServiceNext+0x1c
1105f18c 71ab31f8 00cd6d28 00000000 1105f1e4 ws2_32!NSQUERY::LookupServiceNext+0xae
1105f1ac 71ab6136 00cd6d10 00000000 1105f1e4 ws2_32!WSALookupServiceNextW+0x78
1105fa64 71ab5f93 00cd7120 00000002 1105fad8 ws2_32!QueryDnsForFamily+0x1a4
1105fa98 71ab5e58 00cd7120 00000001 1105fad8 ws2_32!QueryDns+0x6c
1105fb04 71ab5ddc 00cd7120 00000000 00000001 ws2_32!LookupAddressForName+0x7e
1105fbf8 71ab2aca 00cd7120 00cd6d60 1105fc5c ws2_32!GetAddrInfoW+0x442
1105fc20 0501ee33 1065a818 05566c80 1105fc5c ws2_32!getaddrinfo+0x72
1105fc94 0501f83b 1065a6c8 000001bb 00000000 Flash10h+0x1aee33
1105fd24 7c9101db 04e9421b 00000013 00000001 Flash10h+0x1af83b
1105ffb4 7c80b729 10c31b90 001ea6a8 0218f0d8 ntdll!RtlAllocateHeap+0xeac
1105ffec 00000000 0501d3e3 10c31b90 00000000 kernel32!BaseThreadStart+0x37

Starting the from bottom working our way up, you can see our original suspect component, Adobe Flash, enters the picture, followed by various functions being called in ws2_32.dll (Win Socket 2.0) before passing into the firewall client (wspwsp.dll) before the thread comes to an end.

Taking it a step further, you can dump the 3rd parameter of the FWC module to reveal what was happening before IE crashed:

0:033> da 00d83bd0
00d83bd0  "video2.nytimes.com"

Sure enough, nytimes.com is where IE kept crashing.

A good strategy when dealing with larger stacks is to start by looking for 3rd party components or Microsoft components that are installed after Windows. Knowing that threads\stacks read from bottom to top, you can then sort of surmise that Flash is is involved but as we move up to the current examine thread that wspwsp.dll plays a role:

Some common commands to employ in WinDbg to assist you:

~*k – view all threads with stacks.

lmvm ComponentName – get info on the component in question (List Modules Verbose Match, I believe):

Check the timestamp for old components that may have updates. This component was actually last modified in 2004 but dates back further:

Since upgrading, wspwsp.dll has been replaced by FwcWsp.dll, dated 2006:

!gle or !gle –allGet Last Error of the current examine thread or all errors from all threads:

Last error for thread 8:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Last error for thread 9:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Last error for thread a:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc00000a3 – {Drive Not Ready}  The drive is not ready for use; its door may be open.  Please check drive %hs and make sure that a disk is inserted and that the drive door is closed.

Last error for thread b:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Last error for thread c:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0150008 – The requested lookup key was not found in any active activation context.

Last error for thread d:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0000034 – Object Name not found.

Last error for thread e:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Last error for thread f:
LastErrorValue: (Win32) 0x2f76 (12150) – <Unable to get error code text>
LastStatusValue: (NTSTATUS) 0xc0000034 – Object Name not found.

Last error for thread 10:
LastErrorValue: (Win32) 0x8 (8) – Not enough storage is available to process this command.
LastStatusValue: (NTSTATUS) 0xc0150008 – The requested lookup key was not found in any active activation context.

Last error for thread 11:
LastErrorValue: (Win32) 0x2f76 (12150) – <Unable to get error code text>
LastStatusValue: (NTSTATUS) 0xc0000034 – Object Name not found.

Last error for thread 12:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc00000a3 – {Drive Not Ready}  The drive is not ready for use; its door may be open.  Please check drive %hs and make sure that a disk is inserted and that the drive door is closed.

Last error for thread 13:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Last error for thread 14:
LastErrorValue: (Win32) 0 (0) – The operation completed successfully.
LastStatusValue: (NTSTATUS) 0 – STATUS_WAIT_0

Note that the output above lists threads in there hexadecimal format, hence why we go from 1-9, then a-f (10-15), then 10-19 (16-25), and so on.

~33s – Change context to thread 33.

Learning to troubleshoot by understanding how to read drwtsn32.logs and using WinDbg opens up a whole new dimension when dealing with the cases of the unexplained.


1 Resetting the BSP did, however, point to a more suitable workaround, which was to instruct users to disable the firewall client while browsing the web since IE was already configured to use a proxy and did not rely on the FWC to handle Internet traffic.

2 I had also filed a bug report with Adobe earlier and while investigating from this angle I discovered another method for capturing Flash related application crashes. For Windows XP, this can be done by going to C:\windows\system32\macromed\flash and opening (or creating the file if not present) mms.cfg with notepad and add the following text: CrashLogEnable=1. The next time Flash Player crashes IE (or any browser for that matter) a minidump will be produced in C:\Documents and Settings\username\Local Settings\Temp named player_crash_log(#).mdmp.

Advertisements

One Response to “The Case of the Random Internet Explorer Crashes”

  1. […] a problem with the LSP application. I encountered this myself way back in my first blog post of the The Case of the Random Internet Explorer Crashes. Share this:EmailPrintDiggFacebookTwitterLinkedInRedditStumbleUponLike this:LikeBe the first to […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: