Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Go Digging for the Exception

Posted by William Diaz on July 4, 2010


After moving to a new Internet Explorer based employee time management system, I started noticing a slew of reports coming in where IE was crashing when trying to access any of the menus within the browser. The error always presented itself as: “Microsoft Visual C++ Runtime Library. Runtime Error!..
122910_1941_AnotherCase1

I gathered some names of the affected workstations and connected after hours. As I normally do now, I checked to see if the Windows XP post mortem default debugger, Dr Watson, had captured the crash. You can do this by looking in C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson for a recently modified drwtsn32.log or user.dmp. However, the log was not be written to and there was no dump of the crashing browser. I would need to manually capture a dump myself.

There are a few different ways to manually capture a dump. Since an error message preceded the process before it crashed, iexplore.exe remained running and I could use SysInternals Process Explorer to dump out the process. In XP, this is quickest, most direct way to perform this as no installation or setup is necessary. To do this, launch Process Explorer, find the process you want to dump, right-click and choose Create Dump:

122910_1941_AnotherCase2

I copied the dump to my computer and opened with WinDbg from the Windows Debugging Tools and ran the !analyze –v command to start the analysis. Often times crash dumps can point to a false positive. In this case, we actually see the victim (ieui.dll) being identified as the problem module:

STACK_TEXT: 
0012e698 7c90df2c 7c809574 00000002 0012e6c4 ntdll!KiFastSystemCallRet
0012e69c 7c809574 00000002 0012e6c4 00000001 ntdll!NtWaitForMultipleObjects+0xc
0012e738 7e4195f9 00000002 0012e760 00000000 kernel32!WaitForMultipleObjectsEx+0x12c
0012e794 5dff6029 00000001 0012e7c8 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x13e
0012e7b4 5dff632d 000004ff ffffffff 00000000 ieui!CoreSC::Wait+0x49
0012e7dc 5dff60d8 000004ff 00000000 3e279915 ieui!CoreSC::WaitMessage+0x54
0012e7e8 3e279915 0014f9f0 00151218 00000000 ieui!WaitMessageEx+0x33
0012e818 3e26ab9c 001518a8 0012e848 3e26bc0b ieframe!CBrowserFrame::FrameMessagePump+0x199
0012e824 3e26bc0b 00000000 00000000 0014f9f0 ieframe!BrowserThreadProc+0x3f
0012e848 3e26bb59 1598000a 0014f9f0 00000000 ieframe!BrowserNewThreadProc+0x7b
0012f8b8 3e26ba09 0014f9f0 77f648e4 00000000 ieframe!SHOpenFolderWindow+0x188
0012fae8 00401484 00145158 0000000a 00410070 ieframe!IEWinMain+0x2d9
0012ff2c 0040131f 00400000 00000000 00020880 iexplore!wWinMain+0x2c6
0012ffc0 7c817067 00000000 00000000 7ffdd000 iexplore!_initterm_e+0x1b1
0012fff0 00000000 004031b9 00000000 00000000 kernel32!BaseProcessStart+0x23

STACK_COMMAND:  ~0s; .ecxr ; kb

FOLLOWUP_IP:
ieui!CoreSC::Wait+49
5dff6029 ??              ???

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  ieui!CoreSC::Wait+49

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ieui

IMAGE_NAME:  ieui.dll

To find the real offender, you need to go digging for the exception by looking at the other threads in the dump. This can be accomplished from the command line by dumping all the threads (~*k) or via WinDbg’s GUI. From the GUI, I expand the View menu and select the Process & Threads window along with the Call Stack window. I started scrolling the threads, looking for the words Error, Unhandled, or Exception. A few threads down I found this: 
image 
Think of an exception as an error. Well written applications can catch exceptions and proceed normally, handling the error in some manner that is not going to crash the application. On the flipside, think of an UnhandledException as an error that the program has encountered and does not know how to proceed, resulting in a catastrophic failure. Another hint that this is the thread we want to focus on is the presence of the MessageBox function in the user32 module higher in the stack. The call to this function is what relays the error message we see when IE is crashing. You can find the error by looking in one of the parameters of the argument in the frame of user32!MessageBox using the display memory command (d).

02298e38 7e4664a2 000c0258 046f5e28 03822620 user32!MessageBoxTimeoutW+0x7a
02298e6c 7e450877 000c0258 02298f30 6d6546f0 user32!MessageBoxTimeoutA+0x9c
02298e8c 3e351f02 000c0258 02298f30 6d6546f0 user32!MessageBoxExA+0x1b
02298eac 7e45082f 000c0258 02298f30 6d6546f0 ieframe!Detour_MessageBoxExA+0x2c
02298ec8 6d64f5dc 000c0258 02298f30 6d6546f0 user32!MessageBoxA+0x45

da – displays ASCII characters. 

0:005> da 02298f30
02298f30 "Runtime Error!..Program: C:\Prog"
02298f50 "ram Files\Internet Explorer\IEXP"
02298f70 "LORE.EXE….This application has"
02298f90 " requested the Runtime to termin"
02298fb0 "ate it in an unusual way..Please"
02298fd0 " contact the application’s suppo"
02298ff0 "rt team for more information…"

dc will also dump printable characters. Follow with the same command minus the memory address to move to the next block if the message extends into it, using the return key afterwards to keep moving into the next block.

0:005> dc 02298f30
02298f30 746e7552 20656d69 6f727245 0a0a2172 Runtime Error!..
02298f40 676f7250 3a6d6172 5c3a4320 676f7250 Program: C:\Prog
02298f50 206d6172 656c6946 6e495c73 6e726574 ram Files\Intern
02298f60 45207465 6f6c7078 5c726572 50584549 et Explorer\IEXP
02298f70 45524f4c 4558452e 0a0d0a0a 73696854 LORE.EXE….This
02298f80 70706120 6163696c 6e6f6974 73616820 application has
02298f90 71657220 74736575 74206465 52206568 requested the R
02298fa0 69746e75 7420656d 6574206f 6e696d72 untime to termin
0:005> dc
02298fb0 20657461 69207469 6e61206e 756e7520 ate it in an unu
02298fc0 6c617573 79617720 6c500a2e 65736165 sual way..Please
02298fd0 6e6f6320 74636174 65687420 70706120 contact the app
02298fe0 6163696c 6e6f6974 73207327 6f707075 lication’s suppo
02298ff0 74207472 206d6165 20726f66 65726f6d rt team for more
02299000 666e6920 616d726f 6e6f6974 000a0d2e information….
02299010 00000000 022993f8 6d64c278 505c3a43 ……).x.dmC:\P
02299020 72676f72 46206d61 73656c69 746e495c rogram Files\Int
0:005>
02299030 656e7265 78452074 726f6c70 495c7265 ernet Explorer\I
02299040 4c505845 2e45524f 00455845 1d032258 EXPLORE.EXE.X"..
02299050 00000000 02299060 02299088 05de2da1 ….`.)…)..-..
02299060 1b0cc8b0 1b0cc938 02299068 1d0334b2 ….8…h.)..4..
02299070 02299094 1d086ba0 00000000 1d0334c0 ..)..k…….4..

See WinDbg help for different parameters and options available for the display memory command.

However, the call stack window doesn’t reveal the entire stack; I want to see what is happening before we get to the exception. I change the examine thread to thread 5 (~5s) command and then use k50 to show me more than the 20 default number of frames:

0:005> k20
ChildEBP RetAddr 
02298970 7e419418 ntdll!KiFastSystemCallRet
022989a8 7e4249c4 user32!NtUserWaitMessage+0xc
022989d0 7e43a956 user32!InternalDialogBox+0xd0
02298c90 7e43a2bc user32!SoftModalMessageBox+0x938
02298de0 7e4663fd user32!MessageBoxWorker+0x2ba
02298e38 7e4664a2 user32!MessageBoxTimeoutW+0x7a
02298e6c 7e450877 user32!MessageBoxTimeoutA+0x9c
02298e8c 3e351f02 user32!MessageBoxExA+0x1b
02298eac 7e45082f ieframe!Detour_MessageBoxExA+0x2c
02298ec8 6d64f5dc user32!MessageBoxA+0x45
WARNING: Stack unwind information not available. Following frames may be wrong.
02298efc 6d64dd5f npjpi160_20!NSGetFactory+0x6a63
022990c4 7c912d58 npjpi160_20!NSGetFactory+0x51e6
02299160 7c864031 ntdll!LdrUnlockLoaderLock+0xb1
022993d0 7c83ab38 kernel32!UnhandledExceptionFilter+0x1c7
022993d8 7c839b21 kernel32!BaseThreadStart+0x4d
02299400 7c9032a8 kernel32!_except_handler3+0x61
02299424 7c90327a ntdll!ExecuteHandler2+0x26
022994d4 7c90e46a ntdll!ExecuteHandler+0x24
022994d4 7c812aeb ntdll!KiUserExceptionDispatcher+0xe
0229982c 7c359aed kernel32!RaiseException+0x53
0229986c 6d4a4180 msvcr71!_CxxThrowException+0x34
02299884 6d4a4b03 jpiexp+0x4180
022998ac 6d4a34b7 jpiexp+0x4b03
022998f4 6d4a6a68 jpiexp+0x34b7
02299930 3dcffd15 jpiexp!DllGetClassObject+0x93f
022999b0 3dd01a0c mshtml!COleSite::ConnectControl2+0x86
0229ba30 3dd01eb7 mshtml!COleSite::CreateObjectNow+0x3ba
0229ba54 3dc04231 mshtml!CCodeLoad::OnObjectAvailable+0x84
0229badc 3db0daad mshtml!CCodeLoad::BindToObject+0x4c6
0229bafc 3db0d8d4 mshtml!CCodeLoad::Init+0x296
0229bbb0 3db0d69f mshtml!COleSite::CreateObject+0x5a5
0229fc5c 3db0d3da mshtml!CObjectElement::CreateObject+0x6af

Msvcr71 is the Microsoft C Runtime Library. This on its own is not very telling but I see the presence of a couple Java platform modules; the time management application is a Oracle Java based application. After some research, I come across a bug report concerning an earlier version of the Java plugin helper dll for Mozila/Netscape browsers:

"JRE when installed, it adds a Java Plugin 1.6.0 for Netscape Navigator which is a helper DLL npjpi160.dll, This plugin internally loads the DLL file jpiexp.dll which is actual plug-in for Internet Explorer, during loading of this DLL, internet explorer crashes…" "A simple fix to ensure that the msvcr71.dll has been loaded prior to loading the java plugin library – jpiexp.dll. Otherwise, loading of jpiexp.dll fails and we’re trying to invoke a function using an invalid function pointer resulting in a browser crash."

Admittedly, I am a bit perplexed by what this means. Looking at one of the suspect modules, npjpi160_20.dll, I wonder if IE can do without it since it seems to have its purpose with browsers other than IE:

0:005> lmvm npjpi160_20
start    end        module name
6d640000 6d662000   npjpi160_20   (export symbols)       npjpi160_20.dll
    Loaded symbol image file: npjpi160_20.dll
    Mapped memory image file: C:\Program Files\Java\jre6\bin\npjpi160_20.dll
    Image path: C:\Program Files\Java\jre6\bin\npjpi160_20.dll
    Image name: npjpi160_20.dll
    Timestamp:        Mon Apr 12 21:29:00 2010 (4BC3C8DC)
    CheckSum:         000303E1
    ImageSize:        00022000
    File version:     6.0.200.2
    Product version:  6.0.200.2
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04e4
    CompanyName:      Sun Microsystems, Inc.
    ProductName:      Java(TM) Platform SE 6 U20
    InternalName:     Java(TM) Plug-in
    OriginalFilename: npjpi160_20.dll
    ProductVersion:   6.0.200.2
    FileVersion:      6.0.200.2
    FileDescription:  Classic Java Plug-in 1.6.0_20 for Netscape and Mozilla
    LegalCopyright:   Copyright © 2004

You can see the presence of plugins and their details by going into the Programs tab of the Internet options and selecting Manage Add-ons. On the problem workstations I could see this was the case:
image
To see what was different between a working and crashing workstation, I looked into the Manage Add-ons window of the workstation that was behaving normally:
image
Some research reveals that jp2exp.dll registers with IE when the Java option Enable the next-generation Java Plug-in is checked from the Java console. This setting is new to JRE 6 and up. This is set as default when the latest Java client is installed, but in our case it seems that with the presence of the older JREs in place, this setting was not being enabled.

In the end, resolving the the crash simply involved turning on Enable the next-generation Java Plug-in (for the vast majority of cases). This would avoid loading npjpi160_20.dll. In some instances, crashes were still happening and using Process Monitor I could see that there were still file operations being made to npjpi160_20.dll. This was rare but had to be worked around by deleting or renaming npjpi160_20.dll.

It’s worth noting that if you revert to turning off Enable the next-generation Java Plug-in that Java does not reload npjpi160_20.dll but internally loads ssv.dll:
image

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: