Windows Explored

After moving to a new Internet Explorer based employee time management system, I started noticing a slew of reports coming in where IE was crashing when trying to access any of the menus within the browser. The error always presented itself as: “Microsoft Visual C++ Runtime Library. Runtime Error!..”

I gathered some names of the affected workstations and connected after hours. As I normally do now, I checked to see if the Windows XP post mortem default debugger, Dr Watson, had captured the crash. You can do this by looking in C:\Documents and Settings\All Users\Application Data\Microsoft\Dr Watson for a recently modified drwtsn32.log or user.dmp. However, the log was not be written to and there was no dump of the crashing browser. I would need to manually capture a dump myself.

There are a few different ways to manually capture a dump. Since an error message preceded the process before it crashed, iexplore.exe remained running and I could use SysInternals Process Explorer to dump out the process. In XP, this is quickest, most direct way to perform this as no installation or setup is necessary. To do this, launch Process Explorer, find the process you want to dump, right-click and choose Create Dump:

I copied the dump to my computer and opened with WinDbg from the Windows Debugging Tools and ran the !analyze –v command to start the analysis. Often times crash dumps can point to a false positive. In this case, we actually see the victim (ieui.dll) being identified as the problem module:

To find the real offender, you need to go digging for the exception by looking at the other threads in the dump. This can be accomplished from the command line by dumping all the threads (~*k) or via WinDbg’s GUI. From the GUI, I expand the View menu and select the Process & Threads window along with the Call Stack window. I started scrolling the threads, looking for the words Error, Unhandled, or Exception. A few threads down I found this: 
 
Think of an exception as an error. Well written applications can catch exceptions and proceed normally, handling the error in some manner that is not going to crash the application. On the flipside, think of an UnhandledException as an error that the program has encountered and does not know how to proceed, resulting in a catastrophic failure. Another hint that this is the thread we want to focus on is the presence of the MessageBox function in the user32 module higher in the stack. The call to this function is what relays the error message we see when IE is crashing. You can find the error by looking in one of the parameters of the argument in the frame of user32!MessageBox using the display memory command (d).

da – displays ASCII characters. 

dc will also dump printable characters. Follow with the same command minus the memory address to move to the next block if the message extends into it, using the return key afterwards to keep moving into the next block.

See WinDbg help for different parameters and options available for the display memory command.

However, the call stack window doesn’t reveal the entire stack; I want to see what is happening before we get to the exception. I change the examine thread to thread 5 (~5s) command and then use k50 to show me more than the 20 default number of frames:

Msvcr71 is the Microsoft C Runtime Library. This on its own is not very telling but I see the presence of a couple Java platform modules; the time management application is a Oracle Java based application. After some research, I come across a bug report concerning an earlier version of the Java plugin helper dll for Mozila/Netscape browsers:

"JRE when installed, it adds a Java Plugin 1.6.0 for Netscape Navigator which is a helper DLL npjpi160.dll, This plugin internally loads the DLL file jpiexp.dll which is actual plug-in for Internet Explorer, during loading of this DLL, internet explorer crashes…" "A simple fix to ensure that the msvcr71.dll has been loaded prior to loading the java plugin library – jpiexp.dll. Otherwise, loading of jpiexp.dll fails and we’re trying to invoke a function using an invalid function pointer resulting in a browser crash."

Admittedly, I am a bit perplexed by what this means. Looking at one of the suspect modules, npjpi160_20.dll, I wonder if IE can do without it since it seems to have its purpose with browsers other than IE:

You can see the presence of plugins and their details by going into the Programs tab of the Internet options and selecting Manage Add-ons. On the problem workstations I could see this was the case:

To see what was different between a working and crashing workstation, I looked into the Manage Add-ons window of the workstation that was behaving normally:

Some research reveals that jp2exp.dll registers with IE when the Java option Enable the next-generation Java Plug-in is checked from the Java console. This setting is new to JRE 6 and up. This is set as default when the latest Java client is installed, but in our case it seems that with the presence of the older JREs in place, this setting was not being enabled.

In the end, resolving the the crash simply involved turning on Enable the next-generation Java Plug-in (for the vast majority of cases). This would avoid loading npjpi160_20.dll. In some instances, crashes were still happening and using Process Monitor I could see that there were still file operations being made to npjpi160_20.dll. This was rare but had to be worked around by deleting or renaming npjpi160_20.dll.

It’s worth noting that if you revert to turning off Enable the next-generation Java Plug-in that Java does not reload npjpi160_20.dll but internally loads ssv.dll: