Windows Explored

This case demonstrates the importance of understanding basic networking components in Windows and how they work in an environment that sits behind a proxy. The methods used to uncover the culprit did not involve a detailed analysis of network traffic or using any utilities or tools that do not already come with Windows. In fact, because this was time sensitive and I was not afforded the luxury of troubleshooting afterhours, I could not rely on any of the various tools I turn to, like Process Monitor, crash dumps, or Network Monitor. Previous troubleshooting steps taken before it got me involved timely uninstall and reinstall of the Java client, an IE Reset, and upgrading from IE 7 to IE 8, all to no avail and creating irate user syndrome.

To start my analysis, I used Process Explorer to easily see which processes were involved in starting the problem application (Task Manager makes it difficult to quickly see which processes are starting and stopping). An Internet Explorer shortcut to a login page actually initiates the launching of the stock trading application, and, after logging in, two processes are spawned, javaws.exe and javaw.exe:

At this point, the two java processes would run for about a minute or so, then present a “Loading Application” splash screen … and then abruptly close, taking the user back to another login URL (insert expletives from the user here). I didn’t know if both processes were shutting down or crashing. I didn’t see the post-mortem Windows XP debugger, Dr Watson, capturing an error in its drwtsn32.log or a user-mode dump. Additionally the Debug Diagnostics service did not detect a crash for either javaws.exe or javaw.exe when attached.

I needed some information that could help me dissect what was happening here. For that, I turned to the Java Control Panel (you can find it in the Windows Control Panel) and selected the radio option for Show Console; the default is to hide the console:

With this turned on, whenever a Java application loads, this console will open, too. Unless you know Java (which I do not) this can be difficult to decrypt. Luckily, the console pointed me to a couple log files:

Log files are often overlooked as sources of troubleshooting application failures. However, I have used them successfully in the past and have blogged about them on a couple occasions here and here. I went to c:\data\logs and opened the user friendly client-log.html. Here is an excerpt of the log file, with the relevant part highlighted:

Further down follows a summary of the Java Exception, where the connection is being dropped:

The application is trying to communicate over SSL (secure socket layer) port 443 but failing. This is the default port for SSL so it is not likely being blocked by our network. The problem seems to stem from a Winsock issue (Windows Socket); the error “Unable to initialize socket” is quite telling, assuming you have a basic grasp of how applications need to communicate from behind a proxy. In short, Winsock specifies how network applications access network services, e.g. TCP/IP. Applications that cannot be internally configured to point to a proxy to get to the outside rely on some other means to find their way out of the network. This would be some kind of Layered Service Provider, and in our network this is the Microsoft Firewall Client for ISA.

Knowing this, I decided to look at the catalog of base service providers on this workstation to see if the Microsoft Firewall Client was listed¹. To do this, you can run netsh winsock show catalog >C:\bsp.txt from the command shell. This will output the provider list to a text file called bsp.txt in the root of C:. A normal workstation in our network (on the left) will contain a few entries for the Microsoft Firewall Client Service Provider in the catalog. Our user’s workstation (on the right) is missing all the same entries:

To verify the functionality of the fwc, I opened the fwc management console from the Programs folder (since this starts up as hidden from the Windows System Tray in our environment). I did not need to look further to realize that, in fact, the MS firewall client for ISA was not working:

Another way of verifying that the fwc is working is to disable the Proxy Server in Internet Explorer. If the FWC for ISA is functioning, then it will handle url requests that go outside of the network. If it has failed, then Internet requests will fail.

To correct, we simply needed to repair or reinstall the fwc. Why this happened here remains unknown. If it happens consistently, then you need to investigate what other Winsock providers are being installed that could be breaking the functionality of the FWC². By default, Windows XP comes with 10 pre-defined Winsock Base Service Providers; see this Microsoft KB article for further details and how to correct a corrupted Winsock. Known offender BSPs can be installed by 3rd party communication apps like AT&T Communication Manager and Vodafone Air Card software:

Removing these from Add or Remove Programs may fix the broken functionality of the fwc for ISA (a repair of the fwc may be required as well).

In some cases, the presence of some BSPs may cause the FWC to incorrectly report the “Firewall Client is not installed properly”. See here: http://support.microsoft.com/kb/936773

¹ If you are GUI oriented, you can also use msinfo32 to view the list of Winsock Base Providers under Components > Network > Protocol:

² You can manually break the FWC by running the following command netsh winsock reset catalog for testing purposes.