The nature of this problem made it difficult or impossible for the help desk to identify because there was nothing to look at that that would tell the technician what was happening when these calls started coming in. They started as a complaint of general system slowness at random times through out the day and were often being assigned to be looked at overnight, which resulted in zero findings because no one knew what they were looking for and could not experience the issue remotely. And if they did, in fact run across the issue while logged on, they could not do anything anyway because the issue of the stalled workstation appeared as a remote connectivity problem and not necessarily a local hardware issue with the workstation.

As I started to here about these issues, I became interested and kept an ear out for a user or two who was encountering the random hang. Identifying a workstation with the problem actually became rather simple because during the hang, a very specific series of events would kick off after the system resumed from the hung state. Isolating the cause, though, was a lot more involved. That’s because the nature of these issues is often software based, e.g. a system or application process was kicking off, or some low level driver was locking up the system. To assist me in that task of finding the culprit, I used a few tools, starting with the Windows XP Event Viewer, then moving to Process Monitor to collect process trace logs, WinDbg to examine manual crash dumps of the hanging system, Performance Monitor, and finally installing Windows 7 after all else failed to take advantage of its enhanced Event Tracing.

Some background. The workstation hangs for the most part coincided with the then recent deployment of new Dell Optiplex 960 and 980 workstations. The hangs were not “hard hangs”, a type of hang where the system becomes completely unresponsive and needs to be manually rebooted. The hangs being seen could be characterized as “soft” in that the workstation would eventually recover after a certain amount of time, usually between 2-5 minutes. During the hang, the mouse was still active but switching between applications was not possible and all keystrokes or commands became queued during the hang. Once the system recovered, any pending operations were executed immediately afterwards. There was no rhyme or reason to the hangs, they were entirely random and would happen several times a day while any user was logged on.

I connected to the workstations after hours and examined the event logs for anything out of the ordinary. Normally, I am looking for error’s or warnings, and I was specifically focused on the System logs, hoping to see disk warnings indicating there were bad blocks on the hard drive. Not seeing anything there, I turned to the application logs but didn’t see anything that stood out there either. Looking at the other workstation, too, did not reveal anything telling.

With nothing to go on, I turned to the generic Information events and noticed that after each reported instance of hang there were a slew of McLogEvent 257 events:

Read the rest of this entry »