Know the Stack (or More Hang Analysis Using Process Explorer)
Posted by William Diaz on April 28, 2011
A few moments after opening Outlook, the user complains of unresponsiveness. We start by running Process Explorer. Process Explorer is “self-contained”, so there is no installation required. You can run it directly from SysInternals Live: http://live.sysinternals.com/. I also have it on our lab so I ran it from there:
- Locate the hung process–outlook.exe in this case. Note, it is consuming half of the CPU time.
- Open the properties of the process and go to the Threads tab. You can usually identify the problem thread by the amount of CPU time it is using. You can see this thread is the resource hog.
- Select the thread and then double-click it or click the Stack button. Here we see an important clue, MSPST32.DLL in frames 5,6,7.
What is MSPST32.DLL? If you are not sure, you can find it in C:\Windows\System32. If you are already assuming the issue has to do with a personal folder or the local mailbox ost file, you are right. We are no longer using personal folders in the office of the affected user so I can safely assume it is an ost issue and resolving simply involved creating a new ost.
This is a classic example of using the stack to identify and resolve an otherwise ambiguous application hang … and it only took about 30 seconds to identify.
Here’s a quick rundown of the process tree:
Process. Just an environment where threads execute code. Think of it like a container.
Thread(s). A process must have at least one thread. Threads are responsible for executing code.
Stack. The stack is composed of the module (e.g. a dll or sys) and a function. If you have symbols configured, you can see the function name, e.g. SomeFile.dll!WriteToBlahCacheHaHa. Each line in the stack is known as a frame. Together, the module name and function can be referred to as a subroutine.
Threads and stacks are read from bottom to top.
See a previous blog post here for troubleshooting the same pst-ost corruption but where it manifested itself as a crash and not a hang.
Alternately, if you dump the hanging outlook.exe and examine with WinDbg from the Windows Debugging Tools you see the same module from the stack in the image above using the !analyze –v –hang command:
Note, we can see the function name here, OSTServiceEntry, but did not see it in Process Explorer. This is because Process Explorer was not pointing to the Microsoft Symbol Server and could not resolve the symbol (I don’t bother with this for quick process reviews on user systems), whereas I have WinDbg pointing to it.
To point Process Explorer to the Microsoft Symbol sever, go to Options > Configure Symbols. The Microsoft symbol server address is http://msdl.microsoft.com/download/symbols. To configure a local cache of symbols you could use srv*c:\symbols* http://msdl.microsoft.com/download/symbols, where c:\symbols can be anything you want. You will also need the windbg help dll (dbghelp.dll), however, but don’t need to install the Windows Debugging Tools to get it, just copy it from a system that already have it installed and point Process Explorer to it.
*It doesn’t matter that this is an .ost file, it still relies upon mspst32.dll the same way that a personal folder does.