Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Hung Up In Outlook

Posted by William Diaz on April 2, 2012


One of the most common complaints your going to encounter in the desktop support role is when Outlook becomes unresponsive while a user is performing any random task in Outlook, whether it be switching between folders, going into a delegated mailbox, or sorting messages. Often, the knee-jerk reaction by the front line technical support is to assume something is wrong with Outlook and begin the gamut of what I term “blind-troubleshooting”. This usually involves running an range of scripted fixes (that is meant to address issues with some of our in-house or 3rd party add-ins), followed by a repair, reinstall, and/or creation of a new Outlook profile. Over the lifetime of this incident(s), the issue can drag out to several days because usually the issue cannot be reproduced immediately after the first fix, so each additional fix is tried at some point later when the user calls back. In some cases, this drags out for a couple weeks to where the user’s workstation is replaced or Windows profile is recreated.

And all to no avail.

Hangs are inevitable. That’s the nature of email in a large enterprise environment. While some hangs can be the result of problems in Outlook (such as a corrupt OST) or with its add-ins, a lot of the time these hangs are the result of Outlook waiting on the Exchange server to process some operation and then return the result. Often, the user might encounter the a warning from the Outlook icon in the system tray that states “Outlook is trying to retrieve data from the Microsoft Exchange Server <Server Name>”. But many times, the user is not going to see this message and assume it’s Outlook, or will simply ignore it, putting you on the spot to explain it. And when this happens, I have seen many a tech go about the gamut of attempted fixes mentioned above in absence of words.

I personally have dealt with many of these requests from certain irate user’s who became special projects of mine. I approach these all the same way: I eliminate unapproved 3rd party add-ins from the equation and then set up a method for gathering process dumps of Outlook when it encounters a hang. The easiest way to do this is to drop Procdump on the user’s workstation along with a small bat file on the desktop that I ask the user to run after they open Outlook, which contains the command to start Procdump and watch Outlook for a hang (e.g. Start c:\sysinternals\procdump -h outlook.exe outlookhang.dmp). When procmon encounters a hung window of 5 seconds, it will dump out the process to a .dmp file that can be examined in WinDbg.

This was the case recently when one of my colleagues was asked to look into a complaint from a user in a remote office about her Outlook getting hung up at random times. I outlined the process above for him and we waited for a few days as dumps were being collected on the user workstation. When all was said and done, I opened the dumps with the Windows Debugging Tools to confirm my suspicion (using the !analyze –v –hang command) that issue was not with Outlook directly but with it waiting on the Exchange server to process some operation:

WAIT_CHAIN_COMMAND:  ~0s;k;;~38s;k;;BLOCKING_THREAD:  00001144DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_RPC

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BlockedOn_RPC

LAST_CONTROL_TRANSFER:  from 7c90df5a to 7c90e514

STACK_TEXT:
0e26f7bc 7c90df5a 7c8025db 000014ad 00000001 ntdll!KiFastSystemCallRet
0e26f7c0 7c8025db 000014ad 00000001 0e26f7f4 ntdll!NtWaitForSingleObject+0xc
0e26f824 77e8090d 000014ad 0000ea60 00000001 kernel32!WaitForSingleObjectEx+0xa8
0e26f840 77e808c3 0022ef6c 00000001 0000ea60 rpcrt4!UTIL_WaitForSyncIO+0x20
0e26f864 77eed64b 0022ef38 0022ef6c 0e26f894 rpcrt4!UTIL_GetOverlappedResultEx+0x1d
0e26f898 77e84df5 00000001 0e26f9d0 0e26f9cc rpcrt4!WS_SyncRecv+0xca
0e26f8b8 77e80cd1 0197bca8 00000084 0e26f9d0 rpcrt4!OSF_CCONNECTION::TransSendReceive+0x9d
0e26f934 77e80c51 80000000 1cd18480 00000001 rpcrt4!OSF_CCONNECTION::SendFragment+0x226
0e26f98c 77e80ab3 00000000 00000058 0e26f9d0 rpcrt4!OSF_CCALL::SendNextFragment+0x1d2
0e26f9d4 77e80a00 0e26fa58 0e26fa14 00000001 rpcrt4!OSF_CCALL::FastSendReceive+0x144
0e26f9f0 77e80f4f 0e26fa58 0e26fa14 0e26fa58 rpcrt4!OSF_CCALL::SendReceiveHelper+0x58
0e26fa1c 77e7a80e 0197bcc0 0e26fa3c 77e7a83f rpcrt4!OSF_CCALL::SendReceive+0x41
0e26fa28 77e7a83f 0e26fa58 35ba4b38 0e26fe38 rpcrt4!I_RpcSendReceive+0x24
0e26fa3c 77ef5675 0e26fa84 0197bd18 00d13d30 rpcrt4!NdrSendReceive+0x2b
0e26fe20 35bb302c 35ba4b38 35ba5002 0e26fe38 rpcrt4!NdrClientCall2+0x222
WARNING: Stack unwind information not available. Following frames may be wrong.
0e26fed0 35fa0dae 0e26fee8 00d15ca0 00d13d30 EMSMDB32!DllCanUnloadNow+0x1c67f
0e26ff3c 35fa0c04 00d13d30 00d15ca0 7c80c1a8 MSMAPI32!Task_RequestCycle+0x5df
0e26ff70 35fa0a9e 00d13d30 09c55d88 00000000 MSMAPI32!Task_RequestCycle+0x435
0e26ffb0 35f768a8 7c80b729 09c55d88 7e41882a MSMAPI32!Task_RequestCycle+0x2cf
0e26ffec 00000000 35f7689f 09c55d88 00000000 MSMAPI32!FOpenThreadImpersonationToken+0x9d2

Most of the dumps were of the same nature. In fact, just to show how common this can be, I reproduced this exact same stack by capturing the hang of another user in a different office complaining of hangs when switching between mailboxes and yet another in yet another office when switching between folders in her mailbox.

You don’t need to be especially adept at reading dumps to understand what is happening here. I’ve been using basic hang analysis of dumps to piece together many hang issues in various apps and don’t have any special training. You would read the stack from the bottom up. You can see MSMAPI (Outlook’s Messaging Application Programming Interface) doing some work, then EMSMDB (the Electronic Messaging System Microsoft Data Base) unloading before various RPC calls take place. In this case, RPC is used for passing some operation to the remote system (the exchange server), where work is waiting to be done. Ideally, upon completion of this work on the remote system, Outlook would then return to a responsive state.

Here is a short description of the remote procedure call:

An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied parameters. The remote server sends a response to the client, and the application continues its process.

Hopefully, this makes your job easier and you can then go off and tell the Network or Exchange admins to start looking at their servers and address the issue there instead of blaming Outlook.

If you want to troubleshoot RPC hangs further, take a look at this MSDN article: Analyzing a Stuck Call Problem.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: