Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Posts Tagged ‘Dump’

A Quick One-Dimensional Stack Debug and Solution

Posted by William Diaz on December 17, 2015


I am a big fan of debugging tools, especially WinDbg. These tools can be quite complicated but most of the time I use them simply for a “one-dimensional” analysis of an application or system crash and use the information from the dump to find a cause and hopefully the most logical fix. In the example here, a user complains that PowerPoint-Excel is often crashing or becoming hung. I was able to locate a couple of errors in the Event Viewer > Application Logs that indicated a crash had taken place within the MS Office product, but it was too generic and failed to resolve to the failing component. I asked the user to manually create a crash dump for me the next time the issue happened by using the Task Manager > Applications > right-clicking the unresponsive PowerPoint or Excel application > Select Create Dump File. The dump file gets sent to the user’s temp directory, e.g. C:\Users\username\AppData\Local\Temp\AppName.DMP.

I copied the dmp and opened with with WinDbg. Note, if you are analyzing a 32bit application dump (often the case with MS Office – because who actually uses 64bit office?) you will want to use the 32bit version of WinDbg. Or, if you are incredibly lazy like me (can’t figure out what happened to my x86 version), you can use the 64bit WinDbg and change it use x86-based processor mode. Just run .load wow64exts followed by .effmach x86, followed by the always faithful !analyze –v –hang.

This resulted in following output (Including only the stack text since everything before this is not useful) :

STACK_TEXT: 
00392774 75751605 00000001 003927c4 00000001 ntdll_776f0000!ZwWaitForMultipleObjects+0x15
00392810 769619f8 003927c4 00392838 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
00392858 5e558ac9 00000001 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
WARNING: Stack unwind information not available. Following frames may be wrong.
00392898 5e58f314 00000001 02b68204 ffffffff MSO!Ordinal10580+0x28e
003928d0 5e589c1c 00000000 003928ec 02b69140 MSO!Ordinal4721+0x114
00392900 5e58a80a 02421880 ffffffff 02b68200 MSO!Ordinal5739+0x18a
00392914 2d5b9771 02421880 2d4d05b6 00392948 MSO!Ordinal9231+0x2d6
00392924 2d4d1ebe 00000001 2dc983fc 00000000 PPCORE!DllGetLCID+0xc2eb7
00392948 2d5666de 00000001 84008571 0af19000 PPCORE!PPMain+0x4b010
0039427c 2d5660c3 02b69140 00000425 84009249 PPCORE!DllGetLCID+0x6fe24
00395544 2d563d9f 8400a15d 003966d4 02c40000 PPCORE!DllGetLCID+0x6f809
00396650 2d4a2c41 02c40000 2d4a2be8 02c40000 PPCORE!DllGetLCID+0x6d4e5
00396668 2d5c949b 003966d4 00000000 00000000 PPCORE!PPMain+0x1bd93
00399a64 2d5c90e3 00399a80 84005dbd 0af1928c PPCORE!DllGetLCID+0xd2be1
00399ab0 2d7beff6 0af19000 00399b7c 00399b08 PPCORE!DllGetLCID+0xd2829
0039be34 5f1ea45e 0039bf00 5f1ec17c 76a3030c PPCORE!DllGetLCID+0x2c873c
0039be48 769a03ab 0039bf00 8405ee7f 00000000 MSO!MsoCrashMainThread+0x21f
0039bed0 77763b8f 0039bf00 77763a6c 00000000 kernel32!UnhandledExceptionFilter+0x127
0039bed8 77763a6c 00000000 0039f934 7771c500 ntdll_776f0000!__RtlUserThreadStart+0x62
0039beec 77763911 00000000 00000000 00000000 ntdll_776f0000!_EH4_CallFilterFunc+0x12
0039bf14 777533dd fffffffe 0039f924 0039c050 ntdll_776f0000!_except_handler4+0x8e
0039bf38 777533af 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler2+0x26
0039bf5c 77753350 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler+0x24
0039bfe8 77700133 0139c000 0039c050 0039c000 ntdll_776f0000!RtlDispatchException+0x127
0039bff4 0039c000 0039c050 c0000005 00000000 ntdll_776f0000!KiUserExceptionDispatcher+0xf
0039c4b4 588a6fa9 00000001 00000800 0a346c84 0x39c000
0039c4cc 588a6f52 0a346cac 0039c4e8 5e567954 GFX!Ordinal818+0x1cf
0039c4f4 588a6ea1 00000020 00000800 00000008 GFX!Ordinal818+0x178
0039c5a0 588a6e0a 004066c0 0039c990 00000001 GFX!Ordinal818+0xc7
0039c5cc 51333fb1 0039c644 0039c990 00000001 GFX!Ordinal818+0x30
0039c638 51333f12 00000001 004066c0 0039c990 OART!Ordinal910+0x4f8
0039c668 63818573 0039c8ac 00000000 00000001 OART!Ordinal910+0x459
0039c8d8 6381848e 0039c910 00000000 00000001 RICHED20!RichListBoxWndProc+0xb6d2
0039c918 6aebac5f 05755318 0578b628 00000000 RICHED20!RichListBoxWndProc+0xb5ed
0039c984 6aebadf3 05757e58 00000020 00000000 MSPTLS!LssbFIsSublineEmpty+0xb13c
0039c9a8 6aeb1aea 0000000d 00000004 0039cae8 MSPTLS!LssbFIsSublineEmpty+0xb2d0
0039ca58 6aeb2295 05755f08 0039cd58 0001165b MSPTLS!LssbFIsSublineEmpty+0x1fc7
0039cae0 6aebf7fb 00000000 0039cd58 00000000 MSPTLS!LssbFIsSublineEmpty+0x2772
0039cb44 6aebff10 05755cc8 00000000 0039cd74 MSPTLS!LssbFIsSublineEmpty+0xfcd8
0039cb74 6aebf0cc 00000000 00000000 0c6651f0 MSPTLS!LssbFIsSublineEmpty+0x103ed
0039cd80 6aea4c98 057556e8 00000111 00010dec MSPTLS!LssbFIsSublineEmpty+0xf5a9
0039cdb4 6380f006 057556e8 00000111 00010dec MSPTLS!LsCreateLine+0x23
0039ced4 638005df 00000003 00000000 ffffffff RICHED20!RichListBoxWndProc+0x2165
0039cf2c 638002da 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x65f3
0039cf68 63816f3b 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x62ee
0039cf94 63816aa1 0c694c00 00000111 ffffffff RICHED20!RichListBoxWndProc+0xa09a
0039cfe4 63816987 00000001 ffffffff 00000111 RICHED20!RichListBoxWndProc+0x9c00
0039d060 6af33a5d 057552c0 00000009 00000009 RICHED20!RichListBoxWndProc+0x9ae6
0039d0f0 6af3ec94 0577ffe8 00000009 00000009 MSPTLS!LsLwMultDivR+0x13786
0039d1a8 6af3854d 0577ffe8 00000000 00000000 MSPTLS!LsLwMultDivR+0x1e9bd
0039d264 6af39b13 0c6534f8 00000000 00000000 MSPTLS!LsLwMultDivR+0x18276
0039d388 6af185ed 0039d448 0c6534f8 0039d49c MSPTLS!LsLwMultDivR+0x1983c
0039d404 6af18f8a 0039d448 0c6534f8 0000ad40 MSPTLS!FsTransformBbox+0xf44e
0039d4ec 6af316a1 0577ffe8 00000000 0000ad40 MSPTLS!FsTransformBbox+0xfdeb
0039d55c 6af31913 0c6534f8 00000009 00000004 MSPTLS!LsLwMultDivR+0x113ca
0039d5f0 6af176b9 00000001 0c6534f8 0c652544 MSPTLS!LsLwMultDivR+0x1163c
0039d644 6af0d8a1 0c652528 00000000 00000000 MSPTLS!FsTransformBbox+0xe51a
0039d6d0 6af0decb 00000000 0c6534f8 00000000 MSPTLS!FsTransformBbox+0x4702
0039d79c 6af02099 0577fa10 00000000 0039d874 MSPTLS!FsTransformBbox+0x4d2c
0039d7b4 63814a25 0577fa10 00000000 0039d874 MSPTLS!FsCreatePageBottomless+0x2d
0039d850 6381478b 0039d874 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x7b84
0039d87c 638145ef 057552c0 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x78ea
0039d9bc 63813bac 3fffffff 00400000 0c679bb8 RICHED20!RichListBoxWndProc+0x774e
0039dca0 6380e3b1 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x6d0b
0039dcd0 637fa38d 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x1510
0039dd4c 63812db5 0578eaec 0039dd8c 63812b45 RICHED20!IID_ITextServices2+0x3a1
0039dd58 63812b45 0c67dd40 0039dda0 00000000 RICHED20!RichListBoxWndProc+0x5f14
0039dd8c 51332427 0039ddcc 0e0d0dd0 0e0d0dd8 RICHED20!RichListBoxWndProc+0x5ca4
0039de50 51332095 0050b0d0 0dc6de00 0dc6def8 OART!Ordinal7653+0x502
0039de68 51331e34 0e0d0c40 0039de88 84001d0e OART!Ordinal7653+0x170
0039df84 5132f344 0e0d0c40 0dc6de60 0e0dc0c4 OART!Ordinal525+0xb7b
0039dfc4 51337c8c 0039e058 0039dffc 51337c59 OART!Ordinal6914+0x2c0
0039dfd0 51337c59 0dc6de00 84001d76 0dc6de00 OART!Ordinal3680+0xe8f
0039dffc 5133b804 0dc6de00 840022a2 0dc6de00 OART!Ordinal3680+0xe5c
0039e028 5133b7a6 0dc6de00 840022e6 0dc6de00 OART!Ordinal5150+0x240
0039e06c 513aeed3 0000000e 0039e088 00000000 OART!Ordinal5150+0x1e2
0039e090 513af827 0039e138 840023c2 027fe980 OART!Ordinal389+0x119
0039e148 518017a0 007fe964 0000b298 0039e198 OART!Ordinal1309+0x69d
0039e174 2d803b5a 02808460 0039e198 0039e1a0 OART!Ordinal5501+0x2d
0039e1c8 2d606bb6 02808464 0039e47c 840023a5 PPCORE!DllGetLCID+0x30d2a0
0039e4a8 2d4d2181 02220b44 02808464 0039e4c8 PPCORE!DllGetLCID+0x1102fc
0039e4b8 2d4d1cb4 02220a00 029a5600 0039e4fc PPCORE!PPMain+0x4b2d3
0039e4c8 2d4cf07c 00000004 029a5600 00000000 PPCORE!PPMain+0x4ae06
0039e4fc 2d4cf049 00000004 029a5600 0039e51c PPCORE!PPMain+0x481ce
0039e50c 2d4cf02a 00000004 0285c860 0039e52c PPCORE!PPMain+0x4819b
0039e51c 2d4ced04 00000004 0039e594 0039e564 PPCORE!PPMain+0x4817c
0039e52c 2d49492f 00000004 84002269 2df23040 PPCORE!PPMain+0x47e56
0039e564 2d49385d 2df23040 005849e0 00000000 PPCORE!PPMain+0xda81
0039e578 2d4937ea 0039e594 840022cd 2df23040 PPCORE!PPMain+0xc9af
0039e5c0 2d486f1e 84003f59 2df23040 005849e0 PPCORE!PPMain+0xc93c
0039f854 2df215ae 2df21575 2df20000 00000000 PPCORE!PPMain+0x70
0039f8e8 7696338a 7efde000 0039f934 777297f2 POWERPNT+0x15ae
0039f8f4 777297f2 7efde000 767616c8 00000000 kernel32!BaseThreadInitThunk+0xe
0039f934 777297c5 2df210c8 7efde000 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0039f94c 00000000 2df210c8 7efde000 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

Stacks are read from the bottom up so you can get a very basic idea of what is happening as you climb the ladder. We are really only interested in the component details … and maybe the method-routine of that component (assuming you have the symbols). For example, we start with the main PowerPoint component, PPCORE (everything after the ! is the method or routine). That calls into OART, which is the Office Art components of the MS Office suite, which goes into RICHED20, which I am guessing is something like rich text box component, then to another office component, MSPTLS, and again into another MS Office art component, GFX. After GFX you can see a couple exceptions (errors) are encountered before passing back into the main PowerPoint dll.

Hint: if you are not sure what these components are or belong to, open a new command browser in WinDbg and use the lmvm command to get the details of it, e.g. lmvm GFX:

start             end                 module name
588a0000 58a4d000   GFX        (export symbols)       GFX.DLL
    Loaded symbol image file: GFX.DLL
    Image path: C:\Program Files (x86)\Microsoft Office\Office14\GFX.DLL
    Image name: GFX.DLL
    Timestamp:        Fri Jun 28 18:43:07 2013 (51CE117B)
    CheckSum:         001B6FED
    ImageSize:        001AD000
    File version:     14.0.7104.5000
    Product version:  14.0.7104.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04e4
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft Office 2010
    InternalName:     GFX
    OriginalFilename: GFX.DLL
    ProductVersion:   14.0.7104.5000
    FileVersion:      14.0.7104.5000
    FileDescription:  Microsoft OfficeArt
    LegalCopyright:   © 2010 Microsoft Corporation.  All rights reserved.

 

So, from the stack above, my first guess is that we have an issue with some graphic module(s) – and maybe to a greater part the video subsystem (not necessarily the fault of MS Office). To confirm, I can look at all the thread stacks by dumping them with ~*kb. Reading from the bottom up, it looks like the problem starts with thread 15 (similar to what !analyze –v –hang shows):

15  Id: 944.1ffc Suspend: 0 Teb: 7efd8000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e9ec8c8 7772c81a 00000bd4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0e9ec92c 7772c6fe 00000000 00000000 0000000e ntdll_776f0000!RtlpWaitOnCriticalSection+0x13e
0e9ec954 637f3de0 63913530 05750f1c 637f837e ntdll_776f0000!RtlEnterCriticalSection+0x150
WARNING: Stack unwind information not available. Following frames may be wrong.
0e9ec974 63809412 0000000e ffffffff 057542c8 RICHED20+0x3de0
0e9ec98c 637fa8a3 ffffffff 00000000 0c66df94 RICHED20!IID_ITextServices2+0xf426
0e9eca10 6380a7d3 ffffffff 00000016 0c66df94 RICHED20!IID_ITextServices2+0x8b7
0e9eca38 637f9c7d 00000000 00000016 00000016 RICHED20!IID_ITextServices2+0x107e7
0e9eca5c 637f951a 0e9ecd64 00000000 00000016 RICHED20!IID_ITextHost+0xaf1
0e9ecb20 637ff8c9 00000016 00000016 0e9ece80 RICHED20!IID_ITextHost+0x38e
0e9ecbc4 63806564 00000016 0e9ece80 00000000 RICHED20!IID_ITextServices2+0x58dd
0e9ed000 63805843 0e9ed21c 0e9ed290 0c6660e4 RICHED20!IID_ITextServices2+0xc578
0e9ed1fc 638057bf 00000527 0e9ed22c 0e9ed21c RICHED20!IID_ITextServices2+0xb857
0e9ed240 51330ed8 0e9ed290 009ed274 51330ee8 RICHED20!IID_ITextServices2+0xb7d3
0e9ed254 51330821 0e9ed290 0e9ed274 8aa7103a OART!Ordinal582+0x86a
0e9ed2b0 5133e25c 02ec3c40 00000000 00000017 OART!Ordinal582+0x1b3
0e9ed2f4 5133e0a9 02ec3c40 8aa711ea 00000001 OART!Ordinal3092+0x234
0e9ed360 5133de56 0e9ed43c 8aa71112 00000001 OART!Ordinal3092+0x81
0e9ed398 5156385e 0e9ed43c 8aa71c22 02caed40 OART!Ordinal5804+0x49
0e9edea8 51c8ac06 027fdda8 000da976 001ecfe4 OART!Ordinal3710+0x1f0b
0e9ee010 51c8e83f 0e9ee1ac 0e9ee194 02d47a00 OART!Ordinal3134+0xdad1
0e9ee1e0 51c8af32 0000005e 02d47a00 0e9ee2e4 OART!Ordinal3134+0x1170a
0e9ee1f0 51c84008 0000005e 8aa7206e 0dc531d8 OART!Ordinal3134+0xddfd
0e9ee2e4 51c8477e 029b7e00 0dc53000 00000000 OART!Ordinal3134+0x6ed3
0e9ee338 51c76c9c 00000000 0dc6d000 029b7e00 OART!Ordinal3134+0x7649
0e9ee358 51c53869 00000000 0dc6d000 029b7e00 OART!Ordinal7663+0x111f5
0e9ee388 51c53d8f 029b7e00 0af6a780 00000000 OART!Ordinal5848+0x31bbe
0e9ee46c 51c541f7 01f6a780 00000007 00000000 OART!Ordinal5848+0x320e4
0e9ee494 51c513be 0af6a780 ffffffff 00000001 OART!Ordinal5848+0x3254c
0e9ee510 5166ed08 00000001 023d5100 00000000 OART!Ordinal5848+0x2f713
0e9ee650 5157b3e7 fffffffe 00000000 0dc70500 OART!Ordinal4701+0x7d347
0e9ee6b8 5157af1c 027fdda8 027fdd80 0e9ee798 OART!Ordinal1690+0x6a98
0e9ee750 51565415 0e9ee790 0e9eed80 027fde94 OART!Ordinal1690+0x65cd
0e9ee764 5156534d 00000000 0e9eed80 027fde94 OART!Ordinal2289+0xadd
0e9eebd4 515651db 0e9eed80 00000000 0e9eecf8 OART!Ordinal2289+0xa15
0e9eed2c 51564f8e 003d9fc8 00000000 00000000 OART!Ordinal2289+0x8a3
0e9eeedc 51564b24 0e9eeefc 00000000 8aa72d96 OART!Ordinal2289+0x656
0e9eef1c 51307adf 0e9eefa0 0e9ef050 0e9eefc4 OART!Ordinal2289+0x1ec
0e9eef2c 513077ae 0e9eefa0 8aa72d4e 00000000 OART!Ordinal4963+0x46
0e9eefc4 5132afe4 0e9ef050 0e9ef068 8aa732ea OART!Ordinal2541+0xfd
0e9ef060 5132afe4 02b61720 0e9ef104 8aa73276 OART!Ordinal4433+0x3e9
0e9ef0fc 5132afe4 02b615d0 0e9ef1a0 8aa73312 OART!Ordinal4433+0x3e9
0e9ef198 513075f0 02b613f0 0e9ef1d8 8aa7335a OART!Ordinal4433+0x3e9
0e9ef1d0 51303605 02b6aab0 02de3c84 8aa7308e OART!Ordinal2385+0x142
0e9ef204 513035b2 02a95dc0 00000000 51303401 OART!Ordinal2513+0xde
0e9ef240 2d4d96eb 8aa7342d 0e9ef560 0e9ef4cc OART!Ordinal2513+0x8b
0e9ef320 2d4d941b 03382e44 0e9ef240 af01221d PPCORE!PPMain+0x5283d
0e9ef404 2d4d91af 0e9ef338 0e9ef530 af01221d PPCORE!PPMain+0x5256d
0e9ef554 2d4d8fbc 027fd180 027fd180 00000002 PPCORE!PPMain+0x52301
0e9ef5a4 2d4d8d6f 027fd180 0e9ef578 00000002 PPCORE!PPMain+0x5210e
0e9ef604 2d4d8c64 0e9ef5c8 0e9ef630 02423788 PPCORE!PPMain+0x51ec1
0e9ef644 2d4d8b76 8aa7317d 0ae0ef20 2dcaba78 PPCORE!PPMain+0x51db6
0e9ef670 2d4d8b43 8aa73191 0ae0aa94 0ae0aa84 PPCORE!PPMain+0x51cc8
0e9ef69c 2d48f094 0e9ef6c3 8aa731d9 0e9ef7b8 PPCORE!PPMain+0x51c95
0e9ef6d4 2d48f053 0e9ef720 5e574ea6 0ae0aa80 PPCORE!PPMain+0x81e6
0e9ef6dc 5e574ea6 0ae0aa80 02421880 8a953774 PPCORE!PPMain+0x81a5
0e9ef720 5e556cf0 02421880 0e9ef7b8 02859214 MSO!Ordinal3920+0x30
0e9ef740 5e556be1 0e9ef7b8 00000000 0e9ef79c MSO!Ordinal6535+0x5cb
0e9ef75c 5e553ba1 0e9ef7b8 00000000 003fa7f0 MSO!Ordinal6535+0x4bc
0e9ef790 5e551d40 003fa7f0 00000000 5e551d40 MSO!Ordinal3125+0x1f0
0e9ef7ec 7696338a 003fa7f0 0e9ef838 777297f2 MSO!MsoFRegGetDwCore+0x26f
0e9ef7f8 777297f2 003fa7f0 78d117c4 00000000 kernel32!BaseThreadInitThunk+0xe
0e9ef838 777297c5 5e551ce8 003fa7f0 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0e9ef850 00000000 5e551ce8 003fa7f0 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

 

Immediately after, thread 14 reveals Direct 3D 9 runtime component, which is the core multimedia APIs from Microsoft:

14  Id: 944.218c Suspend: 0 Teb: 7ef12000 Unfrozen
ChildEBP RetAddr  Args to Child             
0fe7f8bc 757514b9 00000ba4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0fe7f928 76961194 00000ba4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0fe7f940 76961148 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0fe7f954 668f801f 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
0fe7f970 668f8073 0fe7f984 7696338a 0eef0040 d3d9!CBatchFilterI::WorkerThread+0x24
0fe7f978 7696338a 0eef0040 0fe7f9c4 777297f2 d3d9!CBatchFilterI::LHBatchWorkerThread+0xd
0fe7f984 777297f2 0eef0040 79a81638 00000000 kernel32!BaseThreadInitThunk+0xe
0fe7f9c4 777297c5 668f8066 0eef0040 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0fe7f9dc 00000000 668f8066 0eef0040 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

 

At this point, I decide to check for a later version of the graphics drivers for the users computer and found a newer driver release. Install, reboot, and no further Office applications reported since. Altogether, the process of identifying the basic issue itself took no more than 10 minutes, all without any software engineer credentials.

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | 2 Comments »

Another Case of IE and Outlook Crashes

Posted by William Diaz on February 26, 2015


I haven’t done one of these in quite awhile. So without further ado.

The symptom: Internet Explorer 11 and Outlook were crashing*. This was happening in all cases after applying a new task sequence. The event viewer did not produce any events that would help so I configured the system to capture crash dumps for applications. After rebooting, I reproduced the crash by simply opening IE an grabbed one of the dumps and ran it through an x86 debugger (by default, the 32bit Content tabs in IE 11 run as 32bit processes). Using ! analyze –v command produced the following output:

STACK_TEXT: 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0339e974 6818a97d 005a09f8 0339e994 5226b9f5 0x490054
0339e9d8 68193c6f 0059f730 0059f7ec 5226bb49 D3D10Level9!UMAdapter::UnderlyingGetCaps+0x31
0339eb64 681940c2 0059f730 0339ebb4 00000002 D3D10Level9!UMAdapter::Open+0x165
0339eb98 69e0cae6 0339ebb4 523e5d23 00020009 D3D10Level9!OpenAdapter10_2+0x45
0339ebdc 69de09c2 005a09f8 0059f35c 69dda858 d3d11!NDXGI::CUMDAdapter::CUMDAdapter+0x18b
0339edc8 69de05fd 0339f0f0 00000002 00000029 d3d11!CCreateDeviceCache::CUMDAdapterCache::Load+0x1fc
0339ee18 69de12f3 0339f0b8 0339ee90 00009300 d3d11!CCreateDeviceCache::CAdapterCache::ResolveUMDAndVersion+0xc8
0339f1f4 69de1e4e 00000000 00595190 00000000 d3d11!D3D11CoreCreateDevice+0x353
0339f484 69de1bdd 00595190 00000000 00000000 d3d11!D3D11CreateDeviceAndSwapChain+0x268
0339f4bc 5bd5b4ed 00595190 00000000 00000000 d3d11!D3D11CreateDevice+0x2c
0339f4f8 5bd5c698 00000001 0339f58c 0339f564 mshtml!CDXGIHelper::CreateD3DDevice+0x48
0339f520 5bd5bc31 72657200 001af5f2 005910fc mshtml!CDXHelper::CreateD3D11Device+0x99
0339f6fc 5bd5ba45 00000000 00000000 00000000 mshtml!CDXResourceDomain::EnsureD3DDevice+0x82
0339f718 5bd5b9ad 00000000 00000000 00000000 mshtml!CDXResourceDomain::Initialize+0x67
0339f734 5bb35771 5c81e03c 00000000 00000000 mshtml!CDXResourceDomain::Create+0x48
0339f76c 5bd5a4c9 0339f78c 00000000 00538c78 mshtml!CDXResourceDomain::EnsureSharedDomain+0xa9
0339f7a0 671015ca 670b21cc 00000000 00000000 mshtml!CDXResourceDomain::EarlyStartDisplaySystem+0xe8
0339f7a4 670b21cc 00000000 00000000 00545cf8 ieframe!DirectUI::TouchEdit2::_UpdatePrompt+0x22
0339f7bc 77647df9 00538c78 744d2d1c 0053b958 ieframe!ExecuteWorkItemThreadProc+0x30
0339f830 77632b65 00538c78 00545cf8 744d2cbc ntdll!RtlpTpWorkCallback+0x11d
0339f990 76d6339a 0053b950 0339f9dc 7761bf32 ntdll!TppWorkerThread+0x572
0339f99c 7761bf32 0053b950 744d2cf0 00000000 kernel32!BaseThreadInitThunk+0xe
0339f9dc 7761bf05 776325c1 0053b950 ffffffff ntdll!__RtlUserThreadStart+0x70
0339f9f4 00000000 776325c1 0053b950 00000000 ntdll!_RtlUserThreadStart+0x1b

STACK_COMMAND:  ~2s; .ecxr ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  unknown!printable+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: unknown

IMAGE_NAME:  unknown

DEBUG_FLR_IMAGE_TIMESTAMP:  0

 

I know D3D is Microsoft Direct X. To validate this, simply run the command lmvm d3d11:

0:002> lmvm d3d11
start    end        module name
69dd0000 69f45000   d3d11      (pdb symbols)          c:\symbols\d3d11.pdb\A40CED7361AB405886384123277BE23F1\d3d11.pdb
    Loaded symbol image file: d3d11.dll
    Image path: C:\Windows\System32\d3d11.dll
    Image name: d3d11.dll
    Timestamp:        Wed Mar 27 18:48:45 2013 (5153774D)
    CheckSum:         0017E157
    ImageSize:        00175000
    File version:     6.2.9200.16570
    Product version:  6.2.9200.16570
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     D3D11.dll
    OriginalFilename: D3D11.dll
    ProductVersion:   6.2.9200.16570
    FileVersion:      6.2.9200.16570 (win8_gdr.130327-1526)
    FileDescription:  Direct3D 11 Runtime
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

 

A good guess is that the issue we are encountering is graphics driver related.

I wanted to see if that was the case with Outlook as well. However, that would require a different approach as Outlook was not producing any crash dumps to run through the debugger. Although there are several ways to work around this, the simplest and most GUI friendly is to use DebugDiag from Microsoft. I wrote about this sometime ago and it has since moved to version 2.1 but the approach I still the same for the most part. I simply ran DebugDiag Collection, created a crash rule, specified outlook.exe as the process. After starting Outlook, several dumps were generated. When complete I then launched DebugDiag Analysis, selected the last dump and then selected Start Analysis. The report generates as an htlm file and opens automatically in IE.

The analysis takes some of the guess work of how to proceed when you have little or no understanding of WinDbg. I could see my suspicion that the issue common to both IE and Outlook was the same, which does not surprise me as they both share common modules.

image

As a proof of concept, one way to test issues you suspect may be graphics related to IE if your system uses dedicated graphics is to disable GPU rendering in IE. This is done by going into the Control Panel > Internet Options > Advanced > and checking Use software rendering instead of GPU rendering (note, although the asterisk indicates you should restart your computer, only restarting IE may be required). In my case, this corrected the problem with IE so I was sure now that the issue was related to the graphics card. Before updating the driver, I checked the display information to see if perhaps the drivers were missed and maybe generic drivers were installed. This didn’t seem to be the case:

image

I would still update the drivers anyway. After doing so, both Internet Explorer and Outlook opened without crashing. Interestingly, after updating the drivers, the workstation reported a different video adapter model:

image

In the end, the resolution was to install the correct graphics drivers during the task sequence.

Note, in some cases the crashing application may report via WER (windows error report) popup. If you examine the details, you may note that the fault module name contains a stackhash code a7aa, which is often a indicator that the issue is likely related to a driver issue for the video subsystem:

SNAGHTML31458dc0


*Oddly enough, I could not reproduce the issue when connected remotely using RDP. This should have been a clear indicator as to what the issue really was because RDP does not use the host system’s graphics drivers but those of the connecting system instead. 

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | Leave a Comment »

Help! Everything Is Crashing

Posted by William Diaz on July 25, 2012


This is an XP workstation so likely the post-mortem default debugger is capturing the exception. I UNC-navigate to \\computername\Documents and Settings\All Users\Application Data\Microsoft\DrWatson. I grab both the drwtsn32.log and user.dmp files. They have recent time stamps of the day before which means that they were likely created as a result of the issue the user was experiencing. I start by examining the log file, starting from the bottom working my way up. The user’s initial complaint was the IE was crashing when going to various websites. I expected to find iexplore.exe process crashing in the log. A few searches in the text file later, I find IE crashing on that day:

Application exception occurred:
        App: C:\Program Files\Internet Explorer\iexplore.exe (pid=6828)
        When: 7/24/2012 @ 11:28:13.701
        Exception number: c0000005 (access violation)

Read the rest of this entry »

Posted in Troubleshooting | Tagged: , , , , | Leave a Comment »

Get Last Error

Posted by William Diaz on July 14, 2012


Often times when doing some basic crash or hang analysis on a program, !analyze –v is not going to cut it because the heuristics engine is not going to reveal an interesting stack. Or maybe I don’t know what I’m looking for. Or maybe I don’t know advanced WinDbg debug techniques. Or … whatever. I define an “interesting” stack is one that contains unexpected components. If I don’t see it, I usually move on to some other techniques.

One of the things I try when a dump has nothing to offer me is to just take a look at the last error thrown. To do this, simply employ the get last error command, !gle. For example, an Outlook crash (which can be notoriously difficult to analyze even for the advanced Windbg enthusiast) I was asked to examine where the !analyze –v heuristics engine wasn’t telling me anything meaningful (at least to me) and where !gle might help:

Read the rest of this entry »

Posted in Troubleshooting | Tagged: , , | Leave a Comment »

The Case of the Rebooting Workstation

Posted by William Diaz on June 1, 2012


This was initially described as a log off each time the user opened Outlook. This was the first time I heard of Outlook logging someone off their system when it was opened. This sparked my curiosity, of course, so I asked the techs working on this to leave it alone until I had a chance to look after hours. I started by connecting remotely to the Windows Event Viewer for the problem workstation to see if anything obvious stood out. After about a minute, the MMC console became hung and I could no longer browse events. I thought maybe the workstation became disconnected from the network, so I waited and tried again a few minutes later. I resumed browsing the event logs … only to get disconnected again. Logging in via RDP or VNC was also a no go, as I was getting disconnected after about 2 minutes, barely enough time to get pass the initial desktop and application loading and analyze what was happening. My next approach was to query the workstation for all the running processes via PsList from SysInternals (using the Front End for PsTools); maybe something might stand out and allude to what was happening:

image

Read the rest of this entry »

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , , , | 2 Comments »

Wife.exe Crash Troubleshooting

Posted by William Diaz on May 18, 2012


Not all issues I troubleshoot are at work. Sometimes I might be sitting at home and then this somewhat buggy application starts up. It’s a random occurrence and trying to resolve its problems can be very taxing. The main reason for this is I don’t have the source code and trying to reverse-engineer it is nearly impossible because it’s data structures are not logical. But maybe we can look at its crash dump and find out what’s happening.

Looking at the dump:

Read the rest of this entry »

Posted in Troubleshooting | Tagged: , , | 2 Comments »

The Case of the Runtime Error (or Check Your Time Zones & DST Settings)

Posted by William Diaz on May 7, 2012


One of our helpdesk technician’s in a remote office reached out to me recently and asked me to assist with an application that suddenly started crashing on him with the following error: “Microsoft Visual C++ Runtime Library. Runtime Error! This application has requested the Runtime to terminate it in an unusual way…”
image

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , , | 1 Comment »

Troubleshooting a “Hard Hang”

Posted by William Diaz on May 7, 2012


I recently put together a quad core system from parts my brother was retiring from his home system. Soon after getting everything up and running, the new system would sometimes hard hang while working via VPN. A hard hang is when the OS becomes completely unresponsive but is still running. I had earlier prepared myself for the next instance of this encounter by enabling CrashOnCtrlScroll in the Windows registry so that the next time it happened I could manually crash the system from the keyboard and examine the memory dump with WinDbg for signs of the responsible culprit. You can read about how to enable this option in this earlier blog Forcing a System Crash on an Unresponsive PC.

Upon opening a kernel dump, the analyze –v command is a clickable hyperlink1.

Posted in Troubleshooting | Tagged: , , , | 1 Comment »

Recovering Text From A Unresponsive Application

Posted by William Diaz on May 2, 2012


A couple weeks ago a co-worker asked me if it was possible to recover text from a hung application. The user had spent a good amount of time typing into text field and upon trying to submit the information, IE became hung and would not recover. The user didn’t want to have to retype everything from scratch again. The short answer to this is yes. The long answer is “Yes, if you can be patient.” I have never actually needed to perform this myself because it’s not always that practical and, to be honest, the task can be somewhat tedious. But if you really need to recover text and Word didn’t auto-recover or Outlook lost the draft after you clicked send, or IE is in the process of a “GUI crash”, then turn to the power of the dump.

The idea here is if the application is still running but stalled, it still resides in memory, along with anything you typed into it. When you dump the process, you are dumping its presence in memory to a file that you can pick apart. Dumping a hung process is simple enough. On an XP system, open Process Explorer, right-click the process, and choose Create Dump. In Vista & Windows 7, this option is now built into the Task Manager. You can then copy the dump to a system where WinDbg is installed to open it.

Posted in Troubleshooting Tools | Tagged: , , | Leave a Comment »

Getting the 32 Bit Stacks in a 64 Bit Dump

Posted by William Diaz on February 8, 2012


After loading up Word today, I noticed it would become hung each time I went to the Insert tab. I was able to reproduce the behavior after killing the process and starting it up again. To troubleshoot, I started by going to the Windows 7 Task Manager, right-clicking, and selecting Create Dump File from the context menu:
Read the rest of this entry »

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , , | 1 Comment »