Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Archive for December, 2015

A Quick One-Dimensional Stack Debug and Solution

Posted by William Diaz on December 17, 2015

I am a big fan of debugging tools, especially WinDbg. These tools can be quite complicated but most of the time I use them simply for a “one-dimensional” analysis of an application or system crash and use the information from the dump to find a cause and hopefully the most logical fix. In the example here, a user complains that PowerPoint-Excel is often crashing or becoming hung. I was able to locate a couple of errors in the Event Viewer > Application Logs that indicated a crash had taken place within the MS Office product, but it was too generic and failed to resolve to the failing component. I asked the user to manually create a crash dump for me the next time the issue happened by using the Task Manager > Applications > right-clicking the unresponsive PowerPoint or Excel application > Select Create Dump File. The dump file gets sent to the user’s temp directory, e.g. C:\Users\username\AppData\Local\Temp\AppName.DMP.

I copied the dmp and opened with with WinDbg. Note, if you are analyzing a 32bit application dump (often the case with MS Office – because who actually uses 64bit office?) you will want to use the 32bit version of WinDbg. Or, if you are incredibly lazy like me (can’t figure out what happened to my x86 version), you can use the 64bit WinDbg and change it use x86-based processor mode. Just run .load wow64exts followed by .effmach x86, followed by the always faithful !analyze –v –hang.

This resulted in following output (Including only the stack text since everything before this is not useful) :

00392774 75751605 00000001 003927c4 00000001 ntdll_776f0000!ZwWaitForMultipleObjects+0x15
00392810 769619f8 003927c4 00392838 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
00392858 5e558ac9 00000001 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
WARNING: Stack unwind information not available. Following frames may be wrong.
00392898 5e58f314 00000001 02b68204 ffffffff MSO!Ordinal10580+0x28e
003928d0 5e589c1c 00000000 003928ec 02b69140 MSO!Ordinal4721+0x114
00392900 5e58a80a 02421880 ffffffff 02b68200 MSO!Ordinal5739+0x18a
00392914 2d5b9771 02421880 2d4d05b6 00392948 MSO!Ordinal9231+0x2d6
00392924 2d4d1ebe 00000001 2dc983fc 00000000 PPCORE!DllGetLCID+0xc2eb7
00392948 2d5666de 00000001 84008571 0af19000 PPCORE!PPMain+0x4b010
0039427c 2d5660c3 02b69140 00000425 84009249 PPCORE!DllGetLCID+0x6fe24
00395544 2d563d9f 8400a15d 003966d4 02c40000 PPCORE!DllGetLCID+0x6f809
00396650 2d4a2c41 02c40000 2d4a2be8 02c40000 PPCORE!DllGetLCID+0x6d4e5
00396668 2d5c949b 003966d4 00000000 00000000 PPCORE!PPMain+0x1bd93
00399a64 2d5c90e3 00399a80 84005dbd 0af1928c PPCORE!DllGetLCID+0xd2be1
00399ab0 2d7beff6 0af19000 00399b7c 00399b08 PPCORE!DllGetLCID+0xd2829
0039be34 5f1ea45e 0039bf00 5f1ec17c 76a3030c PPCORE!DllGetLCID+0x2c873c
0039be48 769a03ab 0039bf00 8405ee7f 00000000 MSO!MsoCrashMainThread+0x21f
0039bed0 77763b8f 0039bf00 77763a6c 00000000 kernel32!UnhandledExceptionFilter+0x127
0039bed8 77763a6c 00000000 0039f934 7771c500 ntdll_776f0000!__RtlUserThreadStart+0x62
0039beec 77763911 00000000 00000000 00000000 ntdll_776f0000!_EH4_CallFilterFunc+0x12
0039bf14 777533dd fffffffe 0039f924 0039c050 ntdll_776f0000!_except_handler4+0x8e
0039bf38 777533af 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler2+0x26
0039bf5c 77753350 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler+0x24
0039bfe8 77700133 0139c000 0039c050 0039c000 ntdll_776f0000!RtlDispatchException+0x127
0039bff4 0039c000 0039c050 c0000005 00000000 ntdll_776f0000!KiUserExceptionDispatcher+0xf
0039c4b4 588a6fa9 00000001 00000800 0a346c84 0x39c000
0039c4cc 588a6f52 0a346cac 0039c4e8 5e567954 GFX!Ordinal818+0x1cf
0039c4f4 588a6ea1 00000020 00000800 00000008 GFX!Ordinal818+0x178
0039c5a0 588a6e0a 004066c0 0039c990 00000001 GFX!Ordinal818+0xc7
0039c5cc 51333fb1 0039c644 0039c990 00000001 GFX!Ordinal818+0x30
0039c638 51333f12 00000001 004066c0 0039c990 OART!Ordinal910+0x4f8
0039c668 63818573 0039c8ac 00000000 00000001 OART!Ordinal910+0x459
0039c8d8 6381848e 0039c910 00000000 00000001 RICHED20!RichListBoxWndProc+0xb6d2
0039c918 6aebac5f 05755318 0578b628 00000000 RICHED20!RichListBoxWndProc+0xb5ed
0039c984 6aebadf3 05757e58 00000020 00000000 MSPTLS!LssbFIsSublineEmpty+0xb13c
0039c9a8 6aeb1aea 0000000d 00000004 0039cae8 MSPTLS!LssbFIsSublineEmpty+0xb2d0
0039ca58 6aeb2295 05755f08 0039cd58 0001165b MSPTLS!LssbFIsSublineEmpty+0x1fc7
0039cae0 6aebf7fb 00000000 0039cd58 00000000 MSPTLS!LssbFIsSublineEmpty+0x2772
0039cb44 6aebff10 05755cc8 00000000 0039cd74 MSPTLS!LssbFIsSublineEmpty+0xfcd8
0039cb74 6aebf0cc 00000000 00000000 0c6651f0 MSPTLS!LssbFIsSublineEmpty+0x103ed
0039cd80 6aea4c98 057556e8 00000111 00010dec MSPTLS!LssbFIsSublineEmpty+0xf5a9
0039cdb4 6380f006 057556e8 00000111 00010dec MSPTLS!LsCreateLine+0x23
0039ced4 638005df 00000003 00000000 ffffffff RICHED20!RichListBoxWndProc+0x2165
0039cf2c 638002da 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x65f3
0039cf68 63816f3b 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x62ee
0039cf94 63816aa1 0c694c00 00000111 ffffffff RICHED20!RichListBoxWndProc+0xa09a
0039cfe4 63816987 00000001 ffffffff 00000111 RICHED20!RichListBoxWndProc+0x9c00
0039d060 6af33a5d 057552c0 00000009 00000009 RICHED20!RichListBoxWndProc+0x9ae6
0039d0f0 6af3ec94 0577ffe8 00000009 00000009 MSPTLS!LsLwMultDivR+0x13786
0039d1a8 6af3854d 0577ffe8 00000000 00000000 MSPTLS!LsLwMultDivR+0x1e9bd
0039d264 6af39b13 0c6534f8 00000000 00000000 MSPTLS!LsLwMultDivR+0x18276
0039d388 6af185ed 0039d448 0c6534f8 0039d49c MSPTLS!LsLwMultDivR+0x1983c
0039d404 6af18f8a 0039d448 0c6534f8 0000ad40 MSPTLS!FsTransformBbox+0xf44e
0039d4ec 6af316a1 0577ffe8 00000000 0000ad40 MSPTLS!FsTransformBbox+0xfdeb
0039d55c 6af31913 0c6534f8 00000009 00000004 MSPTLS!LsLwMultDivR+0x113ca
0039d5f0 6af176b9 00000001 0c6534f8 0c652544 MSPTLS!LsLwMultDivR+0x1163c
0039d644 6af0d8a1 0c652528 00000000 00000000 MSPTLS!FsTransformBbox+0xe51a
0039d6d0 6af0decb 00000000 0c6534f8 00000000 MSPTLS!FsTransformBbox+0x4702
0039d79c 6af02099 0577fa10 00000000 0039d874 MSPTLS!FsTransformBbox+0x4d2c
0039d7b4 63814a25 0577fa10 00000000 0039d874 MSPTLS!FsCreatePageBottomless+0x2d
0039d850 6381478b 0039d874 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x7b84
0039d87c 638145ef 057552c0 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x78ea
0039d9bc 63813bac 3fffffff 00400000 0c679bb8 RICHED20!RichListBoxWndProc+0x774e
0039dca0 6380e3b1 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x6d0b
0039dcd0 637fa38d 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x1510
0039dd4c 63812db5 0578eaec 0039dd8c 63812b45 RICHED20!IID_ITextServices2+0x3a1
0039dd58 63812b45 0c67dd40 0039dda0 00000000 RICHED20!RichListBoxWndProc+0x5f14
0039dd8c 51332427 0039ddcc 0e0d0dd0 0e0d0dd8 RICHED20!RichListBoxWndProc+0x5ca4
0039de50 51332095 0050b0d0 0dc6de00 0dc6def8 OART!Ordinal7653+0x502
0039de68 51331e34 0e0d0c40 0039de88 84001d0e OART!Ordinal7653+0x170
0039df84 5132f344 0e0d0c40 0dc6de60 0e0dc0c4 OART!Ordinal525+0xb7b
0039dfc4 51337c8c 0039e058 0039dffc 51337c59 OART!Ordinal6914+0x2c0
0039dfd0 51337c59 0dc6de00 84001d76 0dc6de00 OART!Ordinal3680+0xe8f
0039dffc 5133b804 0dc6de00 840022a2 0dc6de00 OART!Ordinal3680+0xe5c
0039e028 5133b7a6 0dc6de00 840022e6 0dc6de00 OART!Ordinal5150+0x240
0039e06c 513aeed3 0000000e 0039e088 00000000 OART!Ordinal5150+0x1e2
0039e090 513af827 0039e138 840023c2 027fe980 OART!Ordinal389+0x119
0039e148 518017a0 007fe964 0000b298 0039e198 OART!Ordinal1309+0x69d
0039e174 2d803b5a 02808460 0039e198 0039e1a0 OART!Ordinal5501+0x2d
0039e1c8 2d606bb6 02808464 0039e47c 840023a5 PPCORE!DllGetLCID+0x30d2a0
0039e4a8 2d4d2181 02220b44 02808464 0039e4c8 PPCORE!DllGetLCID+0x1102fc
0039e4b8 2d4d1cb4 02220a00 029a5600 0039e4fc PPCORE!PPMain+0x4b2d3
0039e4c8 2d4cf07c 00000004 029a5600 00000000 PPCORE!PPMain+0x4ae06
0039e4fc 2d4cf049 00000004 029a5600 0039e51c PPCORE!PPMain+0x481ce
0039e50c 2d4cf02a 00000004 0285c860 0039e52c PPCORE!PPMain+0x4819b
0039e51c 2d4ced04 00000004 0039e594 0039e564 PPCORE!PPMain+0x4817c
0039e52c 2d49492f 00000004 84002269 2df23040 PPCORE!PPMain+0x47e56
0039e564 2d49385d 2df23040 005849e0 00000000 PPCORE!PPMain+0xda81
0039e578 2d4937ea 0039e594 840022cd 2df23040 PPCORE!PPMain+0xc9af
0039e5c0 2d486f1e 84003f59 2df23040 005849e0 PPCORE!PPMain+0xc93c
0039f854 2df215ae 2df21575 2df20000 00000000 PPCORE!PPMain+0x70
0039f8e8 7696338a 7efde000 0039f934 777297f2 POWERPNT+0x15ae
0039f8f4 777297f2 7efde000 767616c8 00000000 kernel32!BaseThreadInitThunk+0xe
0039f934 777297c5 2df210c8 7efde000 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0039f94c 00000000 2df210c8 7efde000 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

Stacks are read from the bottom up so you can get a very basic idea of what is happening as you climb the ladder. We are really only interested in the component details … and maybe the method-routine of that component (assuming you have the symbols). For example, we start with the main PowerPoint component, PPCORE (everything after the ! is the method or routine). That calls into OART, which is the Office Art components of the MS Office suite, which goes into RICHED20, which I am guessing is something like rich text box component, then to another office component, MSPTLS, and again into another MS Office art component, GFX. After GFX you can see a couple exceptions (errors) are encountered before passing back into the main PowerPoint dll.

Hint: if you are not sure what these components are or belong to, open a new command browser in WinDbg and use the lmvm command to get the details of it, e.g. lmvm GFX:

start             end                 module name
588a0000 58a4d000   GFX        (export symbols)       GFX.DLL
    Loaded symbol image file: GFX.DLL
    Image path: C:\Program Files (x86)\Microsoft Office\Office14\GFX.DLL
    Image name: GFX.DLL
    Timestamp:        Fri Jun 28 18:43:07 2013 (51CE117B)
    CheckSum:         001B6FED
    ImageSize:        001AD000
    File version:     14.0.7104.5000
    Product version:  14.0.7104.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04e4
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft Office 2010
    InternalName:     GFX
    OriginalFilename: GFX.DLL
    ProductVersion:   14.0.7104.5000
    FileVersion:      14.0.7104.5000
    FileDescription:  Microsoft OfficeArt
    LegalCopyright:   © 2010 Microsoft Corporation.  All rights reserved.


So, from the stack above, my first guess is that we have an issue with some graphic module(s) – and maybe to a greater part the video subsystem (not necessarily the fault of MS Office). To confirm, I can look at all the thread stacks by dumping them with ~*kb. Reading from the bottom up, it looks like the problem starts with thread 15 (similar to what !analyze –v –hang shows):

15  Id: 944.1ffc Suspend: 0 Teb: 7efd8000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e9ec8c8 7772c81a 00000bd4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0e9ec92c 7772c6fe 00000000 00000000 0000000e ntdll_776f0000!RtlpWaitOnCriticalSection+0x13e
0e9ec954 637f3de0 63913530 05750f1c 637f837e ntdll_776f0000!RtlEnterCriticalSection+0x150
WARNING: Stack unwind information not available. Following frames may be wrong.
0e9ec974 63809412 0000000e ffffffff 057542c8 RICHED20+0x3de0
0e9ec98c 637fa8a3 ffffffff 00000000 0c66df94 RICHED20!IID_ITextServices2+0xf426
0e9eca10 6380a7d3 ffffffff 00000016 0c66df94 RICHED20!IID_ITextServices2+0x8b7
0e9eca38 637f9c7d 00000000 00000016 00000016 RICHED20!IID_ITextServices2+0x107e7
0e9eca5c 637f951a 0e9ecd64 00000000 00000016 RICHED20!IID_ITextHost+0xaf1
0e9ecb20 637ff8c9 00000016 00000016 0e9ece80 RICHED20!IID_ITextHost+0x38e
0e9ecbc4 63806564 00000016 0e9ece80 00000000 RICHED20!IID_ITextServices2+0x58dd
0e9ed000 63805843 0e9ed21c 0e9ed290 0c6660e4 RICHED20!IID_ITextServices2+0xc578
0e9ed1fc 638057bf 00000527 0e9ed22c 0e9ed21c RICHED20!IID_ITextServices2+0xb857
0e9ed240 51330ed8 0e9ed290 009ed274 51330ee8 RICHED20!IID_ITextServices2+0xb7d3
0e9ed254 51330821 0e9ed290 0e9ed274 8aa7103a OART!Ordinal582+0x86a
0e9ed2b0 5133e25c 02ec3c40 00000000 00000017 OART!Ordinal582+0x1b3
0e9ed2f4 5133e0a9 02ec3c40 8aa711ea 00000001 OART!Ordinal3092+0x234
0e9ed360 5133de56 0e9ed43c 8aa71112 00000001 OART!Ordinal3092+0x81
0e9ed398 5156385e 0e9ed43c 8aa71c22 02caed40 OART!Ordinal5804+0x49
0e9edea8 51c8ac06 027fdda8 000da976 001ecfe4 OART!Ordinal3710+0x1f0b
0e9ee010 51c8e83f 0e9ee1ac 0e9ee194 02d47a00 OART!Ordinal3134+0xdad1
0e9ee1e0 51c8af32 0000005e 02d47a00 0e9ee2e4 OART!Ordinal3134+0x1170a
0e9ee1f0 51c84008 0000005e 8aa7206e 0dc531d8 OART!Ordinal3134+0xddfd
0e9ee2e4 51c8477e 029b7e00 0dc53000 00000000 OART!Ordinal3134+0x6ed3
0e9ee338 51c76c9c 00000000 0dc6d000 029b7e00 OART!Ordinal3134+0x7649
0e9ee358 51c53869 00000000 0dc6d000 029b7e00 OART!Ordinal7663+0x111f5
0e9ee388 51c53d8f 029b7e00 0af6a780 00000000 OART!Ordinal5848+0x31bbe
0e9ee46c 51c541f7 01f6a780 00000007 00000000 OART!Ordinal5848+0x320e4
0e9ee494 51c513be 0af6a780 ffffffff 00000001 OART!Ordinal5848+0x3254c
0e9ee510 5166ed08 00000001 023d5100 00000000 OART!Ordinal5848+0x2f713
0e9ee650 5157b3e7 fffffffe 00000000 0dc70500 OART!Ordinal4701+0x7d347
0e9ee6b8 5157af1c 027fdda8 027fdd80 0e9ee798 OART!Ordinal1690+0x6a98
0e9ee750 51565415 0e9ee790 0e9eed80 027fde94 OART!Ordinal1690+0x65cd
0e9ee764 5156534d 00000000 0e9eed80 027fde94 OART!Ordinal2289+0xadd
0e9eebd4 515651db 0e9eed80 00000000 0e9eecf8 OART!Ordinal2289+0xa15
0e9eed2c 51564f8e 003d9fc8 00000000 00000000 OART!Ordinal2289+0x8a3
0e9eeedc 51564b24 0e9eeefc 00000000 8aa72d96 OART!Ordinal2289+0x656
0e9eef1c 51307adf 0e9eefa0 0e9ef050 0e9eefc4 OART!Ordinal2289+0x1ec
0e9eef2c 513077ae 0e9eefa0 8aa72d4e 00000000 OART!Ordinal4963+0x46
0e9eefc4 5132afe4 0e9ef050 0e9ef068 8aa732ea OART!Ordinal2541+0xfd
0e9ef060 5132afe4 02b61720 0e9ef104 8aa73276 OART!Ordinal4433+0x3e9
0e9ef0fc 5132afe4 02b615d0 0e9ef1a0 8aa73312 OART!Ordinal4433+0x3e9
0e9ef198 513075f0 02b613f0 0e9ef1d8 8aa7335a OART!Ordinal4433+0x3e9
0e9ef1d0 51303605 02b6aab0 02de3c84 8aa7308e OART!Ordinal2385+0x142
0e9ef204 513035b2 02a95dc0 00000000 51303401 OART!Ordinal2513+0xde
0e9ef240 2d4d96eb 8aa7342d 0e9ef560 0e9ef4cc OART!Ordinal2513+0x8b
0e9ef320 2d4d941b 03382e44 0e9ef240 af01221d PPCORE!PPMain+0x5283d
0e9ef404 2d4d91af 0e9ef338 0e9ef530 af01221d PPCORE!PPMain+0x5256d
0e9ef554 2d4d8fbc 027fd180 027fd180 00000002 PPCORE!PPMain+0x52301
0e9ef5a4 2d4d8d6f 027fd180 0e9ef578 00000002 PPCORE!PPMain+0x5210e
0e9ef604 2d4d8c64 0e9ef5c8 0e9ef630 02423788 PPCORE!PPMain+0x51ec1
0e9ef644 2d4d8b76 8aa7317d 0ae0ef20 2dcaba78 PPCORE!PPMain+0x51db6
0e9ef670 2d4d8b43 8aa73191 0ae0aa94 0ae0aa84 PPCORE!PPMain+0x51cc8
0e9ef69c 2d48f094 0e9ef6c3 8aa731d9 0e9ef7b8 PPCORE!PPMain+0x51c95
0e9ef6d4 2d48f053 0e9ef720 5e574ea6 0ae0aa80 PPCORE!PPMain+0x81e6
0e9ef6dc 5e574ea6 0ae0aa80 02421880 8a953774 PPCORE!PPMain+0x81a5
0e9ef720 5e556cf0 02421880 0e9ef7b8 02859214 MSO!Ordinal3920+0x30
0e9ef740 5e556be1 0e9ef7b8 00000000 0e9ef79c MSO!Ordinal6535+0x5cb
0e9ef75c 5e553ba1 0e9ef7b8 00000000 003fa7f0 MSO!Ordinal6535+0x4bc
0e9ef790 5e551d40 003fa7f0 00000000 5e551d40 MSO!Ordinal3125+0x1f0
0e9ef7ec 7696338a 003fa7f0 0e9ef838 777297f2 MSO!MsoFRegGetDwCore+0x26f
0e9ef7f8 777297f2 003fa7f0 78d117c4 00000000 kernel32!BaseThreadInitThunk+0xe
0e9ef838 777297c5 5e551ce8 003fa7f0 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0e9ef850 00000000 5e551ce8 003fa7f0 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b


Immediately after, thread 14 reveals Direct 3D 9 runtime component, which is the core multimedia APIs from Microsoft:

14  Id: 944.218c Suspend: 0 Teb: 7ef12000 Unfrozen
ChildEBP RetAddr  Args to Child             
0fe7f8bc 757514b9 00000ba4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0fe7f928 76961194 00000ba4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0fe7f940 76961148 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0fe7f954 668f801f 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
0fe7f970 668f8073 0fe7f984 7696338a 0eef0040 d3d9!CBatchFilterI::WorkerThread+0x24
0fe7f978 7696338a 0eef0040 0fe7f9c4 777297f2 d3d9!CBatchFilterI::LHBatchWorkerThread+0xd
0fe7f984 777297f2 0eef0040 79a81638 00000000 kernel32!BaseThreadInitThunk+0xe
0fe7f9c4 777297c5 668f8066 0eef0040 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0fe7f9dc 00000000 668f8066 0eef0040 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b


At this point, I decide to check for a later version of the graphics drivers for the users computer and found a newer driver release. Install, reboot, and no further Office applications reported since. Altogether, the process of identifying the basic issue itself took no more than 10 minutes, all without any software engineer credentials.

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | Leave a Comment »

The Case of the Phantom Proxy Settings

Posted by William Diaz on December 3, 2015

Not too long ago we began to notice that users who were opening IE after logging onto some workstations or Citrix servers they did not have a previously existing profile on were getting proxy settings applied in IE that were not being defined in any group policy object. Our local office proxies are applied later in the GPO processing so if it was anything coming before that, it should have been overridden by local GPO. The quick remedy was simply wait for policy to apply in the background or manually run gpupdate. But we still needed to address the cause as in some cases the proxy server was no longer reachable, preventing users from getting out to the Internet.

To get an idea of what might be happening, we deleted a profile from a machine and logged onto it. We then fired up Process Monitor and left it running while launching Internet Explorer for the first time. After verifying that incorrect proxy setting was getting applied, we stopped the trace and the log was sent my way. Proxy settings are applied in the registry so I knew I would be looking only at registry events, searching for the string of proxy server name in the log, and specifically looking at the operation for RegSetValue. From here, it was a simple matter of opening the Properties for that Event and going to the Stack tab.


The component setting the phantom proxy server (iedkcs.dll) was coming from the Internet Explorer Admin Kit or IEAK. This is otherwise known as Active Setup and its what IE will use when it is initially launched by a new user logon if the IE install was configured using IEAK. Also known as Branding, Active Setup for IE looks for INSTALL.INS if it defined in the registry. You can find this file in Program Files (x86)\Internet Explorer\CUSTOM. Looking in the INSTALL.INS, I could see under [ConnectionSettings] the phantom proxy server. You can read about the internals of this here:

At first this was a bit baffling because I had setup the rollout for IE11 using the standalone IE11 MSU and required dependencies and not the IEAK. However, after glancing at the INSTALL.INS file again, I noticed this was actually coming from an earlier IE 9 deployment, which was created using IEAK by my IE predecessor:


To correct, I referred back to and concluded that this could be avoided by going to [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\>{2A2F9EE5-7FE3-44E8-AD75-72B5704DBCB4}] and deleting "StubPath"="RunDLL32 IEDKCS32.DLL,BrandIE4 CUSTOM".

To correct on new workstations going forward, I simply created a REG DELETE command in the Task Sequence after the IE11 setup to remove the value. For existing machines, a GPP registry is the easiest way to go.

Posted in Inside Windows, Troubleshooting Tools | Tagged: | Leave a Comment »

The Case of the Failed MSI

Posted by William Diaz on December 3, 2015

I recently began encountering a problem where one of our application packages was getting hung during the install process. I could see the program directory correctly populated with application files but it seemed to get hung at end, and msiexec processes continued to run indefinitely. After manually killing each process, I checked to see if the application actually completed installing. When trying to run one of the applications that has a dependency on one of the failed MSI’s APIs, I encountered the following:


Further, Outlook was crashing when trying to load the application’s main component.

Faulting application name: OUTLOOK.EXE, version: 14.0.7012.1000, time stamp: 0x514a1b69
Faulting module name: imFileSite.dll, version: 8.5.3006.93, time stamp: 0x52d811c0
Exception code: 0xc0000005
Fault offset: 0x0008637e
Faulting process id: 0xd18
Faulting application start time: 0x01d12d4e055d29fc
Faulting application path: C:\Program Files (x86)\Microsoft Office\Office14\OUTLOOK.EXE
Faulting module path: C:\Program Files (x86)\Interwoven\WorkSite\iOutlook\imFileSite.dll


My guess was that the application’s components were not getting registered. To confirm, I manually registered all the components in the program directly via the command line:

FOR /R "C:\Program Files (x86)\ApplicationFolder" %G IN (*.dll) DO "%systemroot%\system32\regsvr32.exe" /s "%G").

Afterwards, the application with the dependency launched without error and Outlook was no longer crashing. However, the root issue still needed to be identified as I had no idea what else in the MSI was failing to complete. After manually deleting the components from the file system (it was missing from Programs and Features), I ran the MSI again and this time used Process Explorer to look inside the hung msiexec processes. There were multiple processes running, but I could see one process, though, eating CPU resources, and this was likely where I might find the culprit:


Opening the process details, I select the Threads tab. The thread that is doing all the work (or in this case getting hung up) is indicated by the Cycles tab. From here, I opened the thread by selecting it and clicking Stack:


Stacks are read from the bottom up. You can see a 3rd party component here (HCApi – McAfee HIPS) at play. Note, this might not be entirely abnormal as you can always expect any anti-virus suite to be hooking itself into any number of processes. So, to confirm what I was seeing, I used the Task Manager to dump the msiexec process that was using the CPU time by right-clicking it and select Create dump file (which can also be done via Process Explorer):


Dumps can be analyzed with WinDbg or a tool like DebugDiag 2.0. I have grown increasingly lazy and forgetful over the years with WinDbg. It has a very high learning curve and if you don’t use it much, its easy to forget everything except the old faithful !analyze –v or !analyze –v –hang. I used DebugDiag instead. When it is installed, simply right-click the dump file and select Analyze Crash/hang Issue from the context menu and point it to the crash file.



It will do its best to figure out the issue in the most vague way and you almost always need to do some interpretation of your own. For the most part, the analysis summary can be ignored.


A little bit down in the report points me to what I was seeing in Process Explorer with the problem thread that was using all the CPU time:


You can click the Thread ID like a hyperlink to follow it down in the report, expanding the thread. It reveals a familiar site. However, there is a bit more insight as the entry point is revealed and I can see the problem has something to do with the CustomAction table of the MSI itself.


Through a quick process of elimination using Orca to remove some of the rows in the CustomAction table, I narrow down the cause specifically down to action ISSelfRegisterCosting.


Without that row, the MSI install completes normally; or so at least it seems. I have no idea what removing this action might have elsewhere so this merely a hack. To further confirm the problem is being caused by McAfee, I perform the install on a virtual machine where McAfee is not installed and it proceeds normally. I then reach out to our McAfee enterprise admin and ask him to disable HIPS on one of my workstation. After doing so, the MSI runs and the application installs as expected. He inherits the problem.


This has since been corrected by MacAfee with a HIPS update. One of the DLLs trying to get registered was getting blocked.

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , | 7 Comments »