Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Archive for the ‘Troubleshooting Tools’ Category

A Quick One-Dimensional Stack Debug and Solution

Posted by William Diaz on December 17, 2015


I am a big fan of debugging tools, especially WinDbg. These tools can be quite complicated but most of the time I use them simply for a “one-dimensional” analysis of an application or system crash and use the information from the dump to find a cause and hopefully the most logical fix. In the example here, a user complains that PowerPoint-Excel is often crashing or becoming hung. I was able to locate a couple of errors in the Event Viewer > Application Logs that indicated a crash had taken place within the MS Office product, but it was too generic and failed to resolve to the failing component. I asked the user to manually create a crash dump for me the next time the issue happened by using the Task Manager > Applications > right-clicking the unresponsive PowerPoint or Excel application > Select Create Dump File. The dump file gets sent to the user’s temp directory, e.g. C:\Users\username\AppData\Local\Temp\AppName.DMP.

I copied the dmp and opened with with WinDbg. Note, if you are analyzing a 32bit application dump (often the case with MS Office – because who actually uses 64bit office?) you will want to use the 32bit version of WinDbg. Or, if you are incredibly lazy like me (can’t figure out what happened to my x86 version), you can use the 64bit WinDbg and change it use x86-based processor mode. Just run .load wow64exts followed by .effmach x86, followed by the always faithful !analyze –v –hang.

This resulted in following output (Including only the stack text since everything before this is not useful) :

STACK_TEXT: 
00392774 75751605 00000001 003927c4 00000001 ntdll_776f0000!ZwWaitForMultipleObjects+0x15
00392810 769619f8 003927c4 00392838 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
00392858 5e558ac9 00000001 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
WARNING: Stack unwind information not available. Following frames may be wrong.
00392898 5e58f314 00000001 02b68204 ffffffff MSO!Ordinal10580+0x28e
003928d0 5e589c1c 00000000 003928ec 02b69140 MSO!Ordinal4721+0x114
00392900 5e58a80a 02421880 ffffffff 02b68200 MSO!Ordinal5739+0x18a
00392914 2d5b9771 02421880 2d4d05b6 00392948 MSO!Ordinal9231+0x2d6
00392924 2d4d1ebe 00000001 2dc983fc 00000000 PPCORE!DllGetLCID+0xc2eb7
00392948 2d5666de 00000001 84008571 0af19000 PPCORE!PPMain+0x4b010
0039427c 2d5660c3 02b69140 00000425 84009249 PPCORE!DllGetLCID+0x6fe24
00395544 2d563d9f 8400a15d 003966d4 02c40000 PPCORE!DllGetLCID+0x6f809
00396650 2d4a2c41 02c40000 2d4a2be8 02c40000 PPCORE!DllGetLCID+0x6d4e5
00396668 2d5c949b 003966d4 00000000 00000000 PPCORE!PPMain+0x1bd93
00399a64 2d5c90e3 00399a80 84005dbd 0af1928c PPCORE!DllGetLCID+0xd2be1
00399ab0 2d7beff6 0af19000 00399b7c 00399b08 PPCORE!DllGetLCID+0xd2829
0039be34 5f1ea45e 0039bf00 5f1ec17c 76a3030c PPCORE!DllGetLCID+0x2c873c
0039be48 769a03ab 0039bf00 8405ee7f 00000000 MSO!MsoCrashMainThread+0x21f
0039bed0 77763b8f 0039bf00 77763a6c 00000000 kernel32!UnhandledExceptionFilter+0x127
0039bed8 77763a6c 00000000 0039f934 7771c500 ntdll_776f0000!__RtlUserThreadStart+0x62
0039beec 77763911 00000000 00000000 00000000 ntdll_776f0000!_EH4_CallFilterFunc+0x12
0039bf14 777533dd fffffffe 0039f924 0039c050 ntdll_776f0000!_except_handler4+0x8e
0039bf38 777533af 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler2+0x26
0039bf5c 77753350 0039c000 0039f924 0039c050 ntdll_776f0000!ExecuteHandler+0x24
0039bfe8 77700133 0139c000 0039c050 0039c000 ntdll_776f0000!RtlDispatchException+0x127
0039bff4 0039c000 0039c050 c0000005 00000000 ntdll_776f0000!KiUserExceptionDispatcher+0xf
0039c4b4 588a6fa9 00000001 00000800 0a346c84 0x39c000
0039c4cc 588a6f52 0a346cac 0039c4e8 5e567954 GFX!Ordinal818+0x1cf
0039c4f4 588a6ea1 00000020 00000800 00000008 GFX!Ordinal818+0x178
0039c5a0 588a6e0a 004066c0 0039c990 00000001 GFX!Ordinal818+0xc7
0039c5cc 51333fb1 0039c644 0039c990 00000001 GFX!Ordinal818+0x30
0039c638 51333f12 00000001 004066c0 0039c990 OART!Ordinal910+0x4f8
0039c668 63818573 0039c8ac 00000000 00000001 OART!Ordinal910+0x459
0039c8d8 6381848e 0039c910 00000000 00000001 RICHED20!RichListBoxWndProc+0xb6d2
0039c918 6aebac5f 05755318 0578b628 00000000 RICHED20!RichListBoxWndProc+0xb5ed
0039c984 6aebadf3 05757e58 00000020 00000000 MSPTLS!LssbFIsSublineEmpty+0xb13c
0039c9a8 6aeb1aea 0000000d 00000004 0039cae8 MSPTLS!LssbFIsSublineEmpty+0xb2d0
0039ca58 6aeb2295 05755f08 0039cd58 0001165b MSPTLS!LssbFIsSublineEmpty+0x1fc7
0039cae0 6aebf7fb 00000000 0039cd58 00000000 MSPTLS!LssbFIsSublineEmpty+0x2772
0039cb44 6aebff10 05755cc8 00000000 0039cd74 MSPTLS!LssbFIsSublineEmpty+0xfcd8
0039cb74 6aebf0cc 00000000 00000000 0c6651f0 MSPTLS!LssbFIsSublineEmpty+0x103ed
0039cd80 6aea4c98 057556e8 00000111 00010dec MSPTLS!LssbFIsSublineEmpty+0xf5a9
0039cdb4 6380f006 057556e8 00000111 00010dec MSPTLS!LsCreateLine+0x23
0039ced4 638005df 00000003 00000000 ffffffff RICHED20!RichListBoxWndProc+0x2165
0039cf2c 638002da 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x65f3
0039cf68 63816f3b 0039d894 00000003 00000000 RICHED20!IID_ITextServices2+0x62ee
0039cf94 63816aa1 0c694c00 00000111 ffffffff RICHED20!RichListBoxWndProc+0xa09a
0039cfe4 63816987 00000001 ffffffff 00000111 RICHED20!RichListBoxWndProc+0x9c00
0039d060 6af33a5d 057552c0 00000009 00000009 RICHED20!RichListBoxWndProc+0x9ae6
0039d0f0 6af3ec94 0577ffe8 00000009 00000009 MSPTLS!LsLwMultDivR+0x13786
0039d1a8 6af3854d 0577ffe8 00000000 00000000 MSPTLS!LsLwMultDivR+0x1e9bd
0039d264 6af39b13 0c6534f8 00000000 00000000 MSPTLS!LsLwMultDivR+0x18276
0039d388 6af185ed 0039d448 0c6534f8 0039d49c MSPTLS!LsLwMultDivR+0x1983c
0039d404 6af18f8a 0039d448 0c6534f8 0000ad40 MSPTLS!FsTransformBbox+0xf44e
0039d4ec 6af316a1 0577ffe8 00000000 0000ad40 MSPTLS!FsTransformBbox+0xfdeb
0039d55c 6af31913 0c6534f8 00000009 00000004 MSPTLS!LsLwMultDivR+0x113ca
0039d5f0 6af176b9 00000001 0c6534f8 0c652544 MSPTLS!LsLwMultDivR+0x1163c
0039d644 6af0d8a1 0c652528 00000000 00000000 MSPTLS!FsTransformBbox+0xe51a
0039d6d0 6af0decb 00000000 0c6534f8 00000000 MSPTLS!FsTransformBbox+0x4702
0039d79c 6af02099 0577fa10 00000000 0039d874 MSPTLS!FsTransformBbox+0x4d2c
0039d7b4 63814a25 0577fa10 00000000 0039d874 MSPTLS!FsCreatePageBottomless+0x2d
0039d850 6381478b 0039d874 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x7b84
0039d87c 638145ef 057552c0 00010dec 0c6af128 RICHED20!RichListBoxWndProc+0x78ea
0039d9bc 63813bac 3fffffff 00400000 0c679bb8 RICHED20!RichListBoxWndProc+0x774e
0039dca0 6380e3b1 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x6d0b
0039dcd0 637fa38d 0039dcf0 00000185 00000185 RICHED20!RichListBoxWndProc+0x1510
0039dd4c 63812db5 0578eaec 0039dd8c 63812b45 RICHED20!IID_ITextServices2+0x3a1
0039dd58 63812b45 0c67dd40 0039dda0 00000000 RICHED20!RichListBoxWndProc+0x5f14
0039dd8c 51332427 0039ddcc 0e0d0dd0 0e0d0dd8 RICHED20!RichListBoxWndProc+0x5ca4
0039de50 51332095 0050b0d0 0dc6de00 0dc6def8 OART!Ordinal7653+0x502
0039de68 51331e34 0e0d0c40 0039de88 84001d0e OART!Ordinal7653+0x170
0039df84 5132f344 0e0d0c40 0dc6de60 0e0dc0c4 OART!Ordinal525+0xb7b
0039dfc4 51337c8c 0039e058 0039dffc 51337c59 OART!Ordinal6914+0x2c0
0039dfd0 51337c59 0dc6de00 84001d76 0dc6de00 OART!Ordinal3680+0xe8f
0039dffc 5133b804 0dc6de00 840022a2 0dc6de00 OART!Ordinal3680+0xe5c
0039e028 5133b7a6 0dc6de00 840022e6 0dc6de00 OART!Ordinal5150+0x240
0039e06c 513aeed3 0000000e 0039e088 00000000 OART!Ordinal5150+0x1e2
0039e090 513af827 0039e138 840023c2 027fe980 OART!Ordinal389+0x119
0039e148 518017a0 007fe964 0000b298 0039e198 OART!Ordinal1309+0x69d
0039e174 2d803b5a 02808460 0039e198 0039e1a0 OART!Ordinal5501+0x2d
0039e1c8 2d606bb6 02808464 0039e47c 840023a5 PPCORE!DllGetLCID+0x30d2a0
0039e4a8 2d4d2181 02220b44 02808464 0039e4c8 PPCORE!DllGetLCID+0x1102fc
0039e4b8 2d4d1cb4 02220a00 029a5600 0039e4fc PPCORE!PPMain+0x4b2d3
0039e4c8 2d4cf07c 00000004 029a5600 00000000 PPCORE!PPMain+0x4ae06
0039e4fc 2d4cf049 00000004 029a5600 0039e51c PPCORE!PPMain+0x481ce
0039e50c 2d4cf02a 00000004 0285c860 0039e52c PPCORE!PPMain+0x4819b
0039e51c 2d4ced04 00000004 0039e594 0039e564 PPCORE!PPMain+0x4817c
0039e52c 2d49492f 00000004 84002269 2df23040 PPCORE!PPMain+0x47e56
0039e564 2d49385d 2df23040 005849e0 00000000 PPCORE!PPMain+0xda81
0039e578 2d4937ea 0039e594 840022cd 2df23040 PPCORE!PPMain+0xc9af
0039e5c0 2d486f1e 84003f59 2df23040 005849e0 PPCORE!PPMain+0xc93c
0039f854 2df215ae 2df21575 2df20000 00000000 PPCORE!PPMain+0x70
0039f8e8 7696338a 7efde000 0039f934 777297f2 POWERPNT+0x15ae
0039f8f4 777297f2 7efde000 767616c8 00000000 kernel32!BaseThreadInitThunk+0xe
0039f934 777297c5 2df210c8 7efde000 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0039f94c 00000000 2df210c8 7efde000 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

Stacks are read from the bottom up so you can get a very basic idea of what is happening as you climb the ladder. We are really only interested in the component details … and maybe the method-routine of that component (assuming you have the symbols). For example, we start with the main PowerPoint component, PPCORE (everything after the ! is the method or routine). That calls into OART, which is the Office Art components of the MS Office suite, which goes into RICHED20, which I am guessing is something like rich text box component, then to another office component, MSPTLS, and again into another MS Office art component, GFX. After GFX you can see a couple exceptions (errors) are encountered before passing back into the main PowerPoint dll.

Hint: if you are not sure what these components are or belong to, open a new command browser in WinDbg and use the lmvm command to get the details of it, e.g. lmvm GFX:

start             end                 module name
588a0000 58a4d000   GFX        (export symbols)       GFX.DLL
    Loaded symbol image file: GFX.DLL
    Image path: C:\Program Files (x86)\Microsoft Office\Office14\GFX.DLL
    Image name: GFX.DLL
    Timestamp:        Fri Jun 28 18:43:07 2013 (51CE117B)
    CheckSum:         001B6FED
    ImageSize:        001AD000
    File version:     14.0.7104.5000
    Product version:  14.0.7104.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04e4
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft Office 2010
    InternalName:     GFX
    OriginalFilename: GFX.DLL
    ProductVersion:   14.0.7104.5000
    FileVersion:      14.0.7104.5000
    FileDescription:  Microsoft OfficeArt
    LegalCopyright:   © 2010 Microsoft Corporation.  All rights reserved.

 

So, from the stack above, my first guess is that we have an issue with some graphic module(s) – and maybe to a greater part the video subsystem (not necessarily the fault of MS Office). To confirm, I can look at all the thread stacks by dumping them with ~*kb. Reading from the bottom up, it looks like the problem starts with thread 15 (similar to what !analyze –v –hang shows):

15  Id: 944.1ffc Suspend: 0 Teb: 7efd8000 Unfrozen
ChildEBP RetAddr  Args to Child             
0e9ec8c8 7772c81a 00000bd4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0e9ec92c 7772c6fe 00000000 00000000 0000000e ntdll_776f0000!RtlpWaitOnCriticalSection+0x13e
0e9ec954 637f3de0 63913530 05750f1c 637f837e ntdll_776f0000!RtlEnterCriticalSection+0x150
WARNING: Stack unwind information not available. Following frames may be wrong.
0e9ec974 63809412 0000000e ffffffff 057542c8 RICHED20+0x3de0
0e9ec98c 637fa8a3 ffffffff 00000000 0c66df94 RICHED20!IID_ITextServices2+0xf426
0e9eca10 6380a7d3 ffffffff 00000016 0c66df94 RICHED20!IID_ITextServices2+0x8b7
0e9eca38 637f9c7d 00000000 00000016 00000016 RICHED20!IID_ITextServices2+0x107e7
0e9eca5c 637f951a 0e9ecd64 00000000 00000016 RICHED20!IID_ITextHost+0xaf1
0e9ecb20 637ff8c9 00000016 00000016 0e9ece80 RICHED20!IID_ITextHost+0x38e
0e9ecbc4 63806564 00000016 0e9ece80 00000000 RICHED20!IID_ITextServices2+0x58dd
0e9ed000 63805843 0e9ed21c 0e9ed290 0c6660e4 RICHED20!IID_ITextServices2+0xc578
0e9ed1fc 638057bf 00000527 0e9ed22c 0e9ed21c RICHED20!IID_ITextServices2+0xb857
0e9ed240 51330ed8 0e9ed290 009ed274 51330ee8 RICHED20!IID_ITextServices2+0xb7d3
0e9ed254 51330821 0e9ed290 0e9ed274 8aa7103a OART!Ordinal582+0x86a
0e9ed2b0 5133e25c 02ec3c40 00000000 00000017 OART!Ordinal582+0x1b3
0e9ed2f4 5133e0a9 02ec3c40 8aa711ea 00000001 OART!Ordinal3092+0x234
0e9ed360 5133de56 0e9ed43c 8aa71112 00000001 OART!Ordinal3092+0x81
0e9ed398 5156385e 0e9ed43c 8aa71c22 02caed40 OART!Ordinal5804+0x49
0e9edea8 51c8ac06 027fdda8 000da976 001ecfe4 OART!Ordinal3710+0x1f0b
0e9ee010 51c8e83f 0e9ee1ac 0e9ee194 02d47a00 OART!Ordinal3134+0xdad1
0e9ee1e0 51c8af32 0000005e 02d47a00 0e9ee2e4 OART!Ordinal3134+0x1170a
0e9ee1f0 51c84008 0000005e 8aa7206e 0dc531d8 OART!Ordinal3134+0xddfd
0e9ee2e4 51c8477e 029b7e00 0dc53000 00000000 OART!Ordinal3134+0x6ed3
0e9ee338 51c76c9c 00000000 0dc6d000 029b7e00 OART!Ordinal3134+0x7649
0e9ee358 51c53869 00000000 0dc6d000 029b7e00 OART!Ordinal7663+0x111f5
0e9ee388 51c53d8f 029b7e00 0af6a780 00000000 OART!Ordinal5848+0x31bbe
0e9ee46c 51c541f7 01f6a780 00000007 00000000 OART!Ordinal5848+0x320e4
0e9ee494 51c513be 0af6a780 ffffffff 00000001 OART!Ordinal5848+0x3254c
0e9ee510 5166ed08 00000001 023d5100 00000000 OART!Ordinal5848+0x2f713
0e9ee650 5157b3e7 fffffffe 00000000 0dc70500 OART!Ordinal4701+0x7d347
0e9ee6b8 5157af1c 027fdda8 027fdd80 0e9ee798 OART!Ordinal1690+0x6a98
0e9ee750 51565415 0e9ee790 0e9eed80 027fde94 OART!Ordinal1690+0x65cd
0e9ee764 5156534d 00000000 0e9eed80 027fde94 OART!Ordinal2289+0xadd
0e9eebd4 515651db 0e9eed80 00000000 0e9eecf8 OART!Ordinal2289+0xa15
0e9eed2c 51564f8e 003d9fc8 00000000 00000000 OART!Ordinal2289+0x8a3
0e9eeedc 51564b24 0e9eeefc 00000000 8aa72d96 OART!Ordinal2289+0x656
0e9eef1c 51307adf 0e9eefa0 0e9ef050 0e9eefc4 OART!Ordinal2289+0x1ec
0e9eef2c 513077ae 0e9eefa0 8aa72d4e 00000000 OART!Ordinal4963+0x46
0e9eefc4 5132afe4 0e9ef050 0e9ef068 8aa732ea OART!Ordinal2541+0xfd
0e9ef060 5132afe4 02b61720 0e9ef104 8aa73276 OART!Ordinal4433+0x3e9
0e9ef0fc 5132afe4 02b615d0 0e9ef1a0 8aa73312 OART!Ordinal4433+0x3e9
0e9ef198 513075f0 02b613f0 0e9ef1d8 8aa7335a OART!Ordinal4433+0x3e9
0e9ef1d0 51303605 02b6aab0 02de3c84 8aa7308e OART!Ordinal2385+0x142
0e9ef204 513035b2 02a95dc0 00000000 51303401 OART!Ordinal2513+0xde
0e9ef240 2d4d96eb 8aa7342d 0e9ef560 0e9ef4cc OART!Ordinal2513+0x8b
0e9ef320 2d4d941b 03382e44 0e9ef240 af01221d PPCORE!PPMain+0x5283d
0e9ef404 2d4d91af 0e9ef338 0e9ef530 af01221d PPCORE!PPMain+0x5256d
0e9ef554 2d4d8fbc 027fd180 027fd180 00000002 PPCORE!PPMain+0x52301
0e9ef5a4 2d4d8d6f 027fd180 0e9ef578 00000002 PPCORE!PPMain+0x5210e
0e9ef604 2d4d8c64 0e9ef5c8 0e9ef630 02423788 PPCORE!PPMain+0x51ec1
0e9ef644 2d4d8b76 8aa7317d 0ae0ef20 2dcaba78 PPCORE!PPMain+0x51db6
0e9ef670 2d4d8b43 8aa73191 0ae0aa94 0ae0aa84 PPCORE!PPMain+0x51cc8
0e9ef69c 2d48f094 0e9ef6c3 8aa731d9 0e9ef7b8 PPCORE!PPMain+0x51c95
0e9ef6d4 2d48f053 0e9ef720 5e574ea6 0ae0aa80 PPCORE!PPMain+0x81e6
0e9ef6dc 5e574ea6 0ae0aa80 02421880 8a953774 PPCORE!PPMain+0x81a5
0e9ef720 5e556cf0 02421880 0e9ef7b8 02859214 MSO!Ordinal3920+0x30
0e9ef740 5e556be1 0e9ef7b8 00000000 0e9ef79c MSO!Ordinal6535+0x5cb
0e9ef75c 5e553ba1 0e9ef7b8 00000000 003fa7f0 MSO!Ordinal6535+0x4bc
0e9ef790 5e551d40 003fa7f0 00000000 5e551d40 MSO!Ordinal3125+0x1f0
0e9ef7ec 7696338a 003fa7f0 0e9ef838 777297f2 MSO!MsoFRegGetDwCore+0x26f
0e9ef7f8 777297f2 003fa7f0 78d117c4 00000000 kernel32!BaseThreadInitThunk+0xe
0e9ef838 777297c5 5e551ce8 003fa7f0 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0e9ef850 00000000 5e551ce8 003fa7f0 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

 

Immediately after, thread 14 reveals Direct 3D 9 runtime component, which is the core multimedia APIs from Microsoft:

14  Id: 944.218c Suspend: 0 Teb: 7ef12000 Unfrozen
ChildEBP RetAddr  Args to Child             
0fe7f8bc 757514b9 00000ba4 00000000 00000000 ntdll_776f0000!ZwWaitForSingleObject+0x15
0fe7f928 76961194 00000ba4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0fe7f940 76961148 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0fe7f954 668f801f 00000ba4 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
0fe7f970 668f8073 0fe7f984 7696338a 0eef0040 d3d9!CBatchFilterI::WorkerThread+0x24
0fe7f978 7696338a 0eef0040 0fe7f9c4 777297f2 d3d9!CBatchFilterI::LHBatchWorkerThread+0xd
0fe7f984 777297f2 0eef0040 79a81638 00000000 kernel32!BaseThreadInitThunk+0xe
0fe7f9c4 777297c5 668f8066 0eef0040 ffffffff ntdll_776f0000!__RtlUserThreadStart+0x70
0fe7f9dc 00000000 668f8066 0eef0040 00000000 ntdll_776f0000!_RtlUserThreadStart+0x1b

 

At this point, I decide to check for a later version of the graphics drivers for the users computer and found a newer driver release. Install, reboot, and no further Office applications reported since. Altogether, the process of identifying the basic issue itself took no more than 10 minutes, all without any software engineer credentials.

Advertisements

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | 2 Comments »

The Case of the Phantom Proxy Settings

Posted by William Diaz on December 3, 2015


Not too long ago we began to notice that users who were opening IE after logging onto some workstations or Citrix servers they did not have a previously existing profile on were getting proxy settings applied in IE that were not being defined in any group policy object. Our local office proxies are applied later in the GPO processing so if it was anything coming before that, it should have been overridden by local GPO. The quick remedy was simply wait for policy to apply in the background or manually run gpupdate. But we still needed to address the cause as in some cases the proxy server was no longer reachable, preventing users from getting out to the Internet.

To get an idea of what might be happening, we deleted a profile from a machine and logged onto it. We then fired up Process Monitor and left it running while launching Internet Explorer for the first time. After verifying that incorrect proxy setting was getting applied, we stopped the trace and the log was sent my way. Proxy settings are applied in the registry so I knew I would be looking only at registry events, searching for the string of proxy server name in the log, and specifically looking at the operation for RegSetValue. From here, it was a simple matter of opening the Properties for that Event and going to the Stack tab.

clip_image001

The component setting the phantom proxy server (iedkcs.dll) was coming from the Internet Explorer Admin Kit or IEAK. This is otherwise known as Active Setup and its what IE will use when it is initially launched by a new user logon if the IE install was configured using IEAK. Also known as Branding, Active Setup for IE looks for INSTALL.INS if it defined in the registry. You can find this file in Program Files (x86)\Internet Explorer\CUSTOM. Looking in the INSTALL.INS, I could see under [ConnectionSettings] the phantom proxy server. You can read about the internals of this here: https://support.microsoft.com/en-us/kb/2029043

At first this was a bit baffling because I had setup the rollout for IE11 using the standalone IE11 MSU and required dependencies and not the IEAK. However, after glancing at the INSTALL.INS file again, I noticed this was actually coming from an earlier IE 9 deployment, which was created using IEAK by my IE predecessor:

image

To correct, I referred back to https://support.microsoft.com/en-us/kb/2029043 and concluded that this could be avoided by going to [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\>{2A2F9EE5-7FE3-44E8-AD75-72B5704DBCB4}] and deleting "StubPath"="RunDLL32 IEDKCS32.DLL,BrandIE4 CUSTOM".

To correct on new workstations going forward, I simply created a REG DELETE command in the Task Sequence after the IE11 setup to remove the value. For existing machines, a GPP registry is the easiest way to go.

Posted in Inside Windows, Troubleshooting Tools | Tagged: | Leave a Comment »

The Case of the Failed MSI

Posted by William Diaz on December 3, 2015


I recently began encountering a problem where one of our application packages was getting hung during the install process. I could see the program directory correctly populated with application files but it seemed to get hung at end, and msiexec processes continued to run indefinitely. After manually killing each process, I checked to see if the application actually completed installing. When trying to run one of the applications that has a dependency on one of the failed MSI’s APIs, I encountered the following:

image

Further, Outlook was crashing when trying to load the application’s main component.

Faulting application name: OUTLOOK.EXE, version: 14.0.7012.1000, time stamp: 0x514a1b69
Faulting module name: imFileSite.dll, version: 8.5.3006.93, time stamp: 0x52d811c0
Exception code: 0xc0000005
Fault offset: 0x0008637e
Faulting process id: 0xd18
Faulting application start time: 0x01d12d4e055d29fc
Faulting application path: C:\Program Files (x86)\Microsoft Office\Office14\OUTLOOK.EXE
Faulting module path: C:\Program Files (x86)\Interwoven\WorkSite\iOutlook\imFileSite.dll

 

My guess was that the application’s components were not getting registered. To confirm, I manually registered all the components in the program directly via the command line:

FOR /R "C:\Program Files (x86)\ApplicationFolder" %G IN (*.dll) DO "%systemroot%\system32\regsvr32.exe" /s "%G").

Afterwards, the application with the dependency launched without error and Outlook was no longer crashing. However, the root issue still needed to be identified as I had no idea what else in the MSI was failing to complete. After manually deleting the components from the file system (it was missing from Programs and Features), I ran the MSI again and this time used Process Explorer to look inside the hung msiexec processes. There were multiple processes running, but I could see one process, though, eating CPU resources, and this was likely where I might find the culprit:

image

Opening the process details, I select the Threads tab. The thread that is doing all the work (or in this case getting hung up) is indicated by the Cycles tab. From here, I opened the thread by selecting it and clicking Stack:

image

Stacks are read from the bottom up. You can see a 3rd party component here (HCApi – McAfee HIPS) at play. Note, this might not be entirely abnormal as you can always expect any anti-virus suite to be hooking itself into any number of processes. So, to confirm what I was seeing, I used the Task Manager to dump the msiexec process that was using the CPU time by right-clicking it and select Create dump file (which can also be done via Process Explorer):

image

Dumps can be analyzed with WinDbg or a tool like DebugDiag 2.0. I have grown increasingly lazy and forgetful over the years with WinDbg. It has a very high learning curve and if you don’t use it much, its easy to forget everything except the old faithful !analyze –v or !analyze –v –hang. I used DebugDiag instead. When it is installed, simply right-click the dump file and select Analyze Crash/hang Issue from the context menu and point it to the crash file.

image

image

It will do its best to figure out the issue in the most vague way and you almost always need to do some interpretation of your own. For the most part, the analysis summary can be ignored.

image

A little bit down in the report points me to what I was seeing in Process Explorer with the problem thread that was using all the CPU time:

image

You can click the Thread ID like a hyperlink to follow it down in the report, expanding the thread. It reveals a familiar site. However, there is a bit more insight as the entry point is revealed and I can see the problem has something to do with the CustomAction table of the MSI itself.

image

Through a quick process of elimination using Orca to remove some of the rows in the CustomAction table, I narrow down the cause specifically down to action ISSelfRegisterCosting.

image

Without that row, the MSI install completes normally; or so at least it seems. I have no idea what removing this action might have elsewhere so this merely a hack. To further confirm the problem is being caused by McAfee, I perform the install on a virtual machine where McAfee is not installed and it proceeds normally. I then reach out to our McAfee enterprise admin and ask him to disable HIPS on one of my workstation. After doing so, the MSI runs and the application installs as expected. He inherits the problem.


Update

This has since been corrected by MacAfee with a HIPS update. One of the DLLs trying to get registered was getting blocked.

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , | 9 Comments »

Another Case of IE and Outlook Crashes

Posted by William Diaz on February 26, 2015


I haven’t done one of these in quite awhile. So without further ado.

The symptom: Internet Explorer 11 and Outlook were crashing*. This was happening in all cases after applying a new task sequence. The event viewer did not produce any events that would help so I configured the system to capture crash dumps for applications. After rebooting, I reproduced the crash by simply opening IE an grabbed one of the dumps and ran it through an x86 debugger (by default, the 32bit Content tabs in IE 11 run as 32bit processes). Using ! analyze –v command produced the following output:

STACK_TEXT: 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0339e974 6818a97d 005a09f8 0339e994 5226b9f5 0x490054
0339e9d8 68193c6f 0059f730 0059f7ec 5226bb49 D3D10Level9!UMAdapter::UnderlyingGetCaps+0x31
0339eb64 681940c2 0059f730 0339ebb4 00000002 D3D10Level9!UMAdapter::Open+0x165
0339eb98 69e0cae6 0339ebb4 523e5d23 00020009 D3D10Level9!OpenAdapter10_2+0x45
0339ebdc 69de09c2 005a09f8 0059f35c 69dda858 d3d11!NDXGI::CUMDAdapter::CUMDAdapter+0x18b
0339edc8 69de05fd 0339f0f0 00000002 00000029 d3d11!CCreateDeviceCache::CUMDAdapterCache::Load+0x1fc
0339ee18 69de12f3 0339f0b8 0339ee90 00009300 d3d11!CCreateDeviceCache::CAdapterCache::ResolveUMDAndVersion+0xc8
0339f1f4 69de1e4e 00000000 00595190 00000000 d3d11!D3D11CoreCreateDevice+0x353
0339f484 69de1bdd 00595190 00000000 00000000 d3d11!D3D11CreateDeviceAndSwapChain+0x268
0339f4bc 5bd5b4ed 00595190 00000000 00000000 d3d11!D3D11CreateDevice+0x2c
0339f4f8 5bd5c698 00000001 0339f58c 0339f564 mshtml!CDXGIHelper::CreateD3DDevice+0x48
0339f520 5bd5bc31 72657200 001af5f2 005910fc mshtml!CDXHelper::CreateD3D11Device+0x99
0339f6fc 5bd5ba45 00000000 00000000 00000000 mshtml!CDXResourceDomain::EnsureD3DDevice+0x82
0339f718 5bd5b9ad 00000000 00000000 00000000 mshtml!CDXResourceDomain::Initialize+0x67
0339f734 5bb35771 5c81e03c 00000000 00000000 mshtml!CDXResourceDomain::Create+0x48
0339f76c 5bd5a4c9 0339f78c 00000000 00538c78 mshtml!CDXResourceDomain::EnsureSharedDomain+0xa9
0339f7a0 671015ca 670b21cc 00000000 00000000 mshtml!CDXResourceDomain::EarlyStartDisplaySystem+0xe8
0339f7a4 670b21cc 00000000 00000000 00545cf8 ieframe!DirectUI::TouchEdit2::_UpdatePrompt+0x22
0339f7bc 77647df9 00538c78 744d2d1c 0053b958 ieframe!ExecuteWorkItemThreadProc+0x30
0339f830 77632b65 00538c78 00545cf8 744d2cbc ntdll!RtlpTpWorkCallback+0x11d
0339f990 76d6339a 0053b950 0339f9dc 7761bf32 ntdll!TppWorkerThread+0x572
0339f99c 7761bf32 0053b950 744d2cf0 00000000 kernel32!BaseThreadInitThunk+0xe
0339f9dc 7761bf05 776325c1 0053b950 ffffffff ntdll!__RtlUserThreadStart+0x70
0339f9f4 00000000 776325c1 0053b950 00000000 ntdll!_RtlUserThreadStart+0x1b

STACK_COMMAND:  ~2s; .ecxr ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  unknown!printable+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: unknown

IMAGE_NAME:  unknown

DEBUG_FLR_IMAGE_TIMESTAMP:  0

 

I know D3D is Microsoft Direct X. To validate this, simply run the command lmvm d3d11:

0:002> lmvm d3d11
start    end        module name
69dd0000 69f45000   d3d11      (pdb symbols)          c:\symbols\d3d11.pdb\A40CED7361AB405886384123277BE23F1\d3d11.pdb
    Loaded symbol image file: d3d11.dll
    Image path: C:\Windows\System32\d3d11.dll
    Image name: d3d11.dll
    Timestamp:        Wed Mar 27 18:48:45 2013 (5153774D)
    CheckSum:         0017E157
    ImageSize:        00175000
    File version:     6.2.9200.16570
    Product version:  6.2.9200.16570
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     D3D11.dll
    OriginalFilename: D3D11.dll
    ProductVersion:   6.2.9200.16570
    FileVersion:      6.2.9200.16570 (win8_gdr.130327-1526)
    FileDescription:  Direct3D 11 Runtime
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

 

A good guess is that the issue we are encountering is graphics driver related.

I wanted to see if that was the case with Outlook as well. However, that would require a different approach as Outlook was not producing any crash dumps to run through the debugger. Although there are several ways to work around this, the simplest and most GUI friendly is to use DebugDiag from Microsoft. I wrote about this sometime ago and it has since moved to version 2.1 but the approach I still the same for the most part. I simply ran DebugDiag Collection, created a crash rule, specified outlook.exe as the process. After starting Outlook, several dumps were generated. When complete I then launched DebugDiag Analysis, selected the last dump and then selected Start Analysis. The report generates as an htlm file and opens automatically in IE.

The analysis takes some of the guess work of how to proceed when you have little or no understanding of WinDbg. I could see my suspicion that the issue common to both IE and Outlook was the same, which does not surprise me as they both share common modules.

image

As a proof of concept, one way to test issues you suspect may be graphics related to IE if your system uses dedicated graphics is to disable GPU rendering in IE. This is done by going into the Control Panel > Internet Options > Advanced > and checking Use software rendering instead of GPU rendering (note, although the asterisk indicates you should restart your computer, only restarting IE may be required). In my case, this corrected the problem with IE so I was sure now that the issue was related to the graphics card. Before updating the driver, I checked the display information to see if perhaps the drivers were missed and maybe generic drivers were installed. This didn’t seem to be the case:

image

I would still update the drivers anyway. After doing so, both Internet Explorer and Outlook opened without crashing. Interestingly, after updating the drivers, the workstation reported a different video adapter model:

image

In the end, the resolution was to install the correct graphics drivers during the task sequence.

Note, in some cases the crashing application may report via WER (windows error report) popup. If you examine the details, you may note that the fault module name contains a stackhash code a7aa, which is often a indicator that the issue is likely related to a driver issue for the video subsystem:

SNAGHTML31458dc0


*Oddly enough, I could not reproduce the issue when connected remotely using RDP. This should have been a clear indicator as to what the issue really was because RDP does not use the host system’s graphics drivers but those of the connecting system instead. 

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | Leave a Comment »

Using Resource Monitor to Identify an Outlook Hang

Posted by William Diaz on April 8, 2014


Not too long ago I was asked to start looking into then recent reports of Outlook hanging on first launch. The first launch hang symptom happened as mail items were writing into the OST (cached mode) as user’s came in each morning and opened Outlook. Forcefully closing and reopening Outlook was a workaround but the cause needed to be identified as the impact was potentially the entire firm.

After a couple days I began to see a common theme. I was using Resource Monitor (resmon.exe – also accessible from the Task Manager Performance tab) in an attempt to identify the culprit. For hang scenarios, simply recreate the issue, open Resource Monitor, go to the Overview tab and locate the hung process. Typically a hung process will be displayed in red:

image

Right-click the hung process and select Analyze Wait Chain to get a list of the thread IDs. One of these threads will contain the module(s) that is causing the process to become unresponsive.

image

To look inside the threads, you can use Process Explorer. Just right-click on the target process, select Properties and go to the Threads tab. From here, use the TID column to locate the threads from the wait chain provided by resmon. Double-click the thread or select the Stack button to view the loaded modules in the thread. In this case, I saw a consistent stack in each of the hung Outlook processes on the various workstations:

image

There is a single 3rd party module appearing here, everything else is Microsoft. This made sense as this particular module was part of a recent Outlook add-in upgrade. The other threads were also examined but the only 3rd party modules they were referencing were proven addins that had never caused issues. A POC was tested on with several users over the course of a couple weeks by disabling the add-in via load behavior in the registry and issue did not resurface. The issue was raised with the vendor and a newer client for the add-in was provided that resolved this problem.

Another way to identify the cause assuming you have Windows Debugging Tools installed is to generate a dump of the hung process from the Task Manager. A quick !analyze –v –hang pointed to the same culprit (this is a 32bit process from a 64bit OS so you will need to get the 32bit stacks. Use .load wow64exts and .effmach x86 commands before !analyze):

image

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »

More Web Page Troubleshooting with IE Developer Tools

Posted by William Diaz on February 11, 2014


A user was complaining that they could not access a web page. The initial impression was that it was being blocked. I bypassed the proxy mechanism in place that normally restricts web traffic but saw that the page still could not be loaded. The message alluded to something locally installed that may be the culprit, but this was really just a generic “catch-all” error message: “Is feedly blocked? Feedly is not able to load. It is probably because one of your extensions is blocking it…

image

I decided to load up the Developer Tools in IE to see if it was problem with the script in the web page. To do this, just press F12 or go to Tools > F12 Developer Tools. Afterwards, select the Script tab and select the option to Start debugging. I selected the hyperlink in the right portion of the debug window and it took the cursor to the problem part of the javascript code.

image

Don’t worry, you don’t necessarily need to be a program-code debugger to figure this out. That what the Internet and your favorite search engine is for. Searching the debug error Expected identifier, string or number IE9 turned up two top hits that pointed out that a trailing comma after an object or array is a bad idea and that IE9 specifically cannot handle this problem script.

The workaround is to use an alternative browser like FireFox or Chrome (they are not as restrictive with malformed javascript). Internet Explorer 10 or 11 also seems to be handle this problem more gracefully, too, and allowed the page to load normally.

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »

Configure WinHttp for Proxy

Posted by William Diaz on November 3, 2013


Recently, we were testing a new remote desktop application but I was experiencing problems connecting to the application, and it wasn’t isolated to a single workstation. A few of my other labs were unable to access the application as were other computers as the testing expanded. Admittedly, I should have figured this one out quickly as it was not the first time I encountered an issue trying to access remote desktop applications (i.e. terminal server apps) inside our network. The fact that we were hosting this RDP app internally, though, lead me to dismiss my initial hunch. A quick look with Network Monitor confirmed the problem:

  image

The Windows local security authority processes (lsass.exe) is trying to get out to the cloud to do something, in this case likely some certificate revocation checking (this is usually the case for RDP apps). It keeps on trying by retransmitting until a response is received or it times out, but because we are behind a proxy, lsass.exe will never find a path outside the network. Certificate checking is handled by the Windows crypto API, which relies on WinHttp. Now, by default, Winhttp 5.1 can find a way out of the network if your network is configured to use Web Proxy Autodiscovery (WPAD). We do not, so the fix is to manually configure Winhttp to use some proxy. Since we have this information configured in Internet Explorer (along with proxy exceptions), we just need to import these settings into Winhttp via (elevated command prompt) netsh winhttp import proxy source=ie. Afterwards, the connection problem resolved*.

On the flipside, configuring Winhttp to use a static proxy can also cause connectivity problems for mobile users. I ran into this issue myself today when I was trying to stream a movie from Netflix and kept on encountering the following error: “Whoops. something went wrong… An Internet or home connection network connection problem is preventing playbackError code: H7111-1101”

Capture

To verify my suspicion that the opposite was true—that Winhttp was trying to use a proxy it could not reach to do certificate verification checks—I looked in the Windows event CAPI2 logs and could see that the Netflix certificate check was failing:

Capture2

The details pane further down revealed the cause as the certificate server was “offline”, which is just a generic term for can’t be found. You can verify your winhttp proxy via netsh winhttp show proxy. To correct, simply set the winhttp proxy to use direct connection via netsh winhttp reset proxy when outside the network.


*Certificate verification does not need to be performed for every session, i.e. the initial check is good enough until the crypto API determines that the information is expired and another check needs to be performed.

I am not sure why yet, but Netmon captures on a Windows 8.1 system did not reveal the lsass.exe process. Instead, the process was listed as unavailable.

image

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , | Leave a Comment »

Login Issue and Mandatory Profiles

Posted by William Diaz on October 4, 2013


An interesting little quickie. After moving to mandatory profiles in a Citrix environment, a particular ActiveX web application would no longer allow logins. There was no error message of any kind and it continued to work in another Citrix environment without mandatory profiles. I fired up Process Monitor and ran a little trace of Internet Explorer to capture everything that happened after I clicked login. Nothing interesting really stood out but there might be some hope in an activity log activity I saw occurring with the application:

SNAGHTMLab68c01

Opening the log showed:

[W]    2013/10/03 22:46:00 PM                fyiCryptAcquireContext(): CryptAcquireContext() failure while trying to acquire the crypto context/container (GetLastError() -2146893788, (The profile for the user is a temporary profile.)) Thu Oct 03 22:46:00 2013
[I]    2013/10/03 22:46:00 PM                fyiCryptAcquireContext(): CryptAcquireContext() failed while initializing the crypto context (GetLastError()=-2146893788 (The profile for the user is a temporary profile.)), I will try and re/generate a brand new container Thu Oct 03 22:46:00 2013
[E]    2013/10/03 22:46:00 PM                fyiCryptAcquireContext(): CryptAcquireContext() failure, can’t acquire, nor create a new container (2) (GetLastError() -2146893788, (The profile for the user is a temporary profile.)) Thu Oct 03 22:46:00 2013

 

Some quick research pointed me to RSACryptoServiceProvider fails when used with mandatory profiles. In short:

RSACryptoServiceProvider calls CryptAcquireContext API (http://msdn2.microsoft.com/en-us/library/aa379886.aspx) behind the scenes to get a handle to a key container within a CSP (Cryptographic Service Provider). CryptAcquireContext will fail with NTE_TEMPORARY_PROFILE error when called from a mandatory profile.

Mandatory profiles are read-only user profiles. Since changes to the mandatory profile cannot be saved, PKI design doesn’t allow this operation, and CryptAcquireContext prevents this scenario by failing.

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »

Some CCM Client Workstations Failing to Install Updates

Posted by William Diaz on October 2, 2013


While browsing various reports for workstation compliancy, I noticed that several reports and/or updates failed to install on a large number of computers. Although we don’t expect complete 100% compliancy with thousands of workstation in our environment, there was some sort of mystery going on here because many of the updates that failed to install were along the same computers, i.e. they were not just random computers across the various reports. For example, below is a report for various updates that were missing from generally the same number of computers and computer names:

image

After some research in the CCM logs, I noticed a repetitive theme in the C:\Windows\SysWOW64\CCM\Logs\UpdatesDeployment.log: “Install not allow as another job is still in progress”:

image

Using the SCCM Client Center utility, we compared the time in the logs to the Advertisement > Execution History in the SCCM Client Center and saw nothing that was actually trying to install at that time. Out of ideas, I decided to delete root\CCM namespace (also accomplished with the SCCM Client Center utility) on a few of the problem workstations in the reports above. After a few minutes, I noticed the CCM Cache in C:\Windows\Syswow64\CCM\Cache was rebuilt, pulling several pending updates. The next step was to wait for the SCCM service windows to pass. The next day when I came in, I remotely checked the workstation Event Setup logs and saw that several (sometimes dozens) of various pending updates had successfully installed.

Knowing that we had a problem with the CCM namespace, I followed up with some more research. My digging around eventually led me to what might have been an update that was advertised to these computers but likely pulled before it could be deployed. Specifically, what I found was that each update gets an unique update ID. To the CCM agent, this property is known as the AssignmentID, which resides in the instance of the CCM_DeploymentTaskEx1 of the root\CCM\SoftwareUpdates\DeploymentAgent namespace. I went around to several workstation in the SCCM reports and ran wbemtest.exe and saw the same assignmentId(s) across all the computers in the reports:

Read the rest of this entry »

Posted in Troubleshooting, Troubleshooting Tools | Leave a Comment »

The Case of Google Earth Interrupting Internet Connectivity

Posted by William Diaz on September 11, 2013


So today I received another complaint of Google Earth causing Internet Connectivity problems. In the past, I had ignored these because I was never able to recreate the issue as I did not have enough information on the method being used to cause it. This time I was a little bit more determined so I had one of the helpdesk techs show me first hand what was happening and armed with that I went off to install Google Earth on a lab and attempt to recreate. The steps to recreate are rather straight forward, just do a few searches and lots of zooming or scrolling in Google Earth until. At some point, the telltale sign that Internet connectivity had been lost would be indicated when when the Tour Guide pane became blank:

image

When operating normally, the Tour Guide pane displays images relevant to the place you are searching or viewing. At this point, any browser would also fail to connect to any external resource, e.g. the Internet, returning a page not found or other connectivity failure message. After what seemed like a couple minutes, connectivity would then be restored.

Some troubleshooting was already attempted earlier by pointing the browser to an unmanaged (free-for-all) proxy, which avoided the problem. My guess at that point was that our TMG was somehow cutting the connection for the workstation for some amount of time. Why? I also assumed that Google Earth is simple saturating the TMG with too many requests. Think about it, every search, zoom, scroll, and pan is basically a file-image request. Keep on doing that in a short period and you are likely to trigger some hardware or software appliance that a DoS attack is taking place. To backup my hypothesis, I turned to my latest and favorite tool, Network Monitor. I started a trace, reproduced the issue, and stopped the trace. The capture was fairly large so I needed to set a filter. Some quick searching through the standard filters revealed a filter for Http error (Load Filters > HTTP > http Error). After applying the filter, I could see the issue:

image

502 Bad gateway – Proxy Error (The number of HTTP requests per minute exceeded the configured limit). Some quick research pointed to this Forefront TMG article Overview of flood mitigation. In short, TMG rules may have to be modified or created to bypass flood mitigation for Google Earth. Currently, these are the two servers Google recommends for bypass:

  • maps.google.com
  • geoauth.google.com

This is also known to be an issue for home and small offices that are not behind a proxy. In those cases, it is likely the router’s firewall-DoS configuration that is the culprit.

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »