Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Another Case of IE and Outlook Crashes

Posted by William Diaz on February 26, 2015


I haven’t done one of these in quite awhile. So without further ado.

The symptom: Internet Explorer 11 and Outlook were crashing*. This was happening in all cases after applying a new task sequence. The event viewer did not produce any events that would help so I configured the system to capture crash dumps for applications. After rebooting, I reproduced the crash by simply opening IE an grabbed one of the dumps and ran it through an x86 debugger (by default, the 32bit Content tabs in IE 11 run as 32bit processes). Using ! analyze –v command produced the following output:

STACK_TEXT: 
WARNING: Frame IP not in any known module. Following frames may be wrong.
0339e974 6818a97d 005a09f8 0339e994 5226b9f5 0x490054
0339e9d8 68193c6f 0059f730 0059f7ec 5226bb49 D3D10Level9!UMAdapter::UnderlyingGetCaps+0x31
0339eb64 681940c2 0059f730 0339ebb4 00000002 D3D10Level9!UMAdapter::Open+0x165
0339eb98 69e0cae6 0339ebb4 523e5d23 00020009 D3D10Level9!OpenAdapter10_2+0x45
0339ebdc 69de09c2 005a09f8 0059f35c 69dda858 d3d11!NDXGI::CUMDAdapter::CUMDAdapter+0x18b
0339edc8 69de05fd 0339f0f0 00000002 00000029 d3d11!CCreateDeviceCache::CUMDAdapterCache::Load+0x1fc
0339ee18 69de12f3 0339f0b8 0339ee90 00009300 d3d11!CCreateDeviceCache::CAdapterCache::ResolveUMDAndVersion+0xc8
0339f1f4 69de1e4e 00000000 00595190 00000000 d3d11!D3D11CoreCreateDevice+0x353
0339f484 69de1bdd 00595190 00000000 00000000 d3d11!D3D11CreateDeviceAndSwapChain+0x268
0339f4bc 5bd5b4ed 00595190 00000000 00000000 d3d11!D3D11CreateDevice+0x2c
0339f4f8 5bd5c698 00000001 0339f58c 0339f564 mshtml!CDXGIHelper::CreateD3DDevice+0x48
0339f520 5bd5bc31 72657200 001af5f2 005910fc mshtml!CDXHelper::CreateD3D11Device+0x99
0339f6fc 5bd5ba45 00000000 00000000 00000000 mshtml!CDXResourceDomain::EnsureD3DDevice+0x82
0339f718 5bd5b9ad 00000000 00000000 00000000 mshtml!CDXResourceDomain::Initialize+0x67
0339f734 5bb35771 5c81e03c 00000000 00000000 mshtml!CDXResourceDomain::Create+0x48
0339f76c 5bd5a4c9 0339f78c 00000000 00538c78 mshtml!CDXResourceDomain::EnsureSharedDomain+0xa9
0339f7a0 671015ca 670b21cc 00000000 00000000 mshtml!CDXResourceDomain::EarlyStartDisplaySystem+0xe8
0339f7a4 670b21cc 00000000 00000000 00545cf8 ieframe!DirectUI::TouchEdit2::_UpdatePrompt+0x22
0339f7bc 77647df9 00538c78 744d2d1c 0053b958 ieframe!ExecuteWorkItemThreadProc+0x30
0339f830 77632b65 00538c78 00545cf8 744d2cbc ntdll!RtlpTpWorkCallback+0x11d
0339f990 76d6339a 0053b950 0339f9dc 7761bf32 ntdll!TppWorkerThread+0x572
0339f99c 7761bf32 0053b950 744d2cf0 00000000 kernel32!BaseThreadInitThunk+0xe
0339f9dc 7761bf05 776325c1 0053b950 ffffffff ntdll!__RtlUserThreadStart+0x70
0339f9f4 00000000 776325c1 0053b950 00000000 ntdll!_RtlUserThreadStart+0x1b

STACK_COMMAND:  ~2s; .ecxr ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  unknown!printable+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: unknown

IMAGE_NAME:  unknown

DEBUG_FLR_IMAGE_TIMESTAMP:  0

 

I know D3D is Microsoft Direct X. To validate this, simply run the command lmvm d3d11:

0:002> lmvm d3d11
start    end        module name
69dd0000 69f45000   d3d11      (pdb symbols)          c:\symbols\d3d11.pdb\A40CED7361AB405886384123277BE23F1\d3d11.pdb
    Loaded symbol image file: d3d11.dll
    Image path: C:\Windows\System32\d3d11.dll
    Image name: d3d11.dll
    Timestamp:        Wed Mar 27 18:48:45 2013 (5153774D)
    CheckSum:         0017E157
    ImageSize:        00175000
    File version:     6.2.9200.16570
    Product version:  6.2.9200.16570
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     D3D11.dll
    OriginalFilename: D3D11.dll
    ProductVersion:   6.2.9200.16570
    FileVersion:      6.2.9200.16570 (win8_gdr.130327-1526)
    FileDescription:  Direct3D 11 Runtime
    LegalCopyright:   © Microsoft Corporation. All rights reserved.

 

A good guess is that the issue we are encountering is graphics driver related.

I wanted to see if that was the case with Outlook as well. However, that would require a different approach as Outlook was not producing any crash dumps to run through the debugger. Although there are several ways to work around this, the simplest and most GUI friendly is to use DebugDiag from Microsoft. I wrote about this sometime ago and it has since moved to version 2.1 but the approach I still the same for the most part. I simply ran DebugDiag Collection, created a crash rule, specified outlook.exe as the process. After starting Outlook, several dumps were generated. When complete I then launched DebugDiag Analysis, selected the last dump and then selected Start Analysis. The report generates as an htlm file and opens automatically in IE.

The analysis takes some of the guess work of how to proceed when you have little or no understanding of WinDbg. I could see my suspicion that the issue common to both IE and Outlook was the same, which does not surprise me as they both share common modules.

image

As a proof of concept, one way to test issues you suspect may be graphics related to IE if your system uses dedicated graphics is to disable GPU rendering in IE. This is done by going into the Control Panel > Internet Options > Advanced > and checking Use software rendering instead of GPU rendering (note, although the asterisk indicates you should restart your computer, only restarting IE may be required). In my case, this corrected the problem with IE so I was sure now that the issue was related to the graphics card. Before updating the driver, I checked the display information to see if perhaps the drivers were missed and maybe generic drivers were installed. This didn’t seem to be the case:

image

I would still update the drivers anyway. After doing so, both Internet Explorer and Outlook opened without crashing. Interestingly, after updating the drivers, the workstation reported a different video adapter model:

image

In the end, the resolution was to install the correct graphics drivers during the task sequence.

Note, in some cases the crashing application may report via WER (windows error report) popup. If you examine the details, you may note that the fault module name contains a stackhash code a7aa, which is often a indicator that the issue is likely related to a driver issue for the video subsystem:

SNAGHTML31458dc0

*Oddly enough, I could not reproduce the issue when connected remotely using RDP. This should have been a clear indicator as to what the issue really was because RDP does not use the host system’s graphics drivers but those of the connecting system instead. 

Advertisement

Posted in Troubleshooting, Troubleshooting Tools | Tagged: , , | Leave a Comment »

Why is the IE 11 MSI Generated from the IEAK Failing?

Posted by William Diaz on December 8, 2014


There has been some chatter about the Internet Explorer 11 MSI that is generated from the IEAK failing while the EXE is able to run successfully. I ran into this myself last week while trying to manually execute the package. The IE11_main.log always showed the same error: “ERROR:   Error downloading prerequisite file (KB2834140): 0x800c0005 (2148270085)”

image

After a day or so of troubleshooting I realized the problem. When the IE11 setup runs, it needs to go out to the Internet to check Microsoft for prerequisite updates; both the MSI and the EXE do this. But the MSI (msisexec.exe) executes in the context of the local system account. By default, the local system account tries to find a direct path to the Internet, but if you are behind a proxy this is going to fail as the MSI child processes (specifically ienrcore.exe) are bypassing the proxy.

image
The EXE on the other hand when run manually executes in the context of a user account with Internet Access. That does not mean you are entirely out of the woods with the EXE package. If you are automatically deploying and leveraging SCCM the EXE (like the MSI) is still going to executed by the local system account. To overcome this, you need to configure the local system account to use a proxy (or not use a proxy, or update your proxy). This can be done via BITSAdmin with the following command:

bitsadmin /util /setieproxy localsystem MANUAL_PROXY MyProxy:8080 "<local>"

Another workaround is to simply install the prerequisites before installing IE 11. The required KBs are:

  • KB2670838
  • KB2786081
  • KB2834140
  • KB2882822
  • KB2888049

Last, you should also be able to extract the cab files from these KBs and add them via IEAK custom components step in the IEAK. I have yet to get this to work, however, as the package still wants to go out to the Windows Update site. If you have gotten this to work, let me know how you did it.

UPDATE

I have decided to use the IE11 redistributable and avoid IEAK. Injecting updates into the package via IEAK is too much work. Fortunately for us we control everything we need to via group policy so there is no need for customizations. A script to install the prerequisite updates and then IE11 is much simpler.

Note, if you install updates then install IE11 in the same script, the IE11 installer is still going to want to go out to the Internet to check updates if you just run the executable command. There are two ways to workaround this. One is to install the updates, restart, then install IE11. But this means you have to break up your deployment into two steps. Fortunately, you can work around this. The answer lies in the IE11_main.log using DISM. When a successful install (with Internet Connection) of IE11 is completed you can find the command in the log:

image

Just extract the redistributable and then install IE11 via the main CAB using DISM and then the two MSUs using WUSA. For example:

::Create temp directory if not present to extract IE11 
MKDIR "C:\temp\IE11"
ECHO Installing prerequisite updates for Internet Explorer 11
"%~dp0Windows6.1-KB2670838-x64.msu" /quiet /norestart
"%~dp0Windows6.1-KB2786081-x64.msu" /quiet /norestart
"%~dp0Windows6.1-KB2834140-v2-x64.msu" /quiet /norestart
"%~dp0Windows6.1-KB2882822-x64.msu" /quiet /norestart
"%~dp0Windows6.1-KB2888049-x64.msu" /quiet /norestart
ECHO Installing Internet Explorer 11. Please wait…
::A reboot is required here, otherwise the IE11 Installer wants to still go out to Internet to check updates
::We can avoid this by using DISM instead to install IE11
"%~dp0IE11-Windows6.1-x64-en-us.exe" /X:%systemdrive%\temp\IE11
%systemroot%\system32\dism.exe /Online /Add-Package /PackagePath:%systemdrive%\temp\IE-Win7.CAB /quiet /norestart
%systemroot%\system32\wusa.exe "%systemdrive%\temp\IE11\IE-Spelling-en.MSU" /quiet /norestart
%systemroot%\system32\wusa.exe "%systemdrive%\temp\IE11\IE-Hyphenation-en.MSU" /quiet /norestart
::Cleanup 
RMDIR C:\temp\IE11 /s /q

 

Note: if you are going to be using 32bit Configuration Manager client to execute the batch file above in 64bit Windows, you will need to replace system32 paths to sysnative.

Posted in Uncategorized | Tagged: | 6 Comments »

The Case of My Broken App in Windows 8

Posted by William Diaz on August 1, 2014


This is a case where the Application Event logs came in handy when my application started to crash upon loading on our Windows 8.1 images. Since my app was compiled as .NET, the stack in the error details pointed me to problem object:

image

When TextBlock3 loaded it was doing Environment.GetEnvironmentVariable(“SomeVariable”). The variable in question did not exist on our Windows 8.1 images. Although this did not seem to pose an issue for my Windows 7 dev box (it just would not display the TextBlock), Windows 8 will throw an exception if the variable does not exist and the app will crash. The likely cause my dev PC did not encounter this exception when running was because .NET framework 4.5 was not installed whereas .NET 4.5 is the default framework in Windows 8.

Posted in Troubleshooting | Tagged: | Leave a Comment »

Using Resource Monitor to Identify an Outlook Hang

Posted by William Diaz on April 8, 2014


Not too long ago I was asked to start looking into then recent reports of Outlook hanging on first launch. The first launch hang symptom happened as mail items were writing into the OST (cached mode) as user’s came in each morning and opened Outlook. Forcefully closing and reopening Outlook was a workaround but the cause needed to be identified as the impact was potentially the entire firm.

After a couple days I began to see a common theme. I was using Resource Monitor (resmon.exe – also accessible from the Task Manager Performance tab) in an attempt to identify the culprit. For hang scenarios, simply recreate the issue, open Resource Monitor, go to the Overview tab and locate the hung process. Typically a hung process will be displayed in red:

image

Right-click the hung process and select Analyze Wait Chain to get a list of the thread IDs. One of these threads will contain the module(s) that is causing the process to become unresponsive.

image

To look inside the threads, you can use Process Explorer. Just right-click on the target process, select Properties and go to the Threads tab. From here, use the TID column to locate the threads from the wait chain provided by resmon. Double-click the thread or select the Stack button to view the loaded modules in the thread. In this case, I saw a consistent stack in each of the hung Outlook processes on the various workstations:

image

There is a single 3rd party module appearing here, everything else is Microsoft. This made sense as this particular module was part of a recent Outlook add-in upgrade. The other threads were also examined but the only 3rd party modules they were referencing were proven addins that had never caused issues. A POC was tested on with several users over the course of a couple weeks by disabling the add-in via load behavior in the registry and issue did not resurface. The issue was raised with the vendor and a newer client for the add-in was provided that resolved this problem.

Another way to identify the cause assuming you have Windows Debugging Tools installed is to generate a dump of the hung process from the Task Manager. A quick !analyze –v –hang pointed to the same culprit (this is a 32bit process from a 64bit OS so you will need to get the 32bit stacks. Use .load wow64exts and .effmach x86 commands before !analyze):

image

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »

More Web Page Troubleshooting with IE Developer Tools

Posted by William Diaz on February 11, 2014


A user was complaining that they could not access a web page. The initial impression was that it was being blocked. I bypassed the proxy mechanism in place that normally restricts web traffic but saw that the page still could not be loaded. The message alluded to something locally installed that may be the culprit, but this was really just a generic “catch-all” error message: “Is feedly blocked? Feedly is not able to load. It is probably because one of your extensions is blocking it…”

image

I decided to load up the Developer Tools in IE to see if it was problem with the script in the web page. To do this, just press F12 or go to Tools > F12 Developer Tools. Afterwards, select the Script tab and select the option to Start debugging. I selected the hyperlink in the right portion of the debug window and it took the cursor to the problem part of the javascript code.

image

Don’t worry, you don’t necessarily need to be a program-code debugger to figure this out. That what the Internet and your favorite search engine is for. Searching the debug error Expected identifier, string or number IE9 turned up two top hits that pointed out that a trailing comma after an object or array is a bad idea and that IE9 specifically cannot handle this problem script.

The workaround is to use an alternative browser like FireFox or Chrome (they are not as restrictive with malformed javascript). Internet Explorer 10 or 11 also seems to be handle this problem more gracefully, too, and allowed the page to load normally.

Posted in Troubleshooting Tools | Tagged: | Leave a Comment »

Why Aren’t My Windows Audit Policies Working?

Posted by William Diaz on January 31, 2014


So, recently I had the need to setup auditing on a local workstation to try and determine who or what was deleting a specific set of files. Before we started, we decided to test the auditing on a couple PCs to audit all failed and successful attempts to delete any files or folders within some random test folder. Audit events are recorded in the Security logs of the Windows Event Viewer. Specifically, Audit Object Access events of interest are event ID 4656 (A handle to an object was requested) and 4663 (An attempt was made to access an object). The details should allude to the responsible user account and process. But to our surprise our test folder was not recording any audit events for any of the objects inside.

Maybe an oversight but perhaps auditing was not enabled in group policy. But a quick check revealed that it was being set for both success and failure:

image

Maybe it wasn’t getting applied to the workstation. Although deprecated since Vista, RSOP from the command line is useful for quickly spotting polices that might not be getting applied to the local computer. Sure enough I could see there was a problem with the audit policies: “The policy engine did not attempt to configure the setting…”

image

Read the rest of this entry »

Posted in Inside Windows | 1 Comment »

A Typo

Posted by William Diaz on January 30, 2014


image

Posted in Uncategorized | 1 Comment »

The Case of the Ghost Files in IE

Posted by William Diaz on January 24, 2014


While in the process of composing an web-based email today via Internet Explorer, I noticed something odd when I went to attach a file to the message. In the IE file upload dialog there were a series of files listed on the desktop that I knew were not really there. Judging by the dates they had been hanging around for some time now but I had never noticed them (and, no, they were not hidden or marked as protected operating system files):

SNAGHTML1303b167

The files themselves were no mystery; they are nothing more than text files that dump process, thread, and stack information to the desktop when the JRE crashes. I tried to attach one of these to the web message but nothing would happen. However, from within in the dialog box I could right click and open them like any other files. Attempting to move the file also failed because the file itself apparently didn’t exist:

SNAGHTML1319e5a1

If that was the case, I should then get the same message trying to copy it back to the desktop. Instead, the dialog this time indicated it did exist:

SNAGHTML131cc2f5

I wondered if I could access these ghost files from outside IE by, for example, attaching them to a message in Outlook. Not surprisingly, the same set of files did not appear from the desktop location:

SNAGHTML132052f4

This was likely something specific to IE. To determine where the files were actually residing, I turned to Process Monitor, started a trace for file activity and proceeded to open one of the ghost files via the IE File upload dialog box. Afterwards, I stopped the trace and did a search for the file in question.

image

There was half the mystery, the files actually resided in C:\Users\username\AppData\Local\Microsoft\Windows\Temporary Internet Files\Virtualized\C\Users\username\Desktop. The other half of the mystery was then just a matter of some quick research into IE and Virtualized folders. That lead me to this MSDN article Understanding and Working in Protected Mode Internet Explorer. In short:

A Compatibility Layer handles the needs of many existing extensions. It intercepts attempts to write to medium integrity resources, such as the Documents folder in the user profile and the HKEY_CURRENT_USER registry hive. However it will not intercept writes to system locations like Program Files and HKEY_LOCAL_MACHINE. The compatibility layer uses a Windows Compatibility Shim to automatically redirect these operations to the following low integrity locations:

  • Documents and Settings\%userprofile%\LocalSettings\TemporaryInternet Files\Virtualized (pre-Vista)

If I unchecked Enable Protected Mode in IE the virtual files no longer appeared in the file IE file dialog. Not that you don’t want to do that, just delete the files instead.

Posted in Troubleshooting | Tagged: | Leave a Comment »

“The User Profile Service service failed the logon”

Posted by William Diaz on January 14, 2014


 

Today, I encountered the following on a newly imaged workstation: “The user Profile Service service failed the logon. User profile cannot be loaded.”

User-Profile

Often times when I encounter this it’s a simple matter of hacking the registry to fix it. This is covered in detail in this Microsoft KB You receive a "The User Profile Service failed the logon” error message. It also covered in one of my older (pre-MS KB) blogs.

The article, however, doesn’t cover the other scenario. Local admin and domain techs accounts could logon but not standard users accounts. This is often due to file permissions. For whatever reason, a system-admin protected file was created deep into the profile of the Default user account. This is where all new profiles are created from, and Everyone\Users must have at least read permissions for a new profile to be successfully spawned from the Default. The details are alluded to in the Windows Application event logs, event ID 1509:

Windows cannot copy file \\?\C:\Users\Default\AppData\Local\Microsoft\Windows\Temporary Internet Files\SQM\iesqmdata_setup0.sqm to location \\?\C:\Users\TEMP\AppData\Local\Microsoft\Windows\Temporary Internet Files\SQM\iesqmdata_setup0.sqm. This error may be caused by network problems or insufficient security rights.

DETAIL – Access is denied.

image

Looking at the problem file, we can see that the file has no permissions for Everyone\Users:

SNAGHTML566fb99

The solution is to delete the file (if non-essential) or add the correct user(s) and permissions. And depending on the location, you may need to uncheck Hide protected operation system files in Windows Explorer.

Posted in Troubleshooting | Leave a Comment »

USB BitLocker Policy May Not Apply To Some Android Devices

Posted by William Diaz on November 21, 2013


During recent BitLocker piloting for USB mass storage devices we noticed that Android devices were not affected by our Bitlocker policy, which was to prevent writing of data if the device was not encrypted. Although we do not intend to enforce encryption on these devices for obvious reason, we still wanted to be able to deny write to the devices in the same manner that users could not write to their BlackBerry or iPhone devices where USB BitLocker policy was to deny write was working as expected.

After opening a case with Microsoft, the conclusion was that even though the Android devices were mass storage devices, they were presenting themselves as Windows Portable Devices:

>>  [Device Install (Hardware initiated) – WpdBusEnumRoot\UMB\2&37c186b&0&STORAGE#VOLUME#_??_USBSTOR#DISK&VEN_SMI&PROD_USB_DISK&REV_1100#7&2DEFC6D0&0#]
>>>  Section start 2013/11/12 14:30:18.871
     ump: Creating Install Process: DrvInst.exe 14:30:18.871
     ndv: Retrieving device info…
     ndv: Setting device parameters…
     ndv: Doing WU search last due to CM_DEVCAP_SILENTINSTALL flag.
     ndv: Searching Driver Store and Device Path…
     dvi: {Build Driver List} 14:30:18.949
     dvi:      Searching for compatible ID(s):
     dvi:           wpdbusenum\fs
     …

     dvi:           DrvDate      – 06/21/2006
     dvi:           Version      – 6.1.7600.16385
     inf:      Searched 1 potential matches in published INF directory
     inf:      Searched 35 INFs in directory: ‘C:\Windows\inf’
     dvi: {Build Driver List – exit(0x00000000)} 14:30:19.074
     ndv: Selecting best match from Driver Store (including Device Path)…
     dvi: {DIF_SELECTBESTCOMPATDRV} 14:30:19.074
     dvi:      Using exported function ‘WpdClassInstaller’ in module ‘C:\WINDOWS\system32\wpd_ci.dll’.
     dvi:      Class installer == wpd_ci.dll,WpdClassInstaller
     dvi:      No CoInstallers found
     dvi:      Class installer: Enter 14:30:19.089
     dvi:      Class installer: Exit
     dvi:      Default installer: Enter 14:30:19.089

 

A normal USB mass storage device, e.g. a flash drive, otherwise presents itself in manner that USB BitLocker policy could be applied to because the device is recognized as mass storage:

>>>  [Device Install (Hardware initiated) – STORAGE\Volume\_??_USBSTOR#Disk&Ven_SMI&Prod_USB_DISK&Rev_1100#7&2defc6d0&0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}]
>>>  Section start 2013/11/12 14:30:16.765
     ump: Creating Install Process: DrvInst.exe 14:30:16.796
     ndv: Retrieving device info…
     ndv: Setting device parameters…
     ndv: Doing WU search last due to CM_DEVCAP_SILENTINSTALL flag.
     ndv: Searching Driver Store and Device Path…
     dvi: {Build Driver List} 14:30:16.812
     dvi:      Searching for hardware ID(s):
     dvi:           storage\volume
…
dvi:           DrvDate      – 06/21/2006
     dvi:           Version      – 6.1.7601.17567
     inf:      Searched 1 potential matches in published INF directory
     inf:      Searched 35 INFs in directory: ‘C:\Windows\inf’
     dvi: {Build Driver List – exit(0x00000000)} 14:30:16.937
     ndv: Selecting best match from Driver Store (including Device Path)…
     dvi: {DIF_SELECTBESTCOMPATDRV} 14:30:16.937
     dvi:      Using exported function ‘VolumeClassInstaller’ in module ‘C:\WINDOWS\system32\SysClass.dll’.
     dvi:      Class installer == SysClass.dll,VolumeClassInstaller
     dvi:      Using exported function ‘CriticalDeviceCoInstaller’ in module ‘C:\WINDOWS\system32\SysClass.Dll’.
     dvi:      CoInstaller 1 == SysClass.Dll,CriticalDeviceCoInstaller

This information is written each time you connect a USB device to a Windows computer and can be found in C:\Windows\inf\setupapi.dev.log.

The workaround is to apply a Deny Write Access to WPD devices in Computer Configuration\Administrative Templates\System\Removable Storage Access.

Posted in Uncategorized | Leave a Comment »

« Previous Entries
Next Entries »