Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Archive for the ‘Inside Windows’ Category

Spinning Dots After Windows 10 Updates

Posted by William Diaz on October 23, 2018


Recently, we began to push a new batch of Windows 10 updates to several offices. The morning after, we started to receive complaints that several computers were hung up at what is often described as a “black screen with spinning white dots”.

Obviously, something had gone wrong after the updates were installed to the first batch of deployment offices. In some cases, psychically powering down the workstation seemed to correct from a brief time, but after another reboot the problem would come back*.  Oddly enough, we were able to RDP to all the affected workstations. RDP does not use the video display system of the computer you are connecting to render the display experience, but that of the computer that is doing the connection instead. This allowed us to uninstall the latest update(s) in an attempt to resolve. Unfortunately, this made no difference. Whatever had changed, had permanently affected the computers video display subsystem.

We ended up pulling the update deployment and I had one of the problem computers sent to me so I could look at more closely. My theory was that the Windows logon process was getting injected by something or otherwise was hooked by a process and it was preventing the user logon interface from displaying on the monitor. If that was the case, it should be pretty easy to spot.

I connected via RDP and fire up Process Explorer, and went to the winlogon.exe process to take a look. Sure enough, the issue presented itself:

During the boot phase and before the logon experience is ready, Windows will display the spinning dots as an indication that things are starting up. After that, winlogon.exe launches and begins the process of presenting the UI for logon. You can see here that LogonUI.exe (the actual logon interface) kicks off. The fact that there were dozens of LogonUI.exe processes running was an indication that this process was failing and winlogon was retrying after each failed instance. Looking inside one of the LogonUI processes showed that there was only a single thread and it was never fully initializing.

To see what was injecting into a process can be done a few different ways. The quickest is to use Process Explorer, go to View > check Show Lower Pane then View again, Lower Pane View > DLLs. In this case, I could see 3rd party hooking by various Citrix processes:

Alternatively, you can dump out the process by right-clicking on from Process Explorer or Task Manager and open it in WinDbg. Running a hang analysis on it (!analyze -v -hang) would venture a best guess from the debugger:

If you follow the stack trace up from the Citrix Hook you can see it end at initialization for gdi32full.dll, which is the Windows API that is leveraged by applications to do any number of graphical display work. In this case, this instance of Citrix hooking is the result of the Citrix VDA client being installed. The VDA client allows users to remotely access their office computers over the Internet via a Citrix login portal. However, in order to display the logon UI to remote user, it needs to hook the windows logon process.

As a proof of concept, I disabled hooking of the winlogon.exe process by the Citrix PortICA Winlogon dll, specifically PicaWinlogonHook64.dll. This can be done via the registry by going to HKLM\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\PicaWinlogonHook and setting the DWORD Flag to 0 and restarting the computer (great article here). Upon doing so, the spinning dots issue corrected itself. Note, this itself is not the solution; setting this causes the Windows logon screen to not appear for anyone trying to remotely connect using the company Citrix login – VDA (they will get a grey screen).

After some more research, I found I couple Citrix KBs eluding to this issue and that this was specific to VDA client versions 7.14 (Citrix Virtual Delivery Agent 7.14.1) & 7.15 CU1 (Citrix 7.15 LTSR), which was about half our VDA clients in our environment. Thankfully, it did not affect VDA version 7.15 CU2, which was the other half.

Over the next few weeks we started the process of updating the VDA clients, which in itself became somewhat of a challenge. Ultimately, we ended up doing this part of the update as a task sequence, which ran the VDA cleanup utility in unattended mode (the native uninstall command was failing for some unknown reason in most cases), restarted the machine and then installed 7.15 CU2.

A guide for using the VDA cleanup tool can be found in this Citrix KB: How to Run the VDA Cleanup Utility with SCCM Task Sequences


*Of note, it was found that this issue could also be overcome by running windows Disk Cleanup as an admin and selecting the option Windows Update Cleanup. This has the effect of removing the various updated Windows files (otherwise, they are archived). The downside to this is that you would not be able to remove any problematic update going forward to restore the system to a previous stable point. It also did not guarantee that the issue would not resurface with the next batch of updates … not to mention it was taking 1 hour plus to complete in some cases and was not 100%. I’m still curious why it was working in many cases, though.

Advertisements

Posted in Inside Windows, Troubleshooting, Troubleshooting Tools | Leave a Comment »

IE & The Case of the 65,535 temp files

Posted by William Diaz on October 23, 2018


This is a bit old but I came across the screenshots I captured on one of my soon-to-be-retired workstations and I was reminded that I forgot to document this one.

The problem came about as a run-of-the-mill IE performance “very slow” escalations. Killing IE and restarting it did not resolve and it would consistently happen each time. After connecting to the workstation, I fired up Process Monitor and captured about 30 seconds worth of activity and then stopped the capture. I applied a filter for iexplore.exe and began to scroll down the almost 1 million operations generated in that short capture: A sore thumb stood out. Almost all the operations being performed by IE were file operations specific to temp files in the C:\Users\username\AppData\LocalLow\DLPTemp directory. Looking in that directory, I could see thousands of htp*.tmp files. In this directory was an additional sub-folders named Logs that contained a single log file, HDLP_IE.log. Opening it up revealed what seemed to be the problem: [10744] [OERROR] [Explorer Probe] [McAfee::DLP::HTTP::internals::processor_base_t::make_temp_file] Cannot generate a temporary file path; error=80 Unless you have deep-dived into Windows file system, its not common knowledge that Windows places a limitation on the number of temp files (.tmp) that can be generated into a single folder directory. That limitation is 65,535, and that limitation had already been reached. The issue was specific to only a handful of users, mostly those working in large document databases via the browser. The McAfee DLP plugin for IE “mirrors” these files as .tmp files for content analysis. Normally, these files should be cleaned up by DLP agent and for whatever reason they this was not working. Some quick research pointed to the cause and McAfee lays out the issue here https://kc.mcafee.com/corporate/index?page=content&id=KB84595&actp=null&viewlocale=en_US. The quick fix was to manually delete all the temp files in the directory until workstations could be updated to latest DLP Endpoint.

Posted in Inside Windows, Troubleshooting | Leave a Comment »

The Case of the Broken Windows Photo Viewer

Posted by William Diaz on September 9, 2016


A co-worker of mine was troubleshooting an issue where a user was unable to open images in the Windows Photo Viewer. Whenever she would try to view an image, the Photo Viewer would open but the image would never load; instead the image was replaced by the spinning blue circle. There was no issue opening the image with other image viewers. To troubleshoot, I fired up Process Monitor and captured a trace of the activity for a few seconds while trying to open the image.

Before starting, its worth noting that the Windows Photo Viewer is not an exclusive process that runs out of C:\Program Files (x86)\Windows Photo Viewer. It is actually executed by dllhost.exe, a host processes that is used by other processes (in this case explorer.exe) to launch code. If something were to go awry with hosted code, then it would only affect that instance of the host process and not explorer, which might crash the desktop.

After a few seconds of capture, I stopped collecting activity, went to Tools > Count Occurrences > Column: Result. My guess was that the Windows Photo Viewer was looking for something that it needed to properly load the images but could not find it. I selected the PATH NOT FOUND value and applied it. There I could see that dllhost.exe was unable to locate some files in C:\Windows\System32\spool\drivers\color.

image

I looked at a couple other systems and saw that this folder existed and was populated with the missing files the COM surrogate was looking for. I wasn’t sure how, why, who deleted this folder but I copied it to the problem computer, opened an image with Windows Photo Viewer and it was working.

Posted in Inside Windows, Troubleshooting | Tagged: | Leave a Comment »

The Case of the Phantom Proxy Settings

Posted by William Diaz on December 3, 2015


Not too long ago we began to notice that users who were opening IE after logging onto some workstations or Citrix servers they did not have a previously existing profile on were getting proxy settings applied in IE that were not being defined in any group policy object. Our local office proxies are applied later in the GPO processing so if it was anything coming before that, it should have been overridden by local GPO. The quick remedy was simply wait for policy to apply in the background or manually run gpupdate. But we still needed to address the cause as in some cases the proxy server was no longer reachable, preventing users from getting out to the Internet.

To get an idea of what might be happening, we deleted a profile from a machine and logged onto it. We then fired up Process Monitor and left it running while launching Internet Explorer for the first time. After verifying that incorrect proxy setting was getting applied, we stopped the trace and the log was sent my way. Proxy settings are applied in the registry so I knew I would be looking only at registry events, searching for the string of proxy server name in the log, and specifically looking at the operation for RegSetValue. From here, it was a simple matter of opening the Properties for that Event and going to the Stack tab.

clip_image001

The component setting the phantom proxy server (iedkcs.dll) was coming from the Internet Explorer Admin Kit or IEAK. This is otherwise known as Active Setup and its what IE will use when it is initially launched by a new user logon if the IE install was configured using IEAK. Also known as Branding, Active Setup for IE looks for INSTALL.INS if it defined in the registry. You can find this file in Program Files (x86)\Internet Explorer\CUSTOM. Looking in the INSTALL.INS, I could see under [ConnectionSettings] the phantom proxy server. You can read about the internals of this here: https://support.microsoft.com/en-us/kb/2029043

At first this was a bit baffling because I had setup the rollout for IE11 using the standalone IE11 MSU and required dependencies and not the IEAK. However, after glancing at the INSTALL.INS file again, I noticed this was actually coming from an earlier IE 9 deployment, which was created using IEAK by my IE predecessor:

image

To correct, I referred back to https://support.microsoft.com/en-us/kb/2029043 and concluded that this could be avoided by going to [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\>{2A2F9EE5-7FE3-44E8-AD75-72B5704DBCB4}] and deleting "StubPath"="RunDLL32 IEDKCS32.DLL,BrandIE4 CUSTOM".

To correct on new workstations going forward, I simply created a REG DELETE command in the Task Sequence after the IE11 setup to remove the value. For existing machines, a GPP registry is the easiest way to go.

Posted in Inside Windows, Troubleshooting Tools | Tagged: | Leave a Comment »

Why Aren’t My Windows Audit Policies Working?

Posted by William Diaz on January 31, 2014


So, recently I had the need to setup auditing on a local workstation to try and determine who or what was deleting a specific set of files. Before we started, we decided to test the auditing on a couple PCs to audit all failed and successful attempts to delete any files or folders within some random test folder. Audit events are recorded in the Security logs of the Windows Event Viewer. Specifically, Audit Object Access events of interest are event ID 4656 (A handle to an object was requested) and 4663 (An attempt was made to access an object). The details should allude to the responsible user account and process. But to our surprise our test folder was not recording any audit events for any of the objects inside.

Maybe an oversight but perhaps auditing was not enabled in group policy. But a quick check revealed that it was being set for both success and failure:

image

Maybe it wasn’t getting applied to the workstation. Although deprecated since Vista, RSOP from the command line is useful for quickly spotting polices that might not be getting applied to the local computer. Sure enough I could see there was a problem with the audit policies: “The policy engine did not attempt to configure the setting…

image

Read the rest of this entry »

Posted in Inside Windows | 1 Comment »

TEMP Profiles vs Username.domain Profiles

Posted by William Diaz on February 7, 2013


There is often confusion between the two. But knowing what each profile type means is an important step in understanding what is going wrong with the profile.

TEMP profiles

These are often created when the local profile and the roaming profile are not the same. For example, a user logs on to their normal workstation and as the local profile is compared against the roaming profile, some file is missing, or some folder or file security rights in either of the profiles have been modified that do not allow it to be read or written so that both the profiles cannot be merged. An example of mismatching profile can be found here.

Another cause of TEMP profiles is when a local profile file is opened (or gets “hooked”) by some system or application process before logon or during logon, preventing, for example, ntuser.dat from opening. An example of this can be found here; and you can also recreate this by remotely opening a handle to the ntuser.dat of any user profile by using the remote Registry > Load Hive in regedit and then trying to log on as that person on the workstation which you loaded their hive. I was both victim and the offender in this example here. In both these cases TEMP profiles protect the integrity of the local profile.

And finally, TEMP profiles are also created when the local user profile folder is renamed or removed but the registry profile remains intact. A registry profile on the affected system means the user has at some point logged on to that workstation but Windows cannot locate the folder profile defined in the ProfileImagePath string of the registry profile (more on that below).

Posted in Inside Windows | Leave a Comment »

Installing Active Directory and All Those Other Little Tabs in Windows 7 (you know, the ones you used to have in Windows XP)

Posted by William Diaz on October 23, 2012


 

One of the articles I put together for our KB. Thought I would share since there seems to be some questions as to how to install ADUC get all the tabs back that were available in Windows XP.

Active Directory Users and Computers Setup:

  • Install the Remote Server Admin Tools by visiting http://www.microsoft.com/en-us/download/details.aspx?id=7887. Choose the package that applies to your OS bitness.
    1. Enable the RSAT features for ADUC by going to the Control Panel and select Programs > Programs and Features > Turn Windows features on or off. Alternatively, you can also click Start and start typing in Turn Windows features on or off and select the option from the menu.
    2. Scroll down to and expand Remote Server Administration Tools.
    3. Expand Role Administration Tools.
    4. Expand AD DS and AD LDS Tools.
    5. Expand AD DS Tools
    6. Check option for Active Directory Administrative Center. This will automatically check all the required options in the parent and sub tree.
    7. Click OK.
    8. You can access ADUC via several methods.
      • Start Orb > dsa.msc.
      • Open an MMC console and add the snap-in.
      • Control Panel > Administrative Tools.
      • Start > All Programs > Administrative Tools (After it has been indexed by Windows).
  • The Exchange tabs will not be included by default. To have those, install the Exchange System Management tools. See below.

Exchange System Management Setup:

  1. Go to http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=17836 and download the Exchange System Manager for Windows Vista.
  2. Run ESMVISTA.EXE to unpack the installer.
  3. Open CMD from an elevated command prompt
  4. Change command prompt to location of the ESMVISTA.MSI file, e.g. cd c:\users\username\desktop.
  5. Run the msi file with the /q switch to avoid message that you are not running Windows Vista, i.e. esmvista.msi /q (extension necessary because there will also be the .exe in the same path).
  6. Exchange System Manager can be found under Start > All Programs > Microsoft Exchange.

Adding the Exchange tabs for user properties:

  • You may notice that the Exchange tabs are missing from user properties in ADUC. This is because the console is running in 64 bit mode and does not plugin into the 32 bit Exchange System Manager. To overcome this:
    1. Right-click ADUC select Send To > Desktop (Create Shortcut)*.
    2. Right-click the shortcut and go to Properties.
    3. Change the target path to include -32
  • If you prefer creating an MMC with multiple snap-ins such as ADUC and ESM:
    1. Go to Start Orb > Search\Run (or Windows+R).
    2. Type mmc -32 and launch.**
    3. Go to File > Add/Remove Snap-In.
    4. Add ADUC, ESM, and whatever additional snap-ins you need.
    5. Save.

*Alternatively, you can run dsa.msc -32 from Run (Windows+R) or the Start Orb > Search/Run

**You must run the mmc with the 32 bit switch, otherwise the 32 bit Exchange System Management add-in will not be available in the list of snap-ins.

Adding the Terminal Service tab for user Properties:

  • Go to Turn Windows features on or off
  • Expand Remote Server Administration Tools > Role Administration Tools > and check Remote Desktop Services Tools.
  • This will also add the Sessions and Environment tabs.

Note, the Terminal Service tab is now known as the Remote Desktop Services Profile tab. Further, you will not see this tab if you run dsa.msc or your mmc in 32 bit mode so consider creating two shortcuts with and without the 32 bit switch.

The Missing Dial-In Tab

This tab is not available by design in ADUC via Windows 7. To workaround this, You need to connect to a Windows Server 2008 Server via RDP that has either the Terminal Services or Remote Desktop Service role installed. However, there is a simpler workaround to the missing tab.

  1. Open ADUC or dsa.msc.
  2. Go to View and check option for Advanced Features.
  3. Right-click the user account and go to the Attribute Editor tab.
  4. Locate and select the msNPAllowDialin attribute.
  5. Select edit and change the value to the desired value.

Posted in Inside Windows | Leave a Comment »

No, Its Not Malware

Posted by William Diaz on July 5, 2012


I was recently asked by a tech to look at someone’s system because they were concerned it may have malware installed in it. The reason for their suspicion was because after running Process Explorer they saw that Internet Explorer was being run as a child process of one of the Windows service host processes, svchost.exe:

image

A quick search on the Internet by the tech lead to a lot of hits referring to malware infection of some sort, which is what originally raised his suspicion. For example,  searching “svchost.exe launches iexplore.exe” returns on the first page numerous hits pointing to malware:

image

A more reasonable explanation for why you sometimes see the Windows service host hosting iexplore.exe is because you likely have launched IE via a URL shortcut before opening IE itself. This is because the URL shortcut isn’t a program itself and needs to leverage the DCOM Server Process Launcher service inside svchost.exe to open Internet Explorer.

image

Posted in Inside Windows | Tagged: , | Leave a Comment »

When You Can No Longer Paste From or To that Remote Desktop

Posted by William Diaz on April 19, 2012


Most of the day I am connected to a lab or another workstation through Remote Desktop. Often times I need to copy (or cut) and paste from the remote desktop session. And sometimes this just straight out fails. When this happens, it’s usually just a simple matter of killing the rdpclip.exe process on the remote system and restarting it. If I am unable to copy (or cut) to the remote session, then I make sure to check that rdpclip process in running. If it is, then I need to close the session and reconnect to correct.

Often, the issue being encountered is caused by the so-called viewer chain. I found an old MSDN blog on the issue here: Why does my shared clipboard not work? (Part 1) & Why does my shared clipboard not work? (Part 2). Vista and up have done more to mitigate these issues.

As for why the clipboard stops working locally, it a shared service so probably some application has it opened and will not let go. In that case, your best bet is to start closing suspect applications until it works again (no straight forward ways I know of for identifying the guilty process).

Posted in Inside Windows | Leave a Comment »

Resolving Missing or Stubborn System Tray Icons

Posted by William Diaz on April 4, 2012


A co-worker asked me a question recently after a user he was talking with put him on the spot by mentioning that one of the icons in the Windows System Tray, specifically the McAfee tray icon, had gone missing. She thought something was awry and wondered if something was done overnight to her workstation to make it disappear or if we had suddenly decided to get rid of it.

Missing:
image
Present:
image

Posted in Inside Windows | Leave a Comment »