Windows Explored

Everyday Windows Desktop Support, Advanced Troubleshooting & Other OS Tidbits

Archive for October, 2018

Spinning Dots After Windows 10 Updates

Posted by William Diaz on October 23, 2018


Recently, we began to push a new batch of Windows 10 updates to several offices. The morning after, we started to receive complaints that several computers were hung up at what is often described as a “black screen with spinning white dots”.

Obviously, something had gone wrong after the updates were installed to the first batch of deployment offices. In some cases, psychically powering down the workstation seemed to correct from a brief time, but after another reboot the problem would come back*.  Oddly enough, we were able to RDP to all the affected workstations. RDP does not use the video display system of the computer you are connecting to render the display experience, but that of the computer that is doing the connection instead. This allowed us to uninstall the latest update(s) in an attempt to resolve. Unfortunately, this made no difference. Whatever had changed, had permanently affected the computers video display subsystem.

We ended up pulling the update deployment and I had one of the problem computers sent to me so I could look at more closely. My theory was that the Windows logon process was getting injected by something or otherwise was hooked by a process and it was preventing the user logon interface from displaying on the monitor. If that was the case, it should be pretty easy to spot.

I connected via RDP and fire up Process Explorer, and went to the winlogon.exe process to take a look. Sure enough, the issue presented itself:

During the boot phase and before the logon experience is ready, Windows will display the spinning dots as an indication that things are starting up. After that, winlogon.exe launches and begins the process of presenting the UI for logon. You can see here that LogonUI.exe (the actual logon interface) kicks off. The fact that there were dozens of LogonUI.exe processes running was an indication that this process was failing and winlogon was retrying after each failed instance. Looking inside one of the LogonUI processes showed that there was only a single thread and it was never fully initializing.

To see what was injecting into a process can be done a few different ways. The quickest is to use Process Explorer, go to View > check Show Lower Pane then View again, Lower Pane View > DLLs. In this case, I could see 3rd party hooking by various Citrix processes:

Alternatively, you can dump out the process by right-clicking on from Process Explorer or Task Manager and open it in WinDbg. Running a hang analysis on it (!analyze -v -hang) would venture a best guess from the debugger:

If you follow the stack trace up from the Citrix Hook you can see it end at initialization for gdi32full.dll, which is the Windows API that is leveraged by applications to do any number of graphical display work. In this case, this instance of Citrix hooking is the result of the Citrix VDA client being installed. The VDA client allows users to remotely access their office computers over the Internet via a Citrix login portal. However, in order to display the logon UI to remote user, it needs to hook the windows logon process.

As a proof of concept, I disabled hooking of the winlogon.exe process by the Citrix PortICA Winlogon dll, specifically PicaWinlogonHook64.dll. This can be done via the registry by going to HKLM\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\PicaWinlogonHook and setting the DWORD Flag to 0 and restarting the computer (great article here). Upon doing so, the spinning dots issue corrected itself. Note, this itself is not the solution; setting this causes the Windows logon screen to not appear for anyone trying to remotely connect using the company Citrix login – VDA (they will get a grey screen).

After some more research, I found I couple Citrix KBs eluding to this issue and that this was specific to VDA client versions 7.14 (Citrix Virtual Delivery Agent 7.14.1) & 7.15 CU1 (Citrix 7.15 LTSR), which was about half our VDA clients in our environment. Thankfully, it did not affect VDA version 7.15 CU2, which was the other half.

Over the next few weeks we started the process of updating the VDA clients, which in itself became somewhat of a challenge. Ultimately, we ended up doing this part of the update as a task sequence, which ran the VDA cleanup utility in unattended mode (the native uninstall command was failing for some unknown reason in most cases), restarted the machine and then installed 7.15 CU2.

A guide for using the VDA cleanup tool can be found in this Citrix KB: How to Run the VDA Cleanup Utility with SCCM Task Sequences


*Of note, it was found that this issue could also be overcome by running windows Disk Cleanup as an admin and selecting the option Windows Update Cleanup. This has the effect of removing the various updated Windows files (otherwise, they are archived). The downside to this is that you would not be able to remove any problematic update going forward to restore the system to a previous stable point. It also did not guarantee that the issue would not resurface with the next batch of updates … not to mention it was taking 1 hour plus to complete in some cases and was not 100%. I’m still curious why it was working in many cases, though.

Advertisements

Posted in Inside Windows, Troubleshooting, Troubleshooting Tools | Leave a Comment »

IE & The Case of the 65,535 temp files

Posted by William Diaz on October 23, 2018


This is a bit old but I came across the screenshots I captured on one of my soon-to-be-retired workstations and I was reminded that I forgot to document this one.

The problem came about as a run-of-the-mill IE performance “very slow” escalations. Killing IE and restarting it did not resolve and it would consistently happen each time. After connecting to the workstation, I fired up Process Monitor and captured about 30 seconds worth of activity and then stopped the capture. I applied a filter for iexplore.exe and began to scroll down the almost 1 million operations generated in that short capture: A sore thumb stood out. Almost all the operations being performed by IE were file operations specific to temp files in the C:\Users\username\AppData\LocalLow\DLPTemp directory. Looking in that directory, I could see thousands of htp*.tmp files. In this directory was an additional sub-folders named Logs that contained a single log file, HDLP_IE.log. Opening it up revealed what seemed to be the problem: [10744] [OERROR] [Explorer Probe] [McAfee::DLP::HTTP::internals::processor_base_t::make_temp_file] Cannot generate a temporary file path; error=80 Unless you have deep-dived into Windows file system, its not common knowledge that Windows places a limitation on the number of temp files (.tmp) that can be generated into a single folder directory. That limitation is 65,535, and that limitation had already been reached. The issue was specific to only a handful of users, mostly those working in large document databases via the browser. The McAfee DLP plugin for IE “mirrors” these files as .tmp files for content analysis. Normally, these files should be cleaned up by DLP agent and for whatever reason they this was not working. Some quick research pointed to the cause and McAfee lays out the issue here https://kc.mcafee.com/corporate/index?page=content&id=KB84595&actp=null&viewlocale=en_US. The quick fix was to manually delete all the temp files in the directory until workstations could be updated to latest DLP Endpoint.

Posted in Inside Windows, Troubleshooting | Leave a Comment »