Windows Explored

Recently, we began to push a new batch of Windows 10 updates to several offices. The morning after, we started to receive complaints that several computers were hung up at what is often described as a “black screen with spinning white dots”.

Obviously, something had gone wrong after the updates were installed to the first batch of deployment offices. In some cases, psychically powering down the workstation seemed to correct from a brief time, but after another reboot the problem would come back*.  Oddly enough, we were able to RDP to all the affected workstations. RDP does not use the video display system of the computer you are connecting to render the display experience, but that of the computer that is doing the connection instead. This allowed us to uninstall the latest update(s) in an attempt to resolve. Unfortunately, this made no difference. Whatever had changed, had permanently affected the computers video display subsystem.

We ended up pulling the update deployment and I had one of the problem computers sent to me so I could look at more closely. My theory was that the Windows logon process was getting injected by something or otherwise was hooked by a process and it was preventing the user logon interface from displaying on the monitor. If that was the case, it should be pretty easy to spot.

I connected via RDP and fire up Process Explorer, and went to the winlogon.exe process to take a look. Sure enough, the issue presented itself:

During the boot phase and before the logon experience is ready, Windows will display the spinning dots as an indication that things are starting up. After that, winlogon.exe launches and begins the process of presenting the UI for logon. You can see here that LogonUI.exe (the actual logon interface) kicks off. The fact that there were dozens of LogonUI.exe processes running was an indication that this process was failing and winlogon was retrying after each failed instance. Looking inside one of the LogonUI processes showed that there was only a single thread and it was never fully initializing.

To see what was injecting into a process can be done a few different ways. The quickest is to use Process Explorer, go to View > check Show Lower Pane then View again, Lower Pane View > DLLs. In this case, I could see 3rd party hooking by various Citrix processes:

Alternatively, you can dump out the process by right-clicking on from Process Explorer or Task Manager and open it in WinDbg. Running a hang analysis on it (!analyze -v -hang) would venture a best guess from the debugger:

If you follow the stack trace up from the Citrix Hook you can see it end at initialization for gdi32full.dll, which is the Windows API that is leveraged by applications to do any number of graphical display work. In this case, this instance of Citrix hooking is the result of the Citrix VDA client being installed. The VDA client allows users to remotely access their office computers over the Internet via a Citrix login portal. However, in order to display the logon UI to remote user, it needs to hook the windows logon process.

As a proof of concept, I disabled hooking of the winlogon.exe process by the Citrix PortICA Winlogon dll, specifically PicaWinlogonHook64.dll. This can be done via the registry by going to HKLM\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\PicaWinlogonHook and setting the DWORD Flag to 0 and restarting the computer (great article here). Upon doing so, the spinning dots issue corrected itself. Note, this itself is not the solution; setting this causes the Windows logon screen to not appear for anyone trying to remotely connect using the company Citrix login – VDA (they will get a grey screen).

After some more research, I found I couple Citrix KBs eluding to this issue and that this was specific to VDA client versions 7.14 (Citrix Virtual Delivery Agent 7.14.1) & 7.15 CU1 (Citrix 7.15 LTSR), which was about half our VDA clients in our environment. Thankfully, it did not affect VDA version 7.15 CU2, which was the other half.

Over the next few weeks we started the process of updating the VDA clients, which in itself became somewhat of a challenge. Ultimately, we ended up doing this part of the update as a task sequence, which ran the VDA cleanup utility in unattended mode (the native uninstall command was failing for some unknown reason in most cases), restarted the machine and then installed 7.15 CU2.

A guide for using the VDA cleanup tool can be found in this Citrix KB: How to Run the VDA Cleanup Utility with SCCM Task Sequences

*Of note, it was found that this issue could also be overcome by running windows Disk Cleanup as an admin and selecting the option Windows Update Cleanup. This has the effect of removing the various updated Windows files (otherwise, they are archived). The downside to this is that you would not be able to remove any problematic update going forward to restore the system to a previous stable point. It also did not guarantee that the issue would not resurface with the next batch of updates … not to mention it was taking 1 hour plus to complete in some cases and was not 100%. I’m still curious why it was working in many cases, though.