Windows Explored

…or killing two birds with one stone.

One of our tier 1 technicians asked me to look at their workstation after not being able to install anything from the Software Center or Application Catalog. Usually, this is corrected by uninstalling the CCM client and (also referred to as the SMS agent), upon restarting of the system, it is reinstalled or we just manually push the client from the CCM Manager to the problem machine again. However, in this case, that approach had no effect. The only suspect at this point was the recent Win10 update from 1703 to 1709 that we had started pre-deploying to some technician workstations to weed out problems.

Additionally, I had also received a few complaints from the helpdesk of the CCM Remote Control Viewer crashing the OS (bugcheck  or BSOD) when they were trying to connect to someone’s workstations, usually when trying to resize the viewer window. Initially, I had assumed this was related to the video system or one of its components, and running the memory dump through a debugger hinted that this might be the case:

Time is limited and in-depth crash debugging can be terrible complex so I only use WinDbg when I absolutely have to and do not jump too deep into dump analysis, so I left it at that and asked those having the issue with Remote Control Viewer to run off to computer vendor site and update video drivers. A couple days later, they still reported the issue occasionally happening, and I put this one on the back burner while I investigated the issue with the crashing CCM client, assuming they might be related as both were SMS\CCM client issues that started occurring after the 1703 to 1709 update.

Curious if there were any missing components in the CCM client install, I navigated to C:\Windows\CCM to examine the file structure of the directory. To my surprise, I saw it was littered with thousands of CcmExec.exe.****.dmp files. Basically, each time the CCM client attempted to fire, it would immediately crash and over the course of a few days it had created GBs worth crash dumps. Examining the event viewer > system logs I saw the error, event ID 7031, for the SMS agent.

I grabbed one of the CCM dumps and opened it in WinDbg, ran !analyze -v and saw right after ccmexec was firing msvcr120.dll was being called before the exception was raised.

Running lmvm msvcr120 command indicates this belongs to the Microsoft C++ 2013 visual runtime library:

The current CCM Client has a dependence on the MS C++ 2013 Redistributable. You can see this by going into the ccm setup directory:

Now, this issue was not happening with everyone who updated. They key to finding the cause finding the difference across the workstations. Fortunately, I already had my suspicion as it happens that I also created a deployment for the Citrix VDA client last year and happened to know the internals of the installer also have a dependence on the MS C++ 2013 Redistributable, which you can see if you unpack the installer. The problem machines all had the Citrix VDA client installed prior to the 1703 to 1709 update. Somewhere along the way during the update, something is causing the CCM client and VDA client to clash with the 2013 visual runtimes. Fixing was a matter of uninstalling the VDA client, restarting, and reinstalling the VDA. Since then, we have incorporated some logic into the updated task sequence for the update to uninstall the VDA client prior to the update, and reinstall at the end of the TS.

Admittedly, I don’t think I investigated running a repair or reinstall of the C++ 2013 runtime library as it was nearing the end of the week, I was about to go on vacation and wanted to clear out my tasks so I decided not invest any more time into it. Or maybe I did, forgot, and it had no effect. That might be an alternative to uninstalling the VDA if any body wants to try.

Update

_______________________________________________________________________________________________________________________________________________________

Repairing the C++ runtime 2013 does not fix. Additionally, we started seeing hang issues during 1709 update when including VDA install, so, instead, we have opted to simply leave the VDA in place and unregister the VDA component causing the issue as we do not use it: regsvr32 -u “C:\Program Files\Citrix\Third Party Integration\System Center Configuration Manager\SccmPlugin.dll”

Additionally, some have suggested that moving to VDA 7.15 corrects this. That is not true, at least not in our case.