…or killing two birds with one stone.

One of our tier 1 technicians asked me to look at their workstation after not being able to install anything from the Software Center or Application Catalog. Usually, this is corrected by uninstalling the CCM client and (also referred to as the SMS agent), upon restarting of the system, it is reinstalled or we just manually push the client from the CCM Manager to the problem machine again. However, in this case, that approach had no effect. The only suspect at this point was the recent Win10 update from 1703 to 1709 that we had started pre-deploying to some technician workstations to weed out problems.

Additionally, I had also received a few complaints from the helpdesk of the CCM Remote Control Viewer crashing the OS (bugcheck  or BSOD) when they were trying to connect to someone’s workstations, usually when trying to resize the viewer window. Initially, I had assumed this was related to the video system or one of its components, and running the memory dump through a debugger hinted that this might be the case:

Time is limited and in-depth crash debugging can be terrible complex so I only use WinDbg when I absolutely have to and do not jump too deep into dump analysis, so I left it at that and asked those having the issue with Remote Control Viewer to run off to computer vendor site and update video drivers. A couple days later, they still reported the issue occasionally happening, and I put this one on the back burner while I investigated the issue with the crashing CCM client, assuming they might be related as both were SMS\CCM client issues that started occurring after the 1703 to 1709 update.

Curious if there were any missing components in the CCM client install, I navigated to C:\Windows\CCM to examine the file structure of the directory. To my surprise, I saw it was littered with thousands of CcmExec.exe.****.dmp files. Basically, each time the CCM client attempted to fire, it would immediately crash and over the course of a few days it had created GBs worth crash dumps. Examining the event viewer > system logs I saw the error, event ID 7031, for the SMS agent.

I grabbed one of the CCM dumps and opened it in WinDbg, ran !analyze -v and saw right after ccmexec was firing msvcr120.dll was being called before the exception was raised.

Running lmvm msvcr120 command indicates this belongs to the Microsoft C++ 2013 visual runtime library:

The current CCM Client has a dependence on the MS C++ 2013 Redistributable. You can see this by going into the ccm setup directory:

Now, this issue was not happening with everyone who updated. They key to finding the cause finding the difference across the workstations. Fortunately, I already had my suspicion as it happens that I also created a deployment for the Citrix VDA client last year and happened to know the internals of the installer also have a dependence on the MS C++ 2013 Redistributable, which you can see if you unpack the installer. The problem machines all had the Citrix VDA client installed prior to the 1703 to 1709 update. Somewhere along the way during the update, something is causing the CCM client and VDA client to clash with the 2013 visual runtimes. Fixing was a matter of uninstalling the VDA client, restarting, and reinstalling the VDA. Since then, we have incorporated some logic into the updated task sequence for the update to uninstall the VDA client prior to the update, and reinstall at the end of the TS.

Admittedly, I don’t think I investigated running a repair or reinstall of the C++ 2013 runtime library as it was nearing the end of the week, I was about to go on vacation and wanted to clear out my tasks so I decided not invest any more time into it. Or maybe I did, forgot, and it had no effect. That might be an alternative to uninstalling the VDA if any body wants to try.

Update

_______________________________________________________________________________________________________________________________________________________

Repairing the C++ runtime 2013 does not fix. Additionally, we started seeing hang issues during 1709 update when including VDA install, so, instead, we have opted to simply leave the VDA in place and unregister the VDA component causing the issue as we do not use it: regsvr32 -u “C:\Program Files\Citrix\Third Party Integration\System Center Configuration Manager\SccmPlugin.dll”

Additionally, some have suggested that moving to VDA 7.15 corrects this. That is not true, at least not in our case.

During recent BitLocker piloting for USB mass storage devices we noticed that Android devices were not affected by our Bitlocker policy, which was to prevent writing of data if the device was not encrypted. Although we do not intend to enforce encryption on these devices for obvious reason, we still wanted to be able to deny write to the devices in the same manner that users could not write to their BlackBerry or iPhone devices where USB BitLocker policy was to deny write was working as expected.

After opening a case with Microsoft, the conclusion was that even though the Android devices were mass storage devices, they were presenting themselves as Windows Portable Devices:

^{>>  [Device Install (Hardware initiated) – WpdBusEnumRoot\UMB\2&37c186b&0&STORAGE#VOLUME#_??_USBSTOR#DISK&VEN_SMI&PROD_USB_DISK&REV_1100#7&2DEFC6D0&0#]
>>>  Section start 2013/11/12 14:30:18.871
     ump: Creating Install Process: DrvInst.exe 14:30:18.871
     ndv: Retrieving device info…
     ndv: Setting device parameters…
     ndv: Doing WU search last due to CM_DEVCAP_SILENTINSTALL flag.
     ndv: Searching Driver Store and Device Path…
     dvi: {Build Driver List} 14:30:18.949
     dvi:      Searching for compatible ID(s):
     dvi:           wpdbusenum\fs
     …}

     ^{dvi:           DrvDate      – 06/21/2006
     dvi:           Version      – 6.1.7600.16385
     inf:      Searched 1 potential matches in published INF directory
     inf:      Searched 35 INFs in directory: ‘C:\Windows\inf’
     dvi: {Build Driver List – exit(0x00000000)} 14:30:19.074
     ndv: Selecting best match from Driver Store (including Device Path)…
     dvi: {DIF_SELECTBESTCOMPATDRV} 14:30:19.074
     dvi:      Using exported function ‘WpdClassInstaller’ in module ‘C:\WINDOWS\system32\wpd_ci.dll’.
     dvi:      Class installer == wpd_ci.dll,WpdClassInstaller
     dvi:      No CoInstallers found
     dvi:      Class installer: Enter 14:30:19.089
     dvi:      Class installer: Exit
     dvi:      Default installer: Enter 14:30:19.089}

A normal USB mass storage device, e.g. a flash drive, otherwise presents itself in manner that USB BitLocker policy could be applied to because the device is recognized as mass storage:

^{>>>  [Device Install (Hardware initiated) – STORAGE\Volume\_??_USBSTOR#Disk&Ven_SMI&Prod_USB_DISK&Rev_1100#7&2defc6d0&0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}]
>>>  Section start 2013/11/12 14:30:16.765
     ump: Creating Install Process: DrvInst.exe 14:30:16.796
     ndv: Retrieving device info…
     ndv: Setting device parameters…
     ndv: Doing WU search last due to CM_DEVCAP_SILENTINSTALL flag.
     ndv: Searching Driver Store and Device Path…
     dvi: {Build Driver List} 14:30:16.812
     dvi:      Searching for hardware ID(s):
     dvi:           storage\volume
…
dvi:           DrvDate      – 06/21/2006
     dvi:           Version      – 6.1.7601.17567
     inf:      Searched 1 potential matches in published INF directory
     inf:      Searched 35 INFs in directory: ‘C:\Windows\inf’
     dvi: {Build Driver List – exit(0x00000000)} 14:30:16.937
     ndv: Selecting best match from Driver Store (including Device Path)…
     dvi: {DIF_SELECTBESTCOMPATDRV} 14:30:16.937
     dvi:      Using exported function ‘VolumeClassInstaller’ in module ‘C:\WINDOWS\system32\SysClass.dll’.
     dvi:      Class installer == SysClass.dll,VolumeClassInstaller
     dvi:      Using exported function ‘CriticalDeviceCoInstaller’ in module ‘C:\WINDOWS\system32\SysClass.Dll’.
     dvi:      CoInstaller 1 == SysClass.Dll,CriticalDeviceCoInstaller}

This information is written each time you connect a USB device to a Windows computer and can be found in C:\Windows\inf\setupapi.dev.log.

The workaround is to apply a Deny Write Access to WPD devices in Computer Configuration\Administrative Templates\System\Removable Storage Access.

It confuses the users. Normal folder icon in Windows Explorer:

Then digging down into the folder the user encounters another folder:

Its described correctly as an Application but the average user is probably not going to distinguish the difference between a real Windows system folder icon and a program folder icon and completely ignore the description field telling them it is an application. And so they then try to open the folder and encounter:

Ack! Then the help desk wonders, “Ok, so missing config file somewhere, right?”. Nope, its right there, but it has been named to BillsArchive.exe. File extensions are turned off in Explorer and no one really pays any attention to the description (right?) so they don’t notice that the filename is really BillsArchive.exe.config.

By the way, the program icon folder looks almost exactly the same as a Windows XP system folder icon:

Bottom line is program icons shouldn’t look like system folder icons. This is also a great way for malware to execute itself by simply looking like some harmless folder named “Pictures”, or whatever.