Many Citrix Provisioning (PVS) administrators are probably familiar with the problems that can occur over the two Daylight Saving Time (DST) weekends each year. No? Well take a moment to read the official knowledge article from Citrix, CTX200058.
TLDR; Windows does not like to start up being one hour late or ahead the domain time. The solution that Citrix proposes is to re-open all PVS disks and re-sync the time before the PVS targets have had a chance to reboot after the DST time-change.
This is more of a workaround than a solution, and for some reason it is the only one I can find discussed in other blogs and forums as well. Only this post on Facebook, which leads to a dead URL hints at someone tackling this in a different way.
There must be a better way!
Yes! There might be! It could be permanent too.
Last time I was dealing with this problem, a thought struck me:
"Why not just depend on NTP time from the hypervisor until another GPO applies the NT5DS domain-time settings?"
This is based on the fact that the local CMOS clock, which in this case would propagate from the hypervisor-time is always "set" on the virtual machine from the moment it passes Power-On-Self-Test (POST). If Windows is already basing it's time on the local CMOS clock, this would mean correct time at Windows start-up, even after DST-weekends.
Of course this depends on having set a reliable NTP source on the hypervisors in your environment.
This has only been a theory so far, as I could not come up with a way to test this outside of the actual DST dates without force-simulating a clock skew in a domain environment or stealing Thanos' time stone, whichever would be easiest.
I chose the first option, which proved difficult enough. I did have some problems actually re-producing the error-situations in my test environment based on Windows Server 2019 machines and Citrix Provisioning 7.20, which leads me to believe that these issues may have been solved/improved in either the OS or PVS components, or that my simulated DST environment just wasn't done well enough.
The last environment I experienced problems with DST and PVS in was based on Windows Server 2008 R2 and Citrix Provisioning 7.15.
Note: Since testing this outside of the DST dates has been problematic, I am only publishing this as an unverified solution for now. I will hopefully have it verified sometime after the next time DST changes (31st March 2019) in a proper environment.
The way I applied the proposed fix in my test environment is based on one key configuration that is common in many PVS environments:
- Your maintenance virtual machines must not receive the same Group Policy settings as the production Target Devices (TD) that will later use the vDisk spawned from it. This is general good practice, and will allow you to define two GPO's that will control the Windows Time Service differently for maintenance and production.
I have created a separate OU structure for maintenance and production where the actual PVS Target VMs reside, and linked two GPO's to each OU respectively.
Within these GPO's I have configured the usual PVS Target settings like Event Log redirection etc, but more importantly the setting Computer Configuration > Administrative Templates > System > Windows Time Service > Time Providers > Enable Windows NTP Client.
- For the maintenance environment, I set this to Disabled
- For the production environment, I set this to Enabled
By disabling the NTP client, Windows will automatically fall back to using the local CMOS clock, which again propagates from the hypervisor time. This means that whenever you are patching or building onto the base vDisk of a PVS Collection in the maintenance environment, it will operate with time based from the hypervisor and not the domain. As long as the hypervisor time is correct, the VM time will follow.
w32tm /query /status will output an error when the NTP client is disabled:
Note: A reboot after Group Policy update is neccessary in order for the VM to base its time from the local CMOS clock.
Note 2: For VMware VMs you do not need to tick the box for "Synchronize guest time with host". This is not needed as the VM will at certain points (start-up being one of them) base its time on the hypervisor. This will remain until anything else is set.
By enabling the NTP client via GPO again in production, the PVS Targets will still use the Local CMOS clock at start-up (since it was set in maintenance), until the VM has an established connection with the domain and the GPO that enables the NTP setting from maintenance has been applied.
I also added a service start-action for the "w32time" service in the production GPO to make sure it is started after boot.
You don't have to worry about specifying any other NTP settings for the production environment as Windows will automatically default to using NT5DS (Domain time) unless any other configuration has been set previously. You can verify this with w32tm /query /status, which should output something like this:
This fix can be applied in a numerous of ways, but I prefer to keep it clean and visible with Group Policy with as little as possible changes to default values.
I would say "discuss!" at the end of this post, but I have not implemented any comment-field here yet.
EDIT: I unfortunately did not get the time for experiementing with this solution before the DST time changed, so it will remain a theory on my part for at least 6 more months :)