Red Flags and the Value of Experience

One of the things I hear often said, and something I subscribe to as well, is the idea that a lot of technical knowledge in the world of IT has a very short shelf life. When interviewing candidates, we really tend to focus on what they've done in the last few years when considering their technical skill sets. As a practitioner, I start feeling a little stale on my knowledge if I don't touch an area of IT for awhile. That's not to say that anything beyond 3 years of experience is worthless though, because the value of good experience isn't really in technical skill. What you should be paying those IT greybeards for is their presumed ability to see patterns and discern bad decisions before they are made. That's a skill of it's own, and a muscle that needs to get some exercise by being allowed the opportunity to voice an opinion and have it feel valued. 

A recent real-life example that recently occurred illustrates this well:

Converting Citrix PVS Image from XenServer to vSphere

Having repeated this nightmarish migration several times now, here's the steps I've found to be most efficient:

1. Import your XenServer-optimized PVS image (as a VHD) in to XenCenter as a new VM.
2. Snapshot and boot the VM (just in case you mess up the next step you won't need to import again).
3. Uninstall the Citrix PVS and Citrix Guest Tools / Xen Tools bits. 
4. Delete xen*.sys from c:\windows\system32 and c:\windows\system32\drivers
5. Reboot and make sure everything still comes back up. It should revert to a generic Realtek network driver.
6. Run VMware Converter on the VM. Alternatively you can export the VM from XenCenter as an OVA and then import it to vSphere.
7. Be sure you are using a VMXNet3 NIC on the vSphere VM, not an E1000.
8. Boot the resulting vSphere VM and install VMware tools.
9. Delete the ghost NIC that is left from the Realtek drivers (https://support.citrix.com/article/CTX221733), otherwise you will get the BNIstack error.
10. Install PVS target device software and run the imaging wizard again. 
11. Follow all your normal steps for capturing a new image

If you run in to an IRQL_NOT_EQUAL_OR_LESS BSOD, you may be like me and have some piece of software set to redirect writes to the vDisk cache disk which no longer exists. Make sure you fix that prior to attempting a migration.

The most commonly recommended solutions for BNIstack errors during your first boot after capturing the image:

1. Make sure no ghost NICs are still present
2. Try uninstalling any antivirus and disabling IPv6
3. Install the hotfix for KB 2550978 (https://support.microsoft.com/en-us/help/2550978/0x0000007b-stop-error-after-you-replace-an-identical-iscsi-network-ada)
4. Consider changing the default open retry limits/interval for BNIstack (https://discussions.citrix.com/topic/377414-bsod-with-bnistack-and-cvhdmpsys/)

Happy migrations!

DHCP Fails after uninstalling Citrix PVS Agent

While attempting to migrate a Citrix PVS base image to a new hypervisor, I uninstalled the PVS bits from the VDA and quickly found that DHCP had been broken. I have slamming my head against the wall a bit, I remembered something I had done years ago to this image to avoid an issue where PVS was failing in an older split-scope environment we had - set the PVS service "BNDevice" as a dependency of the DHCP service so that it would request the correct IP address during the hand-off to the OS (Thanks to Syxin https://www.syxin.com/tag/bndevice/).

This obviously was preventing DHCP from starting since the BNDevice service no longer existed after removing PVS tools. Simply needed to reverse that change:

HKLM\System\CurrentControlSet\Services\dhcp\DependsOnService

Remove BNDevice from the list of dependencies.

Upgrading R730 with NVIDIA K1 GRID Card

I recently ordered a couple of Dell R730 servers and then got a subsequent request to add a little graphics horsepower for our VDI environment in the form of some K1 GRID cards. Turns out, the process to add these suckers in to an existing server that wasn't specifically built out for them has a few catches - you can't just drop them in to the server and take off. It's easy to do, but there isn't much official documentation from Dell on it, so here's a quick guide:

SCEP Policy Update Troubleshooting

Because I'm a glutton for punishment, I recently started rolling out System Center Configuration Manager 2012 R2 SP1 and System Center Endpoint Protection across our VDI environment. There are always some considerations to be made in a pooled desktop / gold image type environment when loading software that uniquely identifies devices, but lucky for me SCCM/SCEP handled this just fine without any tweaking. However, there were some nuances to how SCEP policies are applied that caused some serious hair-pulling before I spotted the issues.

Outlook Credential Prompt When Opening Exchange 2013 Public Folder

After completing an Exchange 2007 > 2013 migration recently, I was left with one issue that was preventing us from stamping the project as a roaring success and moving on:

Outlook 2013 users were sometimes receiving a single pop-up prompt for credentials whenever they opened the Public Folder (we have only one). One. Single. Prompt.

Google was frustratingly unhelpful because searching for "outlook prompts for username and password when opening public folders" or something similar just resulted in a lot of folks who were always getting a pop-up that wouldn't go away. It was usually caused by an authentication failure of some sort.

However, we were in a different boat - Users got the prompt once when they first launched Outlook and opened their public folders, but after entering it they could continue - authentication worked. Next time they logged in to their PC, it would happen again. Not a show stopper, but it definitely generated its share of support calls.

Running vCenter 5.x with SQL 2012 AlwaysOn Availability Groups

After proudly starting the listener on our shiny new SQL 2012 AlwaysOn cluster, I was very eager to get vCenter moved off the brave little single-point-of-failure that is our current SQL server (a 2008 VM sitting in the virtual environment itself). I had done some research ahead of time and thought that AlwaysOn was at least sort-of supported by VMWare for protecting vCenter workloads. However, in my haste to play with a fancy new toy, I must have missed the plethora of blog posts indicating that either a) It's not actually supported at all, or b) Only Failover Clustering (shared storage) - not Availability Groups (non-shared storage) - are supported. And if you are about to do what many have done on the forums and suggest KB1037959 as evidence that they ARE supported, think again - that article is referencing support for running various clustered workloads on vSphere, not running your vCenter DB on clustered systems. Outside of a vague mention of AlwaysOn as a possible third party clustering solution to replace vCenter Heartbeat (e.g. "Best effort support"), I haven't been able to find anything official one way or the other.

But the AlwaysOn cluster was ready to go and if no one is going to tell me explicitly that I can't do it - well, that's basically an open invitation.