I have been struggling with a number of errors and warnings between PowerCLI and my vCenter servers. The warnings about my self-signed certificates are no big deal, but the errors of course are. The biggest error I have is a well-known issue documented in this vCheck issue on GitHub:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

This happens intermittently, but frequently with the Get-HardDisk cmdlet which is used in most of the Snapshot related plugins. When it does happen, the vCheck plugin fails to return any meaningful data and normally errors pretty fast – run times for the full set of checks in my environment drops from ~120 minutes to ~8 minutes.

The issue goes back over 3 years and while there were a number of attempts to fix the issue, there was no single fix that worked for everyone, every time. Some would hide the issue till you hit a certain threshold and others would just make it far less likely to occur, but not eliminate it. I eventually opened an issue with VMware support and we found what I think is the solution.

Untrusted Certificates and CAs

I am using the provided certificates for my vCenters. These certificates have an expiry term of 10 years and are signed by a CA also provided by vCenter during the initial install. This is typically known as a self-signed certificate, but more specifically means the cert is not signed by a CA trusted by the client (if it was signed by Verisign but you removed the Verisign CAs from your Trusted CA store, it would be reported as a cert signed by an Untrusted CA and/or a self-signed certificate, depending on the application interfacing with it. I have decided to continue to use these certs as the process for attaching new certificates to a vCenter installation is hairy, to say the least.

This means that when I run Connect-VIServer against my vCenter, I receive the following note about the untrusted CA:

Be sure to use the FQDN to access your vCenter server, or this warning will be swallowed in favor of a “name mismatch” warning.

Generally speaking, most of us don’t care about this error because we are confident that we are connecting to our vCenter server and we tend to ignore this as a cause of problems. I certainly did. I don’t know the specifics surrounding it, but PowerCLI sometimes decides it doesn’t like the Untrusted CA and thus generates the error about Could not establish trust relationship. Sometimes, it’s cool and establishes it just fine. I believe it has something to do with resource exhaustion in tracking the connection, as one of the workarounds suggested on GitHub appeared to work for some by increasing the resources available to a PowerShell session. Regardless of the specifics, connecting to a Trusted CA does not have this issue. So our resolution is to use certificates signed by a Trusted CA!

As suggested above, you can attach new certificates directly from a Trusted CA to your vCenter, but it’s a tricky process. The other alternative is to trust the CA from your vCenter, which we’ll do here. Alternatively, if you want to attach new certs from an already-Trusted CA, check out KB2111219 and any number of blog posts that address this process and skip ahead to the Summary section.

Download and Install the Certificate Bundle

The first step to trusting the vCenter’s included CA is to download the certificate bundle. You can do this by visiting your vCenter on port 443, e.g. https://vcenter.example.com, and clicking on Download trusted root CA certificates:

You will receive a zip file that contains the certs in various formats. Since I’m on Windows, I burrow down to the certs\win directory where there are two CRT files and one CRL. Extract this in a folder somewhere. You only need the CRT that is paired with a CRL; the other CRT is for the ssoserver and that is not something PowerCLI cares about.

In vCenter 6.0, the cert bundle had no directories and just two files ending in .0 and .r0 (now found in the lin and mac directories) that correspond to .crt and .crl respectively, so just extract and rename the files if you that’s the case.

Now, we need to access the certificate store. This varies per OS and version. In Windows 7, you can find the store inside the Internet Options control panel on the Content tab by clicking the Certificates button. Click over to the Trusted Root Certification Authorities tab.

Click the Import button and browse to the CRT you stored earlier. When you import it, you’ll see the name CA – if you see ssoserver, you chose the wrong CRT file, try again with the other. You can now click on the imported CA called CA and click View to see the name. This is important when you have more than one vCenter, as they all import with the name CA, because that’s not confusing! You can see here this is the CA from my vCenter server called vcsa.nelson.va:

You want to repeat this process on any and all nodes that will use PowerCLI to connect to the vCenter in question, not just the server you run vCheck from.

Summary

With either your new certs or the new trust with the existing CA, you shouldn’t see the warning upon accessing your vCenter with Connection-VIServer. Close your PowerShell/PowerCLI sessions run that inside a brand new session and if you did things correctly, you will not see any yellow warning text:

When you run vCheck now, you should no longer see those random SSL/TLS errors! If you disabled some checks, like Phantom Snapshots, because they failed more often than they ran, this is a good time to review if you want to re-enable them. I hope this helps.

I will warn that this solution has only been tested for about a month, but I saw error rates drop from 70% to 0%. I could NOT get the errors to occur with the CA in place, but they would come back the moment I removed the CA. If you see the error return, please let me know in the comments or on twitter and I’ll be glad to share the ticket number reference for engaging support!

Many thanks to Isaac at VMware for this solution, and especially his insistence that I should import the CA even though I swore that couldn’t be the problem 🙂