This is just a quick example of upgrading Tanzu Kubernetes Grid (multicloud) 1.1.3 to 1.2. In this example, TKGm is running on vSphere.
For those not familiar, TKG:
provides a consistent, upstream-compatible implementation of Kubernetes, that is tested, signed, and supported by VMware. Tanzu Kubernetes Grid is central to many of the offerings in the VMware Tanzu portfolio.
TKGm, as I call it, can be deployed into various public clouds, more all the time, and provides the same Kubernetes no matter where it is deployed. 1.2 supports vSphere, Azure, and AWS as host infrastructure, and more will be added over time.
And more!
The documentation for this process is great, and I’m mostly just repeating what it shows. Best to follow those docs, but sometimes having an example is nice.
Initially I have the tkg 1.1.3 CLI.
$ tkg version
Client:
Version: v1.1.3
Git commit: 0e8e58f3363a1d4b4063b9641f44a3172f6ff406
I’m just running one management and one workload cluster.
$ tkg get cluster --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
my-cluster default running 1/1 2/2 v1.18.6+vmware.1 <none>
tkg-mgmt tkg-system running 1/1 1/1 v1.18.6+vmware.1 <none>
So the first step is to download the new 1.2 CLI as well as three OVAs. These artifacts are all downloaded from VMware.
Install the new CLI first.
Next, upload the three OVAs into vSphere and mark them as templates.
Kubernetes v1.19.1: Photon v3 Kubernetes v1.19.1 OVA
Kubernetes v1.18.8: Photon v3 Kubernetes v1.18.8 OVA
Kubernetes v1.17.11: Photon v3 Kubernetes v1.17.11 OVA
First we upgrade the management cluster.
$ cp -rp ~/.tkg/ ~/.tkg-pre-1.2-upgrade
$ tkg get management-cluster
It seems that the TKG settings on this system are out-of-date. Proceeding on this command will cause them to be backed up and overwritten by the latest settings.
Do you want to continue? [y/N]: y
the old providers folder /home/ubuntu/.tkg/providers is backed up to /home/ubuntu/.tkg/providers-20201102220133-xryjaxet
The old bom folder /home/ubuntu/.tkg/bom is backed up to /home/ubuntu/.tkg/bom-20201102220133-sk8je1f4
MANAGEMENT-CLUSTER-NAME CONTEXT-NAME STATUS
tkg-mgmt * tkg-mgmt-admin@tkg-mgmt Success
$ kubectl config use-context tkg-mgmt-admin@tkg-mgmt
Switched to context "tkg-mgmt-admin@tkg-mgmt".
$ kubectl label -n tkg-system cluster.cluster.x-k8s.io/tkg-mgmt cluster-role.tkg.tanzu.vmware.com/management="" --overwrite=true
cluster.cluster.x-k8s.io/tkg-mgmt labeled
$ tkg upgrade management-cluster tkg-mgmt
SNIP!
Patching MachineDeployment with the kubernetes version v1.19.1+vmware.2...
Waiting for kubernetes version to be updated for worker nodes...
Management cluster 'tkg-mgmt' successfully upgraded to TKG version 'v1.2.0' with kubernetes version 'v1.19.1+vmware.2'
$ tkg get cluster --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
my-cluster default running 1/1 2/2 v1.18.6+vmware.1 <none>
tkg-mgmt tkg-system running 1/1 1/1 v1.19.1+vmware.2 management
$ tkg get kubernetesversions
VERSIONS
v1.17.11+vmware.1
v1.17.3+vmware.2
v1.17.6+vmware.1
v1.17.9+vmware.1
v1.18.2+vmware.1
v1.18.3+vmware.1
v1.18.6+vmware.1
v1.18.8+vmware.1
v1.19.1+vmware.2
$ tkg upgrade cluster my-cluster
Logs of the command execution can also be found at: /tmp/tkg-20201102T223108680260342.log
Upgrading workload cluster 'my-cluster' to kubernetes version 'v1.19.1+vmware.2'. Are you sure? [y/N]: y
Validating configuration...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
Create InfrastructureTemplate for upgrade...
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.19.1+vmware.2...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.19.1+vmware.2...
Waiting for kubernetes version to be updated for worker nodes...
Cluster 'my-cluster' successfully upgraded to kubernetes version 'v1.19.1+vmware.2'
Well, that was pretty easy.
Here I create a new cluster. Note the use of the --vsphere-controlplane-endpoint-ip which is available in 1.2 so you can set the k8s API IP address and presumably pre-set the DNS entry for your end users.
tkg create cluster --plan dev 1-2-cluster --vsphere-controlplane-endpoint-ip 10.0.6.10
Logs of the command execution can also be found at: /tmp/tkg-20201103T173324591159647.log
Validating configuration...
Creating workload cluster '1-2-cluster'...
Waiting for cluster to be initialized...
Waiting for cluster nodes to be available...
Waiting for addons installation...
Workload cluster '1-2-cluster' created
That’s it!
Due to the way I had originally deployed TKGm, I had set the VSPHERE_TEMPLATE in the TKG config file and to deploy a new 1.2 cluster I needed to comment that out. Most users won’t have this set AFAIK, and it’s a simple config file change.
$ tkg create cluster --plan dev 1-2-cluster --vsphere-controlplane-endpoint-ip 10.0.6.10
Logs of the command execution can also be found at: /tmp/tkg-20201103T172950440339415.log
Validating configuration...
Error: : workload cluster configuration validation failed: vSphere config validation failed: vSphere template kubernetes version validation failed: unable to get or validate VSPHERE_TEMPLATE for given k8s version: incorrect VSPHERE_TEMPLATE (/Datacenter/vm/photon-3-kube-v1.18.6_vmware.1) specified for Kubernetes version (v1.19.1+vmware.2). TKG CLI will autodetect the correct VM template to use, so VSPHERE_TEMPLATE should be removed unless required to disambiguate among multiple matching templates
Detailed log about the failure can be found at: /tmp/tkg-20201103T172950440339415.log
Commented out the option:
$ grep -i template ~/.tkg/config.yaml
VSPHERE_HAPROXY_TEMPLATE: /Datacenter/vm/photon-3-haproxy-v1.2.4-vmware.1
# VSPHERE_TEMPLATE will be autodetected based on the kubernetes version. Please use VSPHERE_TEMPLATE only to override this behavior
#VSPHERE_TEMPLATE: /Datacenter/vm/photon-3-kube-v1.18.6_vmware.1