No results found

Your search did not match any results.

We suggest you try the following to help find what you're looking for:

  • Check the spelling of your keyword search.
  • Use synonyms for the keyword you typed, for example, try "application" instead of "software."
  • Try one of the popular searches shown below.
  • Start a new search.
Trending Questions
 

Deploying A Multi-Cluster Verrazzano On Oracle Container Engine for Kubernetes (OKE) Part 2

In Part 1, we discussed setting up the network infrastructure for a multi-cluster Verrazzano deployment. Earlier

Author: Ali Mukadam

Updated:

About Ali Mukadam



Technical Director, Asia Pacific Center of Excellence. For the past 16 years, Ali has held technical presales, architect and industry consulting roles in BEA Systems and Oracle across Asia Pacific, focusing on middleware and application development. Although he pretends to be Thor, his real areas of expertise are Application Development, Integration, SOA (Service Oriented Architecture) and BPM (Business Process Management). An early and worthy Docker and Kubernetes adopter, Ali also leads a few open source projects (namely terraform-oci-oke) aimed at facilitating the adoption of Kubernetes and other cloud native technologies on Oracle Cloud Infrastructure.

More tutorials from this author:

  • Deploying Verrazzano on Oracle Container Engine for Kubernetes (OKE)
  • Deploying A Multi-Cluster Verrazzano On Oracle Container Engine for Kubernetes (OKE) Part 1
  • Deploying A Multi-Cluster Verrazzano On Oracle Container Engine for Kubernetes (OKE) Part 2
  • Installing and using Calico on Oracle Container Engine (OKE)
  • Tags

    open-source oke kubernetes terraform devops

    Back to tutorials

    Welcome!

    Verrazzano Logo

    In Part 1, we discussed setting up the network infrastructure for a multi-cluster Verrazzano deployment. Earlier, we focused on deploying Verrazzano on Oracle Container Engine (OKE). In this article, we will configure the clusters so they behave as a kind of global cluster. Below is the multi-clustering process depicted graphically:

    Verrazzano multi-cluster deployment and registration process
    Verrazzano multi-cluster deployment and registration process

    Recall that a Verrazzano multi-cluster has 1 Admin cluster and 1 or more managed clusters and that each Verrazzano cluster is a Kubernetes cluster:

    Verrazzano multi-cluster architecture
    Verrazzano multi-cluster architecture

    Also, remember we have the following setup:

    Remote peering with different regions using star architecture for managed clusters
    Remote peering with different regions

    Plus, we chose the Admin server to be in the Singapore OCI region.

    We will install Verrazzano with dev/prod profile on the Admin cluster and with managed-cluster profile on the managed clusters.

    Note on using kubectx

    In the commands below, I use kubectx to set the Kubernetes context where a context is equivalent to a Kubernetes cluster. Strictly speaking that’s not true but it serves our purpose here. Since we have 1 Admin servers and 3 managed servers in 4 different regions, we have 4 different contexts:

    Verifying your Kubernetes context
    Verifying your Kubernetes context

    To ensure we are always using the correct context, I execute the kubectx <context-name> before every command.

    Installing Verrazzano as Admin

    Installing Verrazzano as the Admin cluster is straightforward. You follow the quickstart guide and you can choose between the dev/prod profile. On the operator host, ensure your context is pointing to “admin”:

    Verifying your Kubernetes context
    Verifying your Kubernetes context

    If it’s pointing to one of the other clusters, change it as follows:

    kubectx admin
    

    We can now begin the installation:

    kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.0.3/operator.yaml
    

    Wait for the deployment to finish:

    kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
    

    And confirm the operator pods are working correctly:

    kubectl -n verrazzano-install get pods
    NAME                                            READY   STATUS    RESTARTS   AGE
    verrazzano-platform-operator-54cf56884f-46zzk   1/1     Running   0          91s
    

    Next, install Verrazzano:

    kubectl apply -f - <<EOF  
    apiVersion: install.verrazzano.io/v1alpha1  
    kind: Verrazzano  
    metadata:  
        name: admin  
    spec:  
        profile: dev  
    EOF
    

    Now we need to wait until the installation is complete:

    kubectl wait \  
        --timeout=20m \  
        --for=condition=InstallComplete \  
        verrazzano/admin
    

    This will take a while. In the meantime, let’s install Verrazzano on the managed clusters.

    Installing Verrazzano on managed clusters

    Change the context to 1 of the managed clusters and install the operator again e.g.

    kubectx sydney kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.0.3/operator.yamlkubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
    

    Repeat the above for all the managed clusters. Before running in each managed cluster, ensure you have changed your context with kubectx <contextname> as above.

    Using the same procedure as for the Admin region, verify that the Verrazzano operator has been successfully installed.

    Now, install Verrazzano for each using the managed profile by changing the context and name accordingly:

    apiVersion: install.verrazzano.io/v1alpha1  
    kind: Verrazzano  
    metadata:  
        name: sydney  
    spec:  
        profile: managed-cluster
    

    Verifying the Admin cluster and managed clusters

    While the managed clusters are being installed, let’s see if we can access the various consoles. Ensure you can login into the Verrazzano and Rancher consoles.

    Change the context again and verify:

    kubectx sydney kubectl wait \  
        --timeout=20m \  
        --for=condition=InstallComplete \  
        verrazzano/sydney
    

    Repeat the verification for each managed cluster.

    Registering the managed clusters

    Verify the the CA certificate type for each managed cluster:

    kubectx sydney kubectl -n verrazzano-system get secret system-tls -o jsonpath='{.data.ca\.crt}'
    

    If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority and you can generate a secret containing the CA certificate in YAML format. If it’s not empty, then the certificate is self-signed and needs to be extracted. Refer to the workflow at the beginning of this article.

    kubectx sydney
    
    CA_CERT=$(kubectl \  
        get secret system-tls \  
        -n verrazzano-system \  
        -o jsonpath="{.data.ca\.crt}" | base64 --decode)
    
    kubectl create secret generic "ca-secret-sydney" \  
      -n verrazzano-mc \  
      --from-literal=cacrt="$CA_CERT" \  
      --dry-run=client -o yaml > managedsydney.yaml
    

    Repeat the above for the 2 other regions replacing the region/context and filenames accordingly.

    Create 3 secrets on the Admin cluster that contains the CA certificate for each managed cluster:

    kubectx adminkubectl apply -f managedsydney.yaml  
    kubectl apply -f managedmumbai.yaml  
    kubectl apply -f managedtokyo.yaml
    

    Get the cluster name for the Admin Cluster:

    kubectl config get contexts
    
    Cluster names
    Cluster names

    Get the API Server address for the Admin server:

    kubectx adminexport CLUSTER_NAME="cluster-cillzxw34tq"API_SERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$CLUSTER_NAME\")].cluster.server}")
    

    Create a ConfigMap that contains the Admin cluster’s API server address:

    kubectx adminkubectl apply -f <<EOF -  
    apiVersion: v1  
    kind: ConfigMap  
    metadata:  
      name: verrazzano-admin-cluster  
      namespace: verrazzano-mc  
    data:  
      server: "${API_SERVER}"  
    EOF
    

    Create the VerrazzanoManagedCluster object for each managed cluster:

    kubectx admin  
    kubectl apply -f <<EOF -  
    apiVersion: clusters.verrazzano.io/v1alpha1  
    kind: VerrazzanoManagedCluster  
    metadata:  
      name: sydney  
      namespace: verrazzano-mc  
    spec:  
      description: "Sydney VerrazzanoManagedCluster object"  
      caSecret: ca-secret-sydney  
    EOFkubectl apply -f <<EOF -  
    apiVersion: clusters.verrazzano.io/v1alpha1  
    kind: VerrazzanoManagedCluster  
    metadata:  
      name: mumbai  
      namespace: verrazzano-mc  
    spec:  
      description: "Mumbai VerrazzanoManagedCluster object"  
      caSecret: ca-secret-mumbai  
    EOFkubectl apply -f <<EOF -  
    apiVersion: clusters.verrazzano.io/v1alpha1  
    kind: VerrazzanoManagedCluster  
    metadata:  
      name: tokyo  
      namespace: verrazzano-mc  
    spec:  
      description: "Tokyo VerrazzanoManagedCluster object"  
      caSecret: ca-secret-tokyo  
    EOF
    

    Wait for the VerrazzanoManagedCluster resource to reach the Ready status:

    kubectx adminkubectl wait --for=condition=Ready \  
        vmc sydney -n verrazzano-mckubectl wait --for=condition=Ready \  
        vmc sydney -n verrazzano-mckubectl wait --for=condition=Ready \  
        vmc sydney -n verrazzano-mc
    

    Export a YAML file created to register the managed cluster:

    kubectx adminkubectl get secret verrazzano-cluster-sydney-manifest \  
        -n verrazzano-mc \  
        -o jsonpath={.data.yaml} | base64 --decode > registersydney.yamlkubectl get secret verrazzano-cluster-mumbai-manifest \  
        -n verrazzano-mc \  
        -o jsonpath={.data.yaml} | base64 --decode > registermumbai.yamlkubectl get secret verrazzano-cluster-tokyo-manifest \  
        -n verrazzano-mc \  
        -o jsonpath={.data.yaml} | base64 --decode > registertokyo.yaml
    

    On each managed cluster, apply the registration file:

    kubectx sydney  
    kubectl apply -f registersydney.yamlkubectx mumbai  
    kubectl apply -f registermumbai.yamlkubectx tokyo  
    kubectl apply -f registertokyo.yaml
    

    Now verify whether the registration completed successfully:

    kubectx admin  
    kubectl get vmc sydney -n verrazzano-mc -o yaml  
    kubectl get vmc mumbai -n verrazzano-mc -o yaml  
    kubectl get vmc tokyo -n verrazzano-mc -o yaml
    

    Additional verifications

    Navigate to the Verrazzano console, login and you should be able to see all 3 clusters:

    Managed clusters in Verrazzano
    Managed clusters in Verrazzano

    Similarly, on the Rancher console, you should be able to see all 4 clusters:

    Admin and managed clusters in Rancher
    Admin and managed clusters in Rancher

    “local” is the Admin cluster whereas the others are the managed clusters.

    Conclusion

    This concludes the exercise of connecting the OKE clusters deployed in different regions into a multi-cluster Verrazzano deployment. Note that in this post, we did not configure things like DNS, Certificates, Ingress Controller etc. Our aim is to get the multi-cluster configuration going. In a future post, we will look at those other things as well.