No results found

Your search did not match any results.

We suggest you try the following to help find what you're looking for:

  • Check the spelling of your keyword search.
  • Use synonyms for the keyword you typed, for example, try "application" instead of "software."
  • Try one of the popular searches shown below.
  • Start a new search.
Trending Questions
 
Cool Devo Migration

Hosting a private Helm Repository on OCI (Oracle Cloud Infrastructure) with ChartMuseum and OCI Object Storage

This guide walks you through how to host Helm Chart privately with ChartMuseum, an open source Helm Chart Repository server with support for various cloud storage backends, including OCI Object Storage.

Author: Developer Relations

Updated:

In another article, we deployed a helm chart (Redis) into Oracle Container Engine (OKE) which we pulled from the stable repository. The stable repo can be found here: here.

We can also manually add the incubator repository if we want to:

helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
        "incubator" has been added to your repositories
        

The container images in those repositories are also publicly available (for example, the default image for redis is bitnami/redis in bitnami’s container redis repository on Dockerhub).

We have already described how to host the container images privately on Oracle Registry Service (OCIR). What if we want to do the same for our helm charts?

Fortunately, there is a solution for that too.

Let me introduce you to chartmuseum, an open source Helm Chart Repository server with support for various cloud storage backends, including OCI Object Storage.

Prerequisites

You’re using the terraform-oci-oke project to provision your OKE cluster. See this post for more details. Below is an architecture of what we’ll build:

Deploying chartmuseum on OCI

If you don’t yet have an OCI account, you can quickly sign up for one today by registering for an Oracle Cloud Free Tier account.

Afterwards, check developer.oracle.com for more developer content.

Install nginx-controller

First, follow this post to install ExternalDNS.

Then, let’s install nginx-controller:

helm install --name nginxcontroller stable/nginx-ingress \
        --set controller.name=controller \
        --set defaultBackend.enabled=true \
        --set defaultBackend.name=defaultbackend \
        --set rbac.create=true \
        --set controller.service.annotations."external-dns\.alpha\.kubernetes\.io/hostname"=chartmuseum.acme.com
        

Replace chartmuseum.acme.com by your preferred hostname. We’ll then use this to access chartmuseum later.

Verify that an OCI Load Balancer has been created and a DNS “A” record has been inserted in the DNS Zone.

Create OCI Bucket

In the OCI Console, navigate to Object Storage and create a bucket called ‘chartmuseum’:

chartmuseum bucket window

Deploy chartmuseum

We want a similar experience to the stable or incubator repos i.e. anonymous GET but protected POST requests and we will use Basic Auth for authentication purposes.

Let’s first create a secret to hold the Basic Auth username and password:

kubectl create secret generic chartmuseum-auth --from-literal=user=curator --from-literal=pass=password
        

You also need to create a second secret that will allow chartmuseum to communicate with the OCI APIs.

Temporarily, copy your API keys to the bastion. You can also do this locally if you have kubectl and access to the kubeconfig locally.

Create a file ‘config’:

[DEFAULT]                                           
        user=<USER_OCID>                                                                                    
        fingerprint=<API_KEY_FINGERPRINT>
        key_file=/home/chartmuseum/.oci/oci.key                         
        tenancy=<TENANCY_OCID>
        region=<REGION>
        

Enter your user and tenancy OCIDs, api key fingerprint and region value. The key_file value has to be /home/chartmuseum/.oci/oci.key.

Next, create the secret:

kubectl create secret generic chartmuseum-secret --from-file=config="/path/to/config" --from-file=key_file="/path/to/apikey.pem"
        

Replace with the appropriate absolute or relative path to the config file and private API key.

Next, download the chartmuseum values file:

curl -o values.yaml https://raw.githubusercontent.com/helm/charts/master/stable/chartmuseum/values.yaml
        

Edit the values.yaml file and replace the parameters as follows:

env:
          open:
            STORAGE: oracle
            STORAGE_ORACLE_COMPARTMENTID:<COMPARTMENT_OCID>
            STORAGE_ORACLE_BUCKET: chartmuseum
            STORAGE_ORACLE_PREFIX: chartmuseum
            DISABLE_API: false
            AUTH_ANONYMOUS_GET: true
            AUTH_REALM: chartmuseumexistingSecret: chartmuseum-auth
        existingSecretMappings:
          BASIC_AUTH_USER: user
          BASIC_AUTH_PASS: passingress:
          enabled: true
          labels:
            dns: "ocidns"annotations:
            kubernetes.io/ingress.class: nginxhosts:
           - name: chartmuseum.acme.com
             path: /
             tls: falseoracle:
          secret:
            enabled: true
            name: chartmuseum-secret
            config: config
            key_file: key_file
        

We can now install chartmuseum:

helm install --name=chartmuseum -f values.yaml stable/chartmuseum
        

Wait for the pod to run:

kubectl get pods -wNAME                                                            READY   STATUS    RESTARTS   AGE                                                                      
        chartmuseum-chartmuseum-748c8dbbd8-7nctc                        1/1     Running   0          5m20s
        

Verify that the ingress has been created:

kubectl get ingNAME                      HOSTS                       ADDRESS   PORTS   AGE                                                                                           
        chartmuseum-chartmuseum   chartmuseum.acme.com             80      7m9s
        

And that it maps to the chartmuseum service:

k describe ing chartmuseum-chartmuseum                                                                                                     
        Name:             chartmuseum-chartmuseum                                                                                                                             
        Namespace:        default                                                                                                                                             
        Address:                                                                                                                                                              
        Default backend:  default-http-backend:80 ()                                                                                                                    
        Rules:                                                                                                                                                                
          Host                       Path  Backends                                                                                                                           
          ----                       ----  --------                                                                                                                           
          chartmuseum.acme.com                                                                                                                                           
                                     /   chartmuseum-chartmuseum:8080 ()                                                                                                
        Annotations:                                                                                                                                                          
          kubernetes.io/ingress.class:  nginx                                                                                                                                 
        Events:                                                                                                                                                               
          Type    Reason  Age    From                      Message                                                                                                            
          ----    ------  ----   ----                      -------                                                                                                            
          Normal  CREATE  6m20s  nginx-ingress-controller  Ingress default/chartmuseum-chartmuseum                                                                            
          Normal  UPDATE  5m31s  nginx-ingress-controller  Ingress default/chartmuseum-chartmuseum
        

Finally, you can verify whether you can reach chartmuseum publicly with your browser.

Chartmuseum welcome message

Pushing a chart to Chartmuseum

Install the helm push plugin:

helm plugin install https://github.com/chartmuseum/helm-push
        

Next, add the repo:

helm repo add --username curator --password password cm http://chartmuseum.acme.com/
        

Let’s first create a basic chart which we can use to test:

helm create mycharthelm package mychart
        Successfully packaged chart and saved it to: /home/opc/chart/mychart-0.1.0.tgz
        

Now, we can push the chart:

helm push mychart cmPushing mychart-0.1.0.tgz to cm...                                                                                                                                    
        Done.
        

If we search for mychart, we will only find the local copy:

helm search mychart                                                                                                                              
        NAME            CHART VERSION   APP VERSION     DESCRIPTION                                                                                                           
        local/mychart   0.1.0           1.0             A Helm chart for Kubernetes
        

Let’s do a repo update:

helm repo update cm                                                                                                                              
        Hang tight while we grab the latest from your chart repositories...                                                                                                   
        ...Skip local chart repository                                                                                                                                        
        ...Successfully got an update from the "cm" chart repository                                                                                                          
        ...Successfully got an update from the "stable" chart repository                                                                                                      
        Update Complete. ⎈ Happy Helming!⎈
        

And a new search:

helm search mychart                                                                                                                              
        NAME            CHART VERSION   APP VERSION     DESCRIPTION                                                                                                           
        cm/mychart      0.1.0           1.0             A Helm chart for Kubernetes                                                                                           
        local/mychart   0.1.0           1.0             A Helm chart for Kubernetes
        

The new chart appears.

Testing authentication

We could remove the repo and add it again without the username and password to test. However, we will use Postman and chartmuseum’s APIs to test. This is the behaviour we want:

  • GET: No Auth Required
  • POST: Auth Required
  • DELETE: Auth Required
GET (without authentication)

You can see we are able to get a list of charts using GET without authentication.

However, a DELETE or POST without authentication fails:

Unauthorized chart deletion

Yet another twist: when we enter the credentials, the chart deletion succeeds:

Successful chart deletion

If we now check OCI Object Storage, we can see our chart there:

OCI dashboard displaying Chartmuseum details

Let’s Encrypt

Now that our chartmuseum is up and running, we want to also secure its usage by encrypting traffic. We will use Let’s Encrypt and cert-manager, “a add-on to automate the management and issuance of TLS certificates from various issuing sources.”

You can follow the guide to install cert-manager or read on.

First, create the CustomResourceDefinitions, a namespace for cert-manager and disable resource validation on the namespace:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yamlkubectl create namespace cert-managerkubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
        

Add the Jetpack Helm repo and update:

helm repo add jetstack https://charts.jetstack.io
        helm repo update jetstack
        

Install the cert-manager using its helm chart:

helm install --name cert-manager --namespace cert-manager \
        

Verify the installation:

kubectl get pods --namespace cert-manager                                                                                                                                    
        NAME                                       READY   STATUS    RESTARTS   AGE                                                                                                                   
        cert-manager-776cd4f499-98vsh              1/1     Running   0          3h14m                                                                                                                 
        cert-manager-cainjector-744b987848-pkk5s   1/1     Running   0          3h14m                                                                                                                 
        cert-manager-webhook-645c7c4f5f-4mbjd      1/1     Running   0          3h14m
        

Test the webhook works by creating a test-resources.yaml file.

apiVersion: v1
        kind: Namespace
        metadata:
          name: cert-manager-test
        ---
        apiVersion: certmanager.k8s.io/v1alpha1
        kind: Issuer
        metadata:
          name: test-selfsigned
          namespace: cert-manager-test
        spec:
          selfSigned: {}
        ---
        apiVersion: certmanager.k8s.io/v1alpha1
        kind: Certificate
        metadata:
          name: selfsigned-cert
          namespace: cert-manager-test
        spec:
          commonName: example.com
          secretName: selfsigned-cert-tls
          issuerRef:
            name: test-selfsigned
        

Let’s create the test resources:

kubectl create -f test-resources.yaml
        

Check the status of the newly-created certificate:

kubectl describe certificate -n cert-manager-testName:         selfsigned-cert                                                                                                                                                                 
        Namespace:    cert-manager-test                                                                                                                                                               
        Labels:                                                                                                                                                                                 
        Annotations:                                                                                                                                                                            
        API Version:  certmanager.k8s.io/v1alpha1                                                                                                                                                     
        Kind:         Certificate                                                                                                                                                                     
        Metadata:                                                                                                                                                                                     
          Creation Timestamp:  2019-06-25T00:45:25Z                                                                                                                                                   
          Generation:          1                                                                                                                                                                      
          Resource Version:    116448                                                                                                                                                                 
          Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/cert-manager-test/certificates/selfsigned-cert                                                                            
          UID:                 8576413b-96e2-11e9-b5fc-0a580aed39b8                                                                                                                                   
        Spec:                                                                                                                                                                                         
          Common Name:  example.com                                                                                                                                                                   
          Issuer Ref:                                                                                                                                                                                 
            Name:       test-selfsigned                                                                                                                                                               
          Secret Name:  selfsigned-cert-tls                                                                                                                                                           
        Status:                                                                                                                                                                                       
          Conditions:                                                                                                                                                                                 
            Last Transition Time:  2019-06-25T00:45:26Z                                                                                                                                               
            Message:               Certificate is up to date and has not expired                                                                                                                      
            Reason:                Ready                                                                                                                                                              
            Status:                True                                                                                                                                                               
            Type:                  Ready                                                                                                                                                              
          Not After:               2019-09-23T00:45:26Z                                                                                                                                               
        Events:
        

You can delete the test resources after testing:

kubectl delete -f test-resources.yaml
        

Configuring Let’s Encrypt Issuers

For reference, you can follow the quick-start guide for using cert-manager with Nginx Ingress.

Let’s start with creating a staging issuer by creating a staging-issuer.yaml:

apiVersion: certmanager.k8s.io/v1alpha1                                                                                                                                                       
        kind: Issuer                                                                                                                                                                                  
        metadata:                                                                                                                                                                                     
          name: cm-staging                                                                                                                                                                            
        spec:                                                                                                                                                                                         
          acme:                                                                                                                                                                                       
          # The ACME server URL                                                                                                                                                                       
            server: https://acme-staging-v02.api.letsencrypt.org/directory                                                                                                                            
            # Email address used for ACME registration                                                                                                                                                
            email: your_email_address                                                                                                                                                             
            # Name of a secret used to store the ACME account private key                                                                                                                             
            privateKeySecretRef:                                                                                                                                                                      
              name: cm-staging                                                                                                                                                                        
            # Enable the HTTP-01 challenge provider                                                                                                                                                   
            http01: {}
        

Replace your email address above.

Create the staging issuer:

kubectl create -f staging-issuer.yaml                                                                                                                                    
        issuer.certmanager.k8s.io/cm-staging created
        

Repeat the above but for production pursposes by creating a production-issuer.yaml:

apiVersion: certmanager.k8s.io/v1alpha1                                                                                                                                                       
        kind: Issuer                                                                                                                                                                                  
        metadata:                                                                                                                                                                                     
          name: cm-prod                                                                                                                                                                               
        spec:                                                                                                                                                                                         
          acme:                                                                                                                                                                                       
          # The ACME server URL                                                                                                                                                                       
            server: https://acme-v02.api.letsencrypt.org/directory                                                                                                                                    
            # Email address used for ACME registration                                                                                                                                                
            email: your_email_address                                                                                                                                                             
            # Name of a secret used to store the ACME account private key                                                                                                                             
            privateKeySecretRef:                                                                                                                                                                      
              name: cm-prod                                                                                                                                                                           
            # Enable the HTTP-01 challenge provider                                                                                                                                                   
            http01: {}
        

As above, ensure you provide your email address and create the issuer:

kubectl create -f production-issuer.yaml
        

Check the status of the staging issuer:

kubectl describe issuer cm-stagingName:         cm-staging                                                                                                                                                                      
        Namespace:    default                                                                                                                                                                         
        Labels:                                                                                                                                                                                 
        Annotations:                                                                                                                                                                            
        API Version:  certmanager.k8s.io/v1alpha1                                                                                                                                                     
        Kind:         Issuer                                                                                                                                                                          
        Metadata:                                                                                                                                                                                     
          Creation Timestamp:  2019-06-25T03:28:16Z                                                                                                                                                   
          Generation:          1                                                                                                                                                                      
          Resource Version:    143856                                                                                                                                                                 
          Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/default/issuers/cm-staging                                                                                                
          UID:                 452908da-96f9-11e9-b5fc-0a580aed39b8                                                                                                                                   
        Spec:                                                                                                                                                                                         
          Acme:                                                                                                                                                                                       
            Email:  your_email_address                                                                                                                                                            
            Http 01:                                                                                                                                                                                  
            Private Key Secret Ref:                                                                                                                                                                   
              Name:  cm-staging                                                                                                                                                                       
            Server:  https://acme-staging-v02.api.letsencrypt.org/directory                                                                                                                           
        Status:                                                                                                                                                                                       
          Acme:                                                                                                                                                                                       
            Uri:  https://acme-staging-v02.api.letsencrypt.org/acme/acct/9718789                                                                                                                      
          Conditions:                                                                                                                                                                                 
            Last Transition Time:  2019-06-25T03:28:17Z                                                                                                                                               
            Message:               The ACME account was registered with the ACME server                                                                                                               
            Reason:                ACMEAccountRegistered                                                                                                                                              
            Status:                True                                                                                                                                                               
            Type:                  Ready                                                                                                                                                              
        Events:
        

Enable TLS on chartmuseum

Edit your values.yaml file for chartmuseum and look for the Ingress section. Pay special attention to the added configuration commands.

annotations:                                                                                                                                                                                
            kubernetes.io/ingress.class: nginx                                                                                                                                                        
            kubernetes.io/tls-acme: "true"                                                                                                                                                            
            certmanager.k8s.io/issuer: "cm-staging"                                                                                                                                                      
            certmanager.k8s.io/acme-challenge-type: http01                                                                                                                                            
                                                                                                                                                                                                      
        ## Chartmuseum Ingress hostnames                                                                                                                                                              
        ## Must be provided if Ingress is enabled                                                                                                                                                     
        ##                                                                                                                                                                                            
          hosts:                                                                                                                                                                                      
            - name: chartmuseum.acme.com                                                                                                                                                         
              path: /                                                                                                                                                                                 
              tls: true                                                                                                                                                                               
              tlsSecret: cm-tls
        

Upgrade your helm chart:

helm upgrade chartmuseum stable/chartmuseum -f values.yamlRelease "chartmuseum" has been upgraded. Happy Helming!
        

Cert-manager will read the annotations and create a certificate:

kubectl get certificate                                                                                                                                            
        NAME        READY   SECRET      AGE                                                                                                                                                           
        cm-tls      True    cm-tls      45m
        

Take a quick peak at the certificate:

kubectl describe certificate cm-tlsName:         cm-tls                                                                                                                                                                                               
        Namespace:    default                                                                                                                                                                                              
        Labels:       app=chartmuseum                                                                                                                                                                                      
                      chart=chartmuseum-2.3.1                                                                                                                                                                              
                      dns=ocidns                                                                                                                                                                                           
                      heritage=Tiller                                                                                                                                                                                      
                      release=chartmuseum                                                                                                                                                                                  
        Annotations:                                                                                                                                                                                                 
        API Version:  certmanager.k8s.io/v1alpha1                                                                                                                                                                          
        Kind:         Certificate                                                                                                                                                                                          
        Metadata:                                                                                                                                                                                                          
          Creation Timestamp:  2019-06-25T03:33:47Z                                                                                                                                                                        
          Generation:          1                                                                                                                                                                                           
          Owner References:                                                                                                                                                                                                
            API Version:           extensions/v1beta1                                                                                                                                                                      
            Block Owner Deletion:  true                                                                                                                                                                                    
            Controller:            true                                                                                                                                                                                    
            Kind:                  Ingress                                                                                                                                                                                 
            Name:                  chartmuseum-chartmuseum                                                                                                                                                                 
            UID:                   d5a9cee1-9673-11e9-a790-0a580aed1020                                                                                                                                                    
          Resource Version:        152567                                                                                                                                                                                  
          Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/cm-tls                                                                                                                
          UID:                     0ac806f6-96fa-11e9-8836-0a580aed4b3e                                                                                                                                                    
        Spec:                                                                                                                                                                                                              
          Acme:                                                                                                                                                                                                            
            Config:                                                                                                                                                                                                        
              Domains:                                                                                                                                                                                                     
                chartmuseum.acme.com                                                                                                                                                                                  
              Http 01:                                                                                                                                                                                                     
                Ingress Class:  nginx                                                                                                                                                                                      
          Dns Names:                                                                                                                                                                                                       
            chartmuseum.acme.com                                                                                                                                                                                      
          Issuer Ref:                                                                                                                                                                                                      
            Kind:       Issuer                                                                                                                                                                                             
            Name:       cm-staging                                                                                                                                                                                         
          Secret Name:  cm-tls                                                                                                                                                                                             
        Status:                                                                                                                                                                                                            
          Conditions:                                                                                                                                                                                                      
            Last Transition Time:  2019-06-25T04:19:27Z                                                                                                                                                                    
            Message:               Certificate is up to date and has not expired                                                                                                                                           
            Reason:                Ready                                                                                                                                                                                   
            Status:                True                                                                                                                                                                                    
            Type:                  Ready                                                                                                                                                                                   
          Not After:               2019-09-23T03:19:27Z                                                                                                                                                                    
        Events:                                                                                                                                                                                                            
          Type    Reason              Age                  From          Message                                                                                                                                           
          ----    ------              ----                 ----          -------                                                                                                                                           
          Normal  Cleanup             44m                  cert-manager  Deleting old Order resource "cm-tls-3817114402"                                                                                                   
          Normal  OrderCreated        43m (x2 over 44m)    cert-manager  Created Order resource "cm-tls-3433596774"                                                                                                        
          Normal  OrderComplete       43m                  cert-manager  Order "cm-tls-3433596774" completed successfully                                                                                                  
          Normal  Cleanup             2m21s                cert-manager  Deleting old Order resource "cm-tls-3433596774"                                                                                                   
          Normal  Generated           2m14s (x3 over 47m)  cert-manager  Generated new private key                                                                                                                         
          Normal  GenerateSelfSigned  2m14s (x3 over 47m)  cert-manager  Generated temporary self signed certificate                                                                                                       
          Normal  OrderCreated        2m14s (x3 over 47m)  cert-manager  Created Order resource "cm-tls-3817114402"                                                                                                        
          Normal  OrderComplete       2m13s (x2 over 47m)  cert-manager  Order "cm-tls-3817114402" completed successfully                                                                                                  
          Normal  CertIssued          2m13s (x3 over 47m)  cert-manager  Certificate issued successfully
        

Once complete, cert-manager will have created a secret with the details of the certificate based on the secret used in the ingress resource. You can use the describe command as well to see some details:

kubectl describe secret cm-tls                                                                                                                                                                
        Name:         cm-tls                                                                                                                                                                                               
        Namespace:    default                                                                                                                                                                                              
        Labels:       certmanager.k8s.io/certificate-name=cm-tls                                                                                                                                                           
        Annotations:  certmanager.k8s.io/alt-names: chartmuseum.acme.com                                                                                                                                              
                      certmanager.k8s.io/common-name: chartmuseum.acme.com                                                                                                                                            
                      certmanager.k8s.io/ip-sans:                                                                                                                                                                          
                      certmanager.k8s.io/issuer-kind: Issuer                                                                                                                                                               
                      certmanager.k8s.io/issuer-name: cm-staging                                                                                                                                                           
                                                                                                                                                                                                                           
        Type:  kubernetes.io/tls                                                                                                                                                                                           
                                                                                                                                                                                                                           
        Data                                                                                                                                                                                                               
        ====                                                                                                                                                                                                               
        ca.crt:   0 bytes                                                                                                                                                                                                  
        tls.crt:  3578 bytes                                                                                                                                                                                               
        tls.key:  1679 bytes
        

If you access chartmuseum now, you’ll see a warning:

Warning message in browser

If you ignore the warning and go ahead anyway, you will be able to access chartmuseum except that now you will be accessing it over https. Your browser will also warn you that it’s added an exception to this site.

Edit the values.yaml for your chartmuseum again and this time change the issuer annotation to cm-prod:

certmanager.k8s.io/issuer: "cm-prod"
        

Run a helm upgrade again:

helm upgrade chartmuseum stable/chartmuseum -f values.yamlRelease "chartmuseum" has been upgraded. Happy Helming!
        

and delete the secret:

kubectl delete secret cm-tls
        

This will cause cert-manager to get a new certificate. You can verify this:

kubectl describe certificate cm-tls
        .
        .
        .Normal  Generated           33s (x4 over 61m)  cert-manager  Generated new private key                                                                                                                           
          Normal  GenerateSelfSigned  33s (x4 over 61m)  cert-manager  Generated temporary self signed certificate                                                                                                         
          Normal  OrderCreated        33s (x4 over 58m)  cert-manager  Created Order resource "cm-tls-3433596774"                                                                                                          
          Normal  CertIssued          31s (x5 over 61m)  cert-manager  Certificate issued successfully                                                                                                                     
          Normal  OrderComplete       31s (x3 over 57m)  cert-manager  Order "cm-tls-3433596774" completed successfully
        

If you now access chartmuseum, you will be able to access it with https and will not be prompted with the security warning/exception.

Chartmuseum welcome message

Conclusion

Chartmuseum enhances your CI/CD capabilities in a number of ways:

  • enables you to host your helm charts privately and securely
  • integrates with your CI/CD deployment tool chain and pipelines
  • supports multiple teams and organizations with multitenant capabilities
  • uses a variety of storage capabilities including local file system, Oracle Object Storage, OpenStack Object storage, and others.

If you want to run it outside your Kubernetes cluster (like on a VM), you can follow this guide instead.

I hope you’ve found this useful!

References: