Kubernetes, Part 3

This post will document some of the issues I ran into while setting up Kubernetes for this site.

Handling www

I want requests to www.andrewtchin.com and andrewtchin.com to display the same site. I have a CNAME from www.andrewtchin.com to andrewtchin.com but did not configure that in the first iteration of the Ingress. This led to a TLS error upon connecting to www.andrewtchin.com since the Ingress used the default self-signed TLS certificate instead of a Let’s Encrypt certificate.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prd"
  name: andrewtchin-com-nginx-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - andrewtchin.com
    secretName: andrewtchin-com-tls
  rules:
  - host: andrewtchin.com
    http:
      paths:
      - path: /
        backend:
          serviceName: andrewtchin-com
          servicePort: 80

To achieve the desired result I simply had to add the additional TLS host and rule for www.andrewtchin.com.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prd"
  name: andrewtchin-com-nginx-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - andrewtchin.com
    - www.andrewtchin.com
    secretName: andrewtchin-com-tls
  rules:
  - host: andrewtchin.com
    http:
      paths:
      - path: /
        backend:
          serviceName: andrewtchin-com
          servicePort: 80
  - host: www.andrewtchin.com
    http:
      paths:
      - path: /
        backend:
          serviceName: andrewtchin-com
          servicePort: 80

As shown here, tls describes the hosts that the Ingress terminates with the corresponding secret (containing both DNS names). Then, rules describes which service traffic matching the host and path should be directed to.

⇒  kubectl describe ingress andrewtchin-com
...
TLS:
  andrewtchin-com-tls terminates andrewtchin.com,www.andrewtchin.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  andrewtchin.com
                       /   andrewtchin-com:80 (10.244.0.62:80)
  www.andrewtchin.com
                       /   andrewtchin-com:80 (10.244.0.62:80)

Certificate Regeneration

After configuring the www subdomain in the Ingress, I needed the Let’s Encrypt certificate to be regenerated to include the additional DNS name. To do this I deleted the secret as below:

⇒  kubectl delete secret andrewtchin-com-tls

After this, visiting www.andrewtchin.com was still not using the correct TLS certificate. Viewing the certificate showed that it was waiting for the certificate request to complete:

⇒  kubectl describe certificate andrewtchin-com-tls
Name:         andrewtchin-com-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         Certificate
...
Spec:
  Dns Names:
    andrewtchin.com
    www.andrewtchin.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       letsencrypt-prd
  Secret Name:  andrewtchin-com-tls
Status:
  Conditions:
    Last Transition Time:  2019-12-28T00:21:18Z
    Message:               Waiting for CertificateRequest "andrewtchin-com-tls-480732090" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason          Age                       From          Message
  ----    ------          ----                      ----          -------
  Normal  Requested       7m20s (x2 over 7m50s)     cert-manager  Created new CertificateRequest resource "andrewtchin-com-tls-480732090"
  Normal  GeneratedKey    7m20s                     cert-manager  Generated a new private key
  Normal  PrivateKeyLost  7m20s                     cert-manager  Lost private key for CertificateRequest "andrewtchin-com-tls-480732090", deleting old resource
  Normal  Issued          2m50s (x1336 over 7m24s)  cert-manager  Certificate issued successfully

Interestingly, the certificate request showed that it was successful:

⇒  kubectl describe certificaterequest andrewtchin-com-tls-480732090
Name:         andrewtchin-com-tls-480732090
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: andrewtchin-com-tls
              cert-manager.io/private-key-secret-name: andrewtchin-com-tls
API Version:  cert-manager.io/v1alpha2
Kind:         CertificateRequest
...
  Conditions:
    Last Transition Time:  2019-12-28T00:21:18Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason             Age    From          Message
  ----    ------             ----   ----          -------
  Normal  CertificateIssued  9m31s  cert-manager  Certificate fetched from issuer successfully

To reissue the certificate, I deleted it:

⇒  kubectl delete certificate andrewtchin-com-tls

This resolved the problem and I was able to access both andrewtchin.com and www.andrewtchin.com correctly.

⇒  kubectl describe certificate andrewtchin-com-tls
Name:         andrewtchin-com-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         Certificate
...
Spec:
  Dns Names:
    andrewtchin.com
    www.andrewtchin.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       letsencrypt-prd
  Secret Name:  andrewtchin-com-tls
Status:
  Conditions:
    Last Transition Time:  2019-12-28T00:38:21Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-03-26T23:38:20Z
Events:
  Type    Reason     Age                From          Message
  ----    ------     ----               ----          -------
  Normal  Requested  35s                cert-manager  Created new CertificateRequest resource "andrewtchin-com-tls-480732090"
  Normal  Issued     32s (x2 over 35s)  cert-manager  Certificate issued successfully

Deploy

For now, deployment is manual. To deploy, I delete the pod and the new image gets pulled when the replacement is created. I will experiment with setting up deployment from Gitlab next.

⇒  kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
andrewtchin-com-55dfd65667-55ngf                     1/1     Running   0          20h

⇒  kubectl delete pod andrewtchin-com-55dfd65667-55ngf
pod "andrewtchin-com-55dfd65667-55ngf" deleted

⇒  kubectl get pods
NAME                                                 READY   STATUS              RESTARTS   AGE
andrewtchin-com-55dfd65667-mmzb2                     0/1     ContainerCreating   0          5s

⇒  kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
andrewtchin-com-55dfd65667-mmzb2                     1/1     Running   0          26s

Conclusion

Overall I am very happy with this setup and look forward to deploying other apps on the cluster!