Willis Tech
E-commerce Platform - staging env in EKS for 200+ stacks

E-commerce Platform - staging env in EKS for 200+ stacks

KubedExternal DNSCert ManagerNginx Ingress ControllerGomplateGit-cryptSamsonBitbucket Pipeline

What's the challenge?

  • Staging environment needs 200 stacks.
    Each stack contains three services (api, storefront, admin)
    You can access stack 1 service by api.stg1.service.com, shop.stg1.service.com and admin.stg1.service.com.

shop service

  • 200 stacks have many configurations and secrets need to manage.
  • Need a platform for DEV, PM and QA to deploy.
  • All services in staging must support TLS.

Why we choose EKS for staging env?

  • We use EKS in our production environment. We need a staging environment in EKS too. By doing so, we can make sure our architectures of different environments are the same and testable.
  • 50+ developers need to verify the result in staging environment parallelly. Using EKS can isolate the environment and benefit from Kubernetes community.

In this project, I contributed to those items

  • Built staging environment in EKS to host 200 stacks and all other e-commerce services (eg: open-api, third-party-api).

  • Created a document and hosted a training session for DEV, QA to teach them how to use EKS.
    After two months, most of our developers can work with EKS efficiently.

  • Used Gomplate (templating tool) to generate ConfigMap file for 200 stacks. By doing so, we only need to mange one file.
    For example, the following ConfigMap file can generate the ConfigMap file with desired STACK_NO:

    metadata:
  name: custom-env
data:
  APP_CONFIG_HOST: api.stg{{.Env.STACK_NO}}.service.com
  APP_CONFIG_HOST: api.stg{{.Env.STACK_NO}}.service.com

  • Used External DNS, Cert Manger and Nginx Ingress Controller to serve all the staging service with HTTPS.

  • Used Kubed to duplicate TSL certificate within EKS cluster. Because we sign 100 domains in one certificate to reduce request count to Let’s Encrypt.

  • Used Git-Ctypt to encrypt secret and commit to our version control. The concept is like Sealed Secret.

  • Set up Samson CI/CD platform for users to deploy services.

  • Our Staging cluster has 120+ worker nodes. We used different node type to fulfill different workloads. For example: System Node, Memory Node and CPU Node. By doing so, we can use computing resource efficiently.