Phase 2 AWS - core services

1 month ago · 25 min read

Server v4.8 Server Admin

Before you begin with the CircleCI server v4.8 core services installation phase, ensure all prerequisites are met, as described in the following page:

Phase 1 - AWS Prerequisites

In the following sections, replace any sections indicated by < > with your details.

1. Create a namespace

Create a namespace to install the application into.

kubectl create ns <namespace>

Once you have created your namespace, we recommend setting your kubectl context too, with the following command: kubectl config set-context --current --namespace <namespace>

2. Pull images

Credentials to pull the images from CircleCI’s image registry will be provided to you as part of the onboarding process. A docker-registry Kubernetes Secret will be used to pull images from Azure Container Registry (ACR). You have two options, depending on whether your application has access to the public internet.

Public
Private

Option 1: Your application has access to the public internet.

This example creates a Kubernetes Secret to enable deployments to pull images from CircleCI’s image registry. The docker-registry Kubernetes Secret takes the following form:

kubectl -n <namespace> create secret docker-registry regcred \
  --docker-server=cciserver.azurecr.io \
  --docker-username=<your-username> \
  --docker-password="<provided-token>" \
  --docker-email=<your-contact-email>

Option 2: Your application does NOT have access to the public internet.

The credentials provided to you allow you to pull and store copies of our images locally. Pull and store the images in whichever Docker repository you have available. The docker-registry Kubernetes Secret takes the following form:

kubectl -n <namespace> create secret docker-registry regcred \
  --docker-server=<your-docker-image-repo> \
  --docker-username=<your-username> \
  --docker-password=<your-access-token> \
  --docker-email=<your-email>

3. Create Helm values

Before installing CircleCI, it is recommended to create a new values.yaml file unique to your installation. The Installation Reference section contains some example values.yaml files that are a good place to start. The following describes the minimum required values to include in values.yaml. Additional customizations are available, see the provided values.yaml for all available options.

For sensitive data there are two options:

Add into the values.yaml file
Add them as Kubernetes Secrets directly

This flexibility allows you to manage Kubernetes Secrets using whichever process you prefer. Whichever option you choose, this sensitive information is stored as a Kubernetes Secret within CircleCI.

During the installation process, you may use the following command to generate a random alphanumeric value as required: cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | head -c <num-of-chars>. This command should work on any *nix based system.

a. API token

The application requires a Kubernetes Secret containing an API token. This API token is used to facilitate internal API communication to api-service. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic api-token \
  --from-literal=api-token=<your-super-secret-random-value>

kubectl -n <namespace> annotate secret/api-token \
  meta.helm.sh/release-name=<helm-release-name> \
  meta.helm.sh/release-namespace=<namespace> \
  helm.sh/resource-policy=keep --overwrite

kubectl -n <namespace> label secret/api-token \
  app.kubernetes.io/managed-by=Helm --overwrite

Option 2: CircleCI creates the Kubernetes Secret for you.

CircleCI will create the Kubernetes Secret "api-token" automatically.

The application requires a session cookie key Kubernetes Secret, which CircleCI uses to sign session cookies. The Secret must be exactly 16 characters long. Use a random string and store it securely. CircleCI will not be able to recover this value if lost. There are two options depending on whether you want to create the Kubernetes Secret, or if you want CircleCI to create it for you.

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic session-cookie \
--from-literal=session-cookie-key=<your-secret-key-16-chars>

kubectl -n <namespace> annotate secret/session-cookie \
  meta.helm.sh/release-name=<helm-release-name> \
  meta.helm.sh/release-namespace=<namespace> \
  helm.sh/resource-policy=keep --overwrite

kubectl -n <namespace> label secret/session-cookie \
  app.kubernetes.io/managed-by=Helm --overwrite

Option 2: CircleCI creates the Kubernetes Secret for you.

CircleCI will create the Kubernetes Secret "session-cookie" automatically.

c. Encryption

The application requires a Kubernetes Secret containing signing and encryption keysets. These keysets are used to encrypt and sign artifacts generated by CircleCI. These keys were created during the prerequisites phase (GCP prerequisites, AWS prerequisites). CircleCI will not be able to recover the values if lost. Depending on how you prefer to manage Kubernetes Secrets, there are two options.

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic signing-keys \
  --from-literal=signing-key=<your-generated-signing-key> \
  --from-literal=encryption-key=<your-generated-encryption-key>

Option 2: CircleCI creates the Kubernetes Secret.

Add the value to values.yaml. CircleCI will create the Secret automatically.

keyset:
  signing: '<your-generated-signing-key>'
  encryption: '<your-generated-encryption-key>'

d. PostgreSQL

i. Credentials

The application requires a Kubernetes Secret containing PostgreSQL credentials. This is true when using either the internal (default) or an externally hosted instance of PostgreSQL. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

You create Secret
CircleCI creates Secret

Option 1: Create the Secret yourself.

kubectl -n <namespace> create secret generic postgresql \
  --from-literal=postgres-password=<postgres-password>

You must then provide the following to the values.yaml file:

postgresql:
  auth:
    existingSecret: postgresql

Option 2: CircleCI creates the Kubernetes Secret.

Add the credentials to values.yaml, and CircleCI will create the Secret automatically.

postgresql:
  auth:
    postgresPassword: "<postgres-password>"

ii. TLS

PostgreSQL may be extended to use TLS encrypted traffic. When deployed internally, this option is disabled by default but may be enabled by adding the following to your PostgreSQL block of your values.yaml

postgresql:
  ...
  tls:
    enabled: true
    autoGenerated: true # Generate automatically self-signed TLS certificates

Certificate files may also be provided, rather than autogenerated. In this case, create a Secret containing the TLS certs and keys needed.

kubectl -n <namespace> create secret generic postgres-tls-secret --from-file=./cert.pem --from-file=./cert.key --from-file=./ca.pem

Then the PostgreSQL block in your values.yaml will contain the contents below.

postgresql:
  ...
  tls:
    enabled: true
    certificatesSecret: "postgres-tls-secret" # Name of an existing secret that contains the certificates
    certFilename: "cert.pem" # Certificate filename
    certKeyFilename: "cert.key" # Certificate key filename
    certCAFilename: "ca.pem" # CA Certificate filename

e. MongoDB credentials

The application requires a Kubernetes Secret containing MongoDB credentials. This is true when using either the internal (default) or an externally hosted instance of MongoDB. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic mongodb-credentials \
  --from-literal=mongodb-root-password=<root-password> \
  --from-literal=mongodb-password=<user-password>

You must then provide the following to the values.yaml file:

mongodb:
  auth:
    existingSecret: mongodb-credentials

Option 2: CircleCI creates the Kubernetes Secret.

Add the credentials to values.yaml, and CircleCI will create the Secret automatically.

mongodb:
  auth:
    rootPassword: "<root-password>"
    password: "<user-password>"

f. RabbitMQ configurations and auth Secrets

The RabbitMQ installation requires two random alphanumeric strings. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are two options.

You create Secret
CircleCI creates Secret

Option 1: Create the Secret yourself.

kubectl -n <namespace> create secret generic rabbitmq-key \
--from-literal=rabbitmq-password=<secret-alphanumeric-password> \
--from-literal=rabbitmq-erlang-cookie=<secret-alphanumeric-key>

You must then provide the following to the values.yaml file:

rabbitmq:
  auth:
    existingPasswordSecret: rabbitmq-key
    existingErlangSecret: rabbitmq-key

Option 2: CircleCI creates the Kubernetes Secret.

Add the value to values.yaml, and CircleCI will create the Kubernetes Secret automatically.

rabbitmq:
  auth:
    password: "<secret-alphanumeric-password>"
    erlangCookie: "<secret-alphanumeric-key>"

g. Pusher Kubernetes Secret

The application requires a Kubernetes Secret for Pusher. CircleCI will not be able to recover the values if lost. Based on how you prefer to manage Kubernetes Secrets there are 2 options:

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic pusher \
--from-literal=secret=<pusher-secret>

kubectl -n <namespace> annotate secret/pusher \
  meta.helm.sh/release-name=<helm-release-name> \
  meta.helm.sh/release-namespace=<namespace> \
  helm.sh/resource-policy=keep --overwrite

kubectl -n <namespace> label secret/pusher \
  app.kubernetes.io/managed-by=Helm --overwrite

Option 2: CircleCI creates the Kubernetes Secret.

CircleCI will create the Kubernetes Secret pusher automatically.

h. Global

All values in this section are children of global in your values.yaml.

CircleCI domain name (required)

Enter the domain name you specified when creating your Frontend TLS key and certificate (AWS, GCP).

global:
  ...
  domainName: "<full-domain-name-of-your-install>"

License

A license will be provided by CircleCI, add it to values.yaml:

global:
  ...
  license: '<license>'

Registry

The registry to pull images from will have been provided to you, or you may have added the images to your own hosted registry. Add the registry to values.yaml:

global:
  ...
  container:
    registry: <registry-domain eg: cciserver.azurecr.io >
    org: <your-org-if-applicable>

i. Add Elastic IP addresses

If you provisioned an Elastic IP addresses (AWS) in the prerequisites, you can now add the values under the nginx block.

For AWS, under nginx annotations, add the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation with each of the AllocationId values generated as a comma separated list. The number of `AllocationId`s must match the number of subnets the load balancer is deployed into (default 3).

nginx:
  ...
  annotations:
    ...
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <eip-id-1>,<eip-id-2>,<eip-id-3>

j. TLS

For TLS, you have a few options:

Lets Encrypt
Key & certificate
AWS Certificate Manager
Terminate TLS

Let’s Encrypt

Let’s Encrypt will request and manage certificates for you. This is a good option when the load balancer is publicly accessible. The following snippet (using your own email) can be added to values.yaml:

kong:
  acme:
    enabled: true
    email: contact@example.com

Let’s Encrypt may take up to 30 minutes to be reflected in your browser.

Supply a private key and certificate

You can supply a private key and certificate, which you may have created during the prerequisites steps. The key and certificates will need to be base64 encoded. You can retrieve and encode the values with the following commands:

cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/privkey.pem | base64
cat /etc/letsencrypt/live/<CIRCLECI_SERVER_DOMAIN>/fullchain.pem | base64

And add them to values.yaml:

tls:
  certificate: '<full-chain>'
  privateKey: '<private-key>'

Use ACM

Have AWS Certificate Manager (ACM) automatically request and manage certificates for you. Follow the ACM documentation for instructions on how to generate ACM certificates.

Enable aws_acm and add the service.beta.kubernetes.io/aws-load-balancer-ssl-cert annotation to point at the ACM ARN:

nginx:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm-arn>
  aws_acm:
    enabled: true

If you have already deployed CircleCI server, enabling ACM is a destructive change to the load balancer. The service will have to be regenerated to allow the use of your ACM certificates and so the associated load balancer will also be regenerated. You will need to update your DNS records to the new load balancer once you have redeployed CircleCI server.

Disable TLS within CircleCI

You can choose to disable TLS termination within CircleCI. The system will still need to be accessed over HTTPS, so TLS termination will be required somewhere upstream of CircleCI. Implement this by following the first option (do nothing) and forward the following ports to your CircleCI load balancer:

Frontend / API Gateway [TCP 80, 443]
Nomad server [TCP 4647]

k. GitHub integration

To configure GitHub with CircleCI, there are two options for providing credentials to the deployment. Steps for both GitHub and GitHub Enterprise (GHE) are given in the next two sections.

i. GitHub

These instructions are for the GitHub.com, not GitHub Enterprise. Use the client ID and secret you created with your GitHub OAuth application in the prerequisites phase (AWS, GCP).

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic github-secret \
  --from-literal=clientId=<client-id> \
  --from-literal=clientSecret=<client-secret>

Option 2: CircleCI creates the Kubernetes Secret.

Add the client ID and Secret to the values.yaml file. CircleCI will create the Kubernetes Secret automatically.

github:
  clientId: "<client-id>"
  clientSecret: "<client-secret>"

ii. GitHub Enterprise

The instructions for GitHub Enterprise are similar, with a few extra steps to enable Enterprise and create the required default token.

In the case of GitHub Enterprise add the defaultToken created in the prerequisites phase (AWS, GCP) to the GitHub section. The hostname should not include the protocol, ex: github.exampleorg.com.

You create Secret
CircleCI creates Secret

Option 1: Create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic github-secret \
  --from-literal=clientId=<client-id> \
  --from-literal=clientSecret=<client-secret> \
  --from-literal=defaultToken=<default-token>

You must then provide the following to the values.yaml file:

github:
  enterprise: true
  hostname: "<github-enterprise-hostname>"

Option 2: CircleCI creates the Kubernetes Secret.

Add clientID, clientSecret and defaultToken to the values.yaml file. You must also set enterprise to true, and provide the hostname for your enterprise GitHub. CircleCI will create the Kubernetes Secret automatically.

github:
  ...
  clientId: "<client-id>"
  clientSecret: "<client-secret>"
  enterprise: true
  hostname: "<github-enterprise-hostname>"
  defaultToken: "<token>"

l. Object storage

Regardless of your storage provider, the bucket name you created during the prerequisites phase will need to be included.

object_storage:
  bucketName: "<bucket-name>"

S3 compatible

Add an s3 section as a child of object_storage. The endpoint in the case of AWS S3 is the regional endpoint, it is of the form https://s3.<region>.amazonaws.com. Otherwise it is the API endpoint fo your object storage server.

object_storage:
  ...
  s3:
    enabled: true
    endpoint: "<storage-server-or-s3-endpoint>"

Under object_storage.s3, you may provide the accessKey and secretKey, the irsaRole, or nothing. They were created during the prerequisites steps.

Use IAM keys
Use IRSA
You create Secret

Option 1: Use IAM keys.

Add the following to the object_storage.s3 section:

object_storage:
  ...
  s3:
    ...
    accessKey: "<access-key>"
    secretKey: "<secret-key>"

Option 2: Use IRSA.

A Kubernetes Secret will automatically be generated for you using your credentials.

Add the following to the object_storage.s3 section:

object_storage:
  ...
  s3:
    ...
    region: "<role-region>"
    irsaRole: "<irsa-arn>"

Disable Presigned Mode (Optional) If you wish to store artifacts larger than 5GB, you will need to update your trust policy for your IRSA role. Then disable presigned mode by adding the following to the object_storage.s3 section:

object_storage:
  ...
  s3:
    ...
    presigned: false
    storageRole: "<irsa-arn>"

Option 3: Create the Kubernetes Secret yourself

Instead of providing AWS accessKey and secretKey credentials in your values.yaml file, you may choose to create the Kubernetes Secret yourself.

kubectl -n <namespace> create secret generic object-storage-secret \
  --from-literal=s3AccessKey=<access-key> \
  --from-literal=s3SecretKey=<secret-key>

CircleCI server will use the credentials provided to authenticate to S3.

m. Installing behind a proxy

Depending on your security requirements, you might want to install CircleCI server behind a proxy. Installing behind a proxy gives you the power to monitor and control access between your installation and the broader Internet. For further information including limitations of installation behind a proxy, see the Installing server behind a proxy guide.

The following fields need to be configured in your values.yaml:

Toggle proxy.enabled to "1"
Enter details for proxy.http.host and proxy.https.host, along with their associated ports. These values can be the same but they both need to be configured.
For authentication you will need to configure proxy.http.auth.enabled and proxy.https.auth.enabled as "1". You will also need to configure the respective username and password for both HTTP and HTTPS.
configure the no_proxy hosts and subnets. This should include localhost, your GitHub Enterprise host (optional), the hostname of your CircleCI installation (see Known Limitations for an explanation), and the CIDR of Nomad.

proxy:
  enabled: "1"
  http:
    host: "<proxy.example.internal>"
    port: "3128"
    auth:
      enabled: "1"
      username: "<proxy-user>"
      password: "<proxy-password>"
  https:
    host: "<proxy.example.internal>"
    port: "3128"
    auth:
      enabled: "1"
      username: "<proxy-user>"
      password: "<proxy-password>"
  no_proxy:
    - localhost
    - 127.0.0.1
    - "<github.example.internal>"
    - "<circleci.example.internal>"
    - "<nomad-subnet-cidr>"
    - "<vpc-or-subnet-cidr>"   # VPC or subnets to exclude from the proxy (optional)

n. Encrypting environment variables

All environment variables stored in contexts are encrypted using either Google Tink or HashiCorp Vault. We recommend the use of Tink as Vault has been deprecated.

Use Tink

The following steps cover using Tink as an alternative to Vault:

Enable Tink in your values.yaml:
```
tink:
  enabled: true
  keyset: ""
```
When tink.enabled is true, Vault will not be deployed.

Tink or Vault must be set once during installation, and cannot be changed after deployment.
Generate a keyset, which Tink uses to manage key rotation. The easiest way to do this is to use Google’s Tinkey CLI utility. Once installed, use the following command:
```
tinkey create-keyset --key-template XCHACHA20_POLY1305
```
CircleCI server will store your generated keyset in a Kubernetes Secret. You may generate this Secret in either of the following ways:
- You create secret
- CircleCI creates secret
Option 1: Create the Kubernetes Secret yourself

Following the example below, create a Kubernetes Secret with the name tink and a key keyset. Apply this Secret to the namespace of your CircleCI server installation.
apiVersion: v1 kind: Secret metadata: name: tink data: keyset: <your-keyset>
Option 2: CircleCI server will create the Kubernetes Secret

You may add the keyset to your values.yaml under tink as in the example below. CircleCI will generate the required Secret to store your keyset.
tink: enabled: true keyset: "<your-keyset>"

If your Tink keyset is somehow lost, you will need to generate a new keyset and then recreate your contexts and their associated Secrets.

o. Setting up OIDC (optional)

CircleCI server optionally supports OpenID Connect (OIDC) tokens in jobs.

This feature is not enabled by default and requires a few additional steps to set up, as follows:

Generate a JSON Web Key (JWK) using the default parameters and copy the Public and Private Keypair Set to a secure and convenient location (for example, ~/oidc-service-jwk.json). This key pair will be used by oidc-service to sign the OIDC tokens used in jobs.
The JWK will need to be base64 encoded without line wrapping before using it on server:
- On Linux:
  $ base64 -w0 ~/oidc-service-jwk.json
- On macOS:
  $ base64 -b0 ~/oidc-service-jwk.json
  Copy the output to a secure location, so it can be referenced in the next step.

Finally, enable oidc-service and supply it the base64 encoded JWK from your server Helm values file:

oidc_service:
  isEnabled: true
  json_web_keys: << set your base64 encoded JWK from step 2 here >>

4. Deploy

Once you have completed the fields detailed above, you can deploy CircleCI’s core services:

USERNAME=<provided-username>
PASSWORD=<token>
namespace=<your-namespace>
helm registry login cciserver.azurecr.io/circleci-server -u $USERNAME -p $PASSWORD
helm install circleci-server oci://cciserver.azurecr.io/circleci-server -n $namespace --version 4.7.0 -f <path-to-values.yaml>

5. Create DNS entry

Create a DNS entry for your nginx load balancer, for example, circleci.your.domain.com and app.circleci.your.domain.com. The DNS entry should align with the DNS names used when creating your TLS certificate and GitHub OAuth app during the prerequisites steps. All traffic will be routed through this DNS record.

You need the IP address, or, if using AWS, the DNS name of the nginx load balancer. You can find this information with the following command:

kubectl get service -l app=circleci-proxy

6. Validation

You should now be able to navigate to your CircleCI server installation and log in to the application successfully.

Now we will move on to build services. It may take a while for all your services to be up. You can periodically check by running the following command (you are looking for the frontend pod to show a status of running and ready should show 1/1):

kubectl get pods -n <YOUR_CIRCLECI_NAMESPACE>

Machine provisioner and Nomad server pods are expected to fail at this stage. You will set up your execution environments in the next phase of the installation.

Next steps

Phase 3 AWS: Execution Environments Installation

[path]

[title]

Phase 2 AWS - core services

1. Create a namespace

2. Pull images

3. Create Helm values

a. API token

c. Encryption

d. PostgreSQL

i. Credentials

ii. TLS

e. MongoDB credentials

f. RabbitMQ configurations and auth Secrets

g. Pusher Kubernetes Secret

h. Global

CircleCI domain name (required)

License

Registry

i. Add Elastic IP addresses

j. TLS

k. GitHub integration

i. GitHub

ii. GitHub Enterprise

l. Object storage

S3 compatible

m. Installing behind a proxy

n. Encrypting environment variables

Use Tink

o. Setting up OIDC (optional)

4. Deploy

5. Create DNS entry

6. Validation

Next steps

Phase 2 AWS - core services

1. Create a namespace

2. Pull images

3. Create Helm values

a. API token

b. Session cookie

c. Encryption

d. PostgreSQL

i. Credentials

ii. TLS

e. MongoDB credentials

f. RabbitMQ configurations and auth Secrets

g. Pusher Kubernetes Secret

h. Global

CircleCI domain name (required)

License

Registry

i. Add Elastic IP addresses

j. TLS

k. GitHub integration

i. GitHub

ii. GitHub Enterprise

l. Object storage

S3 compatible

m. Installing behind a proxy

n. Encrypting environment variables

Use Tink

o. Setting up OIDC (optional)

4. Deploy

5. Create DNS entry

6. Validation

Next steps