What is CI/CD?

Imagine there are multiple developers who make code changes every day. These changes need to be integrated yet not introduce errors. They need to be tested before being deployed. What can be done? The test is automated and then deployed to the development/staging environment before it goes to production. This automation is triggered with every revision. The process is called continuous integration (CI).

After the integration and testing are complete, the application needs to be deployed automatically and continuously to production. This is called continuous deployment. Alongside this is continuous delivery, wherein the application is delivered continuously to production but requires approval before deployment. Once approved, the process continues to trigger scripts or methods to deploy the application.

Example of CI/CD pipeline

The CI/CD pipeline or workflow may be different for each organization. It depends on the programming language used, the organization’s policy, CI/CD security tools, the deployment target, the application type, etc.

Here’s an example of a workflow.

You can see that there are some activities in the CI, such as Plan, Code, Build, and Test. Let’s explore them in brief.

• Plan: Before an application is deployed, features and bug fixing need to be planned. Jira, Trello and Basecamp are some tools that can help plan features.
• Code: Here, developers write code changes for features or fix bugs. Tools used include Visual Studio Code, Geany, and Visual Studio, among others. The tools depend on the programming language of the application.
• Build: The build process can be automated using certain tools. Some recommendations are Github Action, CircleCI, and GitLab CI. The build result is usually as packages, binaries, zip files, etc. Note that different programming languages may have different steps and results.
• Test: After the code is built, it needs to be tested. Test units are usually run here. The test process is not only a unit test, however; you can also do integration tests, performance tests, functional tests, and acceptance tests.

If the test is passed, the code is released and deployed. You can add the approval process before the deployment process as mentioned before, i.e., continuous delivery.

The next stage is monitoring. Some of the tools used are Grafana, Prometheus, Jaeger, AWS CloudWatch, and Datadog. The feedback from the monitoring process is valuable for the next planning process. Think of it as a loop.

CI/CD best practices

Security can be incorporated in the build or test step. You can add security checking in the pull request actions to avoid security problems before merging the changes from the request.

You can add static code analysis, which is a process that will analyze code and check for vulnerabilities. There are many other security practices that you can add to your workflow, such as dynamic program analysis, penetration testing, etc.

Below are a few general CI/CD security best practices. They aren’t specific to a platform.

Don’t use plain text credentials

A mistake that some people make is to use hard-coded or plain text credentials in the CI/CD pipeline. Use encrypted credentials instead. You can use a secret manager to handle multiple secrets which are reused at many CI/CD pipelines. Credentials can include, but are not limited to, passwords, secrets tokens, and API keys.

Rotate credentials used by CI/CD pipeline

If you create new keys or update new passwords, you will need to update multiple CI/CD pipelines. A secret manager can be used to rotate credentials. You’ll only need to update the secrets in the same.

Validate inputs

You should validate each input in the CI/CD pipeline. For example, you have a pipeline that allows a user to trigger it with an input to define the version you want to deploy. You have a script to publish by that version, but do not validate it and use the value as it is.

When you don’t validate the input, attackers or hackers can use it to inject scripts (script injection). They may do this to find out credentials, download malware, and so on. Hence, it’s crucial to validate inputs to make sure they are expected.

Use OpenID authentication

Consider using OpenID as an authentication option in your CI/CD pipeline as it reduces the need to store credentials or secrets. You should also use it when connecting to cloud services like AWS, Azure, and GCP.

Having a sound understanding of basic cyber security is a plus so that you can implement the best practices in your CI/CD pipeline. If possible, learn more about DevOps (development and operations) and DevSecOps (development, security, and operations).

To sum up some of the important CI/CD security best practices, ensure you don’t expose your credentials, use a secret manager whenever possible, and use OpenID as an authentication method when connecting to cloud services. Implementing the principle of least privilege is an additional step you can take.

Security practices for continuous integration

The Continuous Delivery Foundation (CDF) provides best practices for CI/CD. For CI, these are:

• Have a complete application package in the build step that contains the binaries, version tag, and difference report.
• Don’t blindly include all files; you should know what you add to the packages. This way, you can avoid including injected files that may pose a security risk.
• Try to understand the build parameters. You don’t want to deliver debug symbols to production.

This reference contains CDF's best practices.

It’s important to conduct unit testing in CI to reduce the chance of delivering defective software. Unit testing validates each unit of the code to ensure it performs as expected.

You can check here for the published config pull request, and here for the sample running job. You can also check here for the published package.

Sample:

Let’s implement a security practice to have source code analysis in our previous workflow.

1. Create a file with the name “codeql.yml” under the directory with the name “.github/workflows”.
2. Copy and paste the following code:

name: "CodeQL"
on:
push:
branches: [ "main" ]
pull_request:
The branches below must be a subset of the branches above
branches: [ "main" ]
schedule:

cron: '37 18 * * 0'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'csharp' ]
steps:

name: Checkout repository
uses: actions/checkout@v3

Initializes the CodeQL tools for scanning.


name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}


name: Autobuild
uses: github/codeql-action/autobuild@v2


name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
with:
category: "/language:${{matrix.language}}"

1. You can push and pull requests to check the result.

Security practices for continuous delivery/deployment

Here are some best practices for CD provided by CDF:

• Have approval and approval gates. This is usually done using notification. For example, asking the product owner to approve the changes.
• There are some common deployment methods, such as canary, blue/green, and rolling blue/green deployments. You need to consider which model fits your business.
• The CD process should have immutable deployments. This means you should not have manual changes.
• You should define policies and rules that align with the organization's needs. They will control the automation process for continuous delivery/deployment.

Security practices can be implemented after delivering the packages. For example, if you use Docker and push docker image, you can scan it to check the binaries that are delivered to it. There are various tools for scanning the docker image, such as Snyk and Trivy. Docker registry also provides scan options that help to check the image and find vulnerabilities.

It’s important to avoid unnecessary content in software packages or docker images. Choose the appropriate image when delivering the docker image. Secure the docker registry with a private docker registry as well.

Conclusion

Risk-based security is an important aspect of a CI/CD pipeline as it reduces the chance of data breaches and security vulnerabilities. Avoiding plain text credentials, implementing credential rotation, validating inputs, having source code analysis, and omitting unnecessary files in packages are some of the measures to take.