1. The Codecov breach, that led to exfiltration of secrets stored within environment variables in thousands of build pipelines across numerous enterprises. [Codecov breach impacted ‘hundreds’ of customer networks: report | ZDNET] 
2. The Dependency Confusion flaw, which affected dozens of giant enterprises, and abuses flaws in the way external dependencies are fetched to run malicious code on developer workstations and build environments.

 

To counter attacks like these and helping organizations build robust defence systems, the OWASP foundation has released a report that mentions the top 10 security threats to CI/CD pipelines. Let’s explore these 10 threats and review how we at Caravel Labs mitigate these issues to build robust and reliable software for our clients. We will divide this post into a two-article series, where we explore five issues in each. 

 

        1. Insufficient Flow Control Mechanisms 

Insufficient flow control mechanisms refer to the ability of an attacker that has obtained permissions to a system within the CI/CD process (SCM, CI, Artifact repository, etc.) to push malicious code or artifacts down the pipeline, due to a lack of mechanisms that enforce additional approval or review. 

How we address this? 

1. We have exclusive branch policies for every pull request which comes from a developer. A developer cannot approve his/her own pull request. 
2. A guest user will not be able to approve a pull request unless he/she is given permission explicitly by the project administrator.

 

        2. Inadequate Identity and Access Management

Inadequate Identity and Access Management risks stem from the difficulties in managing the number of identities spread across the different systems in the engineering ecosystem, from source control to deployment. Multiple systems are interconnected in software delivery procedures with the goal of transferring code and artifacts from development to production. Passwords and access tokens are just a few of the access and integration options offered by each system. Some of the major concerns in Inadequate Identity and Access Management are: 

• Over-permissive permissions for both users and applications 
• Deactivation of the accounts of previous identities (employees/applications) 

• External Collaborators whose domains are not owned by the organization and hence the security policies do not bind to them. 

How we address this? 

1. We always follow the principle of least privilege access . [Link]
2. Whenever a project ends, auditing is done to ensure all the external and stale accounts are deactivated and deleted.
3. Access tokens have a small renewal period because the longer we use the same token the chances of an attack increase. 
4. Employees are prevented from using their personal accounts for development. 
5. Dedicated accounts are created for each specific context and grant the exact set of permissions required for the context in question. 

 

        3. Dependency Chain Abuse 

Dependency chain abuse risks refer to an attacker’s ability to abuse flaws relating to how engineering workstations and build environments fetch code dependencies. Dependency chain abuse results in a malicious package inadvertently being fetched and executed locally when pulled. Most of the time, packages are downloaded using a specific client for each programming language, typically from language-specific repositories (for instance, Node.js has npm and the npm registry, Python has pip and uses PyPI). 

The typical attacks in these scenarios are as follows: 

• Dependency Confusion: Use of internal package names as the names of malicious packages in public repositories to persuade users to download the malicious package rather than the private one.

• Dependency Hijacking:  Gaining access to a package maintainer's account on a public repository, uploading a new, malicious version of a commonly used package, and using that to compromise unknowing clients who pull the most recent version of the package. 

• Brandjacking: Publication of vulnerable packages that are consistent with the naming style or other features of a particular brand's package to trick developers into downloading them. 

 

Dependency chain abuse can have a significant impact as the attacker can send malicious code directly to production. Some of the well-known incidents:  

• Amazon, Zillow, Lyft, and Slack NodeJS apps targeted by threat actors using the Dependency Confusion vulnerability. [Link] 
• The ua-parser-js NPM library, with 9 million downloads a week, was hijacked to launch crypto miners and steal credentials. [Link] 

• Famous Deep Learning Framework Pytorch Organization faced a dependency confusion in late 2022. [Link] 

How we address this: 

1. We enable checksum verification and signature verification for pulled packages.
2. We always try to use packages which have a decent download number and used by many.
3. We regularly do “npm audit” to track any high and critical severity packages and treat them as issues which need to fix as early as possible. 

 

        4. Poisoned Pipeline Execution (PPE) 

Poisoned Pipeline Execution (PPE) risks refer to the ability of an attacker with access to source control systems - and without access to the build environment, to manipulate the build process by injecting malicious code/commands into the build pipeline configuration, essentially ‘poisoning’ the pipeline and running malicious code as part of the build process. A successful PPE attack has very significant impacts such as: 

• Access to any secret available to the CI job, such as secrets injected as environment variables can give direct access to cloud provider and source control system. 
• Attackers can execute malicious code on other project pipelines if they get access to the pipeline from one exposed system. 

Some well-known incidents in the news: 

• GitHub Actions abused to mine cryptocurrency by pull requests that contained malicious code. [Link] 
• Build Pipeline Security event which occurred in an AWS website. [Link] 

How we address this: 

1. All CI/CD pipelines are configured through Infrastructure as Code (Terraform) and all the Terraform YAML files are passed through code scanners like detect-secrets to ensure there are so secrets in the files.
2. All Infrastructure as Code files are examined using static analysis tools like Checkov to find potential security vulnerabilities.
3. All pipelines are given permission based on the concept of least privilege. 

 

        5. Insufficient Pipeline Based Access Control (PBAC) 

The resources and systems available to pipeline execution nodes both inside and outside the execution environment are extensive. Adversaries exploit Insufficient PBAC (Pipeline-Based Access Controls) while executing malicious code within a pipeline to take advantage of the authorization provided to the pipeline for lateral movement inside or outside the CI/CD system. 

Pipelines are the beating heart of the CI/CD pipelines. PBAC is a term which refers to the context in which each pipeline and each step within that pipeline is running. PBAC includes controls relating to numerous elements having to do with the pipeline execution environment:  

•  Access within the pipeline execution environment: code, secrets, environment variables, and other pipelines.  
• Permissions to the underlying host and other pipeline nodes. 

Some of the well-known incidents are: