How Coalfire Drives FedRAMP Compliance Without Sacrificing Cloud Deployment Speed

By Scott Moskal, Associate – Coalfire
By Christian Stano and David Abeyta, Senior Managers – Coalfire
By Saptarshi Banerjee, Solutions Architect – AWS

Coalfire

Recognizing the evolving landscape of technology and reliance on cloud solutions, the Federal Risk and Authorization Management Program (FedRAMP) is a compliance framework built on NIST 800-53 that serves as a standardized framework to assess, authorize, and continuously monitor the security of cloud products and services.

Complying with FedRAMP standards poses several challenges for cloud service providers. These affect DevOps teams’ tools and processes defining continuous integration and continuous delivery (CI/CD), as DevOps teams typically observe impacts to the metrics associated with DevOps success, such as deployment frequency and lead time for changes.

FedRAMP-related challenges often compound with existing issues, including process overhead, inefficient CI/CD pipelines, and non-unified tooling and processes. This can hinder swift deployment of code and automation out for testing and general use, resulting in suboptimal developer experiences and release cycles.

DevOps teams operating and maintaining a FedRAMP environment also experience higher change failure rates due to differing technical requirements between standard and AWS GovCloud (US) partitions. Common examples of this are differences in Amazon Resource Name (ARN) differences, service endpoint changes, and unavailable or non-compliant AWS services in the AWS GovCloud (US) partition.

DevOps teams often try to fix these problems by adding developers, forking and maintaining new codebases to support FedRAMP, or duplicating CI/CD pipelines which add further costs, overhead, complexity, and error rates.

In this post, Coalfire sheds light on three primary challenges DevOps teams should prioritize solving to improve cloud deployment speeds in their FedRAMP compliance journeys. We will also introduce a secure framework and architecture to address these challenges, optimize deployment velocity, and lower operational expenses.

Coalfire is an AWS Specialization Partner and AWS Marketplace Seller that’s a cybersecurity advisor providing insights to reduce risk, simplify compliance, and help keep businesses secure. Coalfire delivers expert advice, assessments, technical testing, and cyber engineering services.

Challenge #1: AWS GovCloud (US) Logical and Physical Isolation

With AWS GovCloud (US), customers can run unclassified workloads with the security and compliance assurances of the AWS Shared Responsibility Model, but unique challenges arise in deploying to a new, isolated partition.

Many DevOps teams assume they can deploy, maintain, and respond to their AWS GovCloud (US) FedRAMP environment like any other U.S. region. This assumption is disproved due to the service’s logical and physical isolation. Deploying and running workloads and infrastructure as code (IaC) as-is typically results in the eventual need for code rework and an inevitable question about forking codebases.

One of the challenges faced by DevOps teams is the differences in ARNs between the AWS standard cloud and AWS GovCloud (US) partitions. While teams commonly address multi-region deployments using dynamic or region-aware code blocks, few also handle multi-partition deployments with this pattern.

For example, as shown in the code snippet below, the ARNs for AWS services and deployed resources must be manually updated. Each update becomes dependent on the specific partition and region where the resource is deployed.

arn:aws:service:region:account-id:resource-type:resource-id

arn:aws-us-gov:service:region:account-id:resource-type:resource-id

This change may seem small, but multiplying that by every defined ARN could result in hundreds or thousands of code changes.

Another issue is the physical and logical isolation of virtual private cloud (VPC) networks and AWS service endpoints deployed into AWS GovCloud (US). This isolation supports strict compliance boundaries protecting U.S. government workloads.

For teams deploying into AWS GovCloud (US), it means existing networking components must be rearchitected. This also requires revisiting and reworking pre-built hub-and-spoke automations.

One more issue involves AWS Identity and Access Management (IAM), as existing IAM roles and permissions cannot be directly transferred across the partition. Resulting cross-account or hub-and-spoke models from a centralized deployment AWS account needs special considerations for CI/CD pipelines and telemetry architectures.

A common workaround involves creating programmatic IAM users with access keys and secret keys to replace role access. This approach leads to other problems, however, such as overprivileged and overused user secrets passed between services and lowering overall security posture.

These issues result in teams encountering delays from re-architecture, accumulating technical debt with workarounds, and experiencing slower releases to FedRAMP.

Challenge #2: Access Controls

Engineering organizations face another challenge with off-shore development and security clearances preventing teams from accessing, viewing, and troubleshooting services and workloads deployed in the FedRAMP environment. This strict FedRAMP boundary is critical to overall security and compliance, but translates to all users falling under FedRAMP compliant access controls, ultimately reducing the pool of engineers with environment access.

Some organizations resolve this problem by fully mirroring their commercial development teams with a net-new FedRAMP team, but this can result in redundancy and unnecessary cost.

A different approach involves “shifting compliance left.” A smaller team working within the FedRAMP environment fulfills specific deployment and change management responsibilities, while other development and engineering work is done outside the FedRAMP boundary. Compliance controls are baked into the continuous integration process.

Teams can close the FedRAMP development and deployment cadence and velocity gap with commercial environments for feature releases, bug fixes, and incident response.

Challenge #3: Change Management

The final challenge directly impacting developer velocity is FedRAMP’s change management process. The change management process in FedRAMP is defined at the process level by NIST S.P. 800-128 and culminates with the FedRAMP Significant Change Policy.

At a high level, these frameworks define that all engineering changes within the environment must be documented and submitted to a change control board (CCB), which reviews the proposed changes and their impacts to the confidentiality, integrity, and availability of the environment.

Changes cannot be made before CCB review, and wait times for approval eventually create lagging deployment strands in FedRAMP environments. Cloud deployment speed takes a direct hit as a result.

Coalfire’s proposed secure, compliant framework outlined below streamlines the amount of manual work required for the CCB by automating artifact and evidence collection—ultimately speeding up the change management and cloud deployment process.

Building a Velocity-Optimized FedRAMP Framework

Figure 1 shows an architecture that preserves existing development practices and velocity and minimizes operational expense.

Figure 1 – Architecture for a FedRAMP-compliant framework.

The proposed workflow focuses on shifting compliance left into the software development lifecycle (SDLC) process and branching artifacts into FedRAMP-specific deployment pipelines at the conclusion of the continuous integration process. The workflow can be described in the five main steps below:

Engineering teams check in a code branch in their remote code repository to deploy new features, fix bugs, provision new infrastructure, or respond to incidents.

Figure 2 – Focused view on Step 1.

The checked-in code branch integrates with the continuous integration tooling where:
- Automated testing, security scans of the code, and container images are performed.
- Artifact generation is automatically prepared for FedRAMP’s requirements around vulnerabilities (plan of actions and milestones, or POAM), software bill of materials (SBOM), and security impact analysis (SIA). These artifacts are then written to the FedRAMP artifact Amazon Simple Storage Service (Amazon S3) bucket.
- Code and images passing manual review and automated test gates are merged into the main branch and the build artifacts are promoted to two artifact repository zones:
  - Commercial zone for Amazon Elastic Container Registry (Amazon ECR) (container images) and the artifact Amazon S3 bucket (for binaries).
  - FedRAMP zone for Amazon ECR (container images) and the FedRAMP binaries S3 bucket.
  - A merge of a pull request into the main Git repository branch also triggers a code sync between the commercial code repository and in-boundary code repositories zone, such as AWS CodeCommit. Additional details of this sync are covered in Step 4.

Figure 3 – Focused view on Step 2.

Once the code has been approved in the commercial pipeline and artifacts staged, the pipeline splits between continued commercial deployment and FedRAMP deployment:
- The commercial deployment tool(s) zone (AWS CodePipeline, or other self-hosted deployment tools shown in Figure 4 as examples) picks up the artifacts generated from the continuous integration process and deploys or updates resources in the production workloads according to a team’s normal deployment process.
- The in-boundary retrieval tool(s) zone performs a cross-partition pull into the boundary.
There are two ways to pull the FedRAMP artifacts, container images, and binaries into the in-boundary AWS GovCloud (US) storage areas:
- Option 1: IAM user service account
  - Create an IAM user service account in the AWS commercial account containing the FedRAMP artifact Amazon S3 buckets and Amazon ECR. This IAM user should have GetObject and BatchGetImage permissions to only the FedRAMP artifacts and ECR. The access key and secret access key tied to this IAM user is then used with either the artifact retrieval AWS Lambda (for serverless preferences) or injected as a protected secret into a GitLab Runner on Amazon Elastic Compute Cloud (Amazon EC2); this is one of the few FIPS-compliant runner options.
  - A script or function can be written to pull the commercial-side FedRAMP artifacts, binaries, and containers using this IAM user service account. Following a successful pull into the FedRAMP boundary, artifacts and binaries can be stored in a FedRAMP-ready S3 bucket. Containers can be stored in a FedRAMP-ready ECR repository. These FedRAMP-ready S3 buckets and ECR repositories act as a staging zone for the FedRAMP engineering team to review and promote prior to deployment.
  - Note that although this approach is simpler to set up initially, leveraging IAM user secrets across environments is generally not a recommended approach for security.
- Option 2 (Recommended): Amazon EKS AssumeRoleWithWebIdentity assumption
  - Create an OpenID Connect (OIDC) provider IAM role in the AWS commercial account containing the FedRAMP artifact S3 buckets and Amazon ECR. This IAM role should have GetObject and BatchGetImage permissions to only the FedRAMP artifacts and ECR. The FedRAMP in-boundary retrieval cluster provides the commercial account with the OIDC issuer URL from the cluster.
  - The commercial account configures a Kubernetes service account to assume an IAM role using the OIDC issuer URL from the FedRAMP in-boundary retrieval cluster. Then, a cluster administrator annotates the service account in the FedRAMP in-boundary retrieval cluster to use the role from the commercial account.
  - A script or function can be written to pull the commercial-side FedRAMP artifacts, binaries, and containers using this IAM user service account. Following a successful pull into the FedRAMP boundary, artifacts and binaries can be stored in a FedRAMP-ready S3 bucket. Containers can be stored in a FedRAMP-ready ECR repository. These FedRAMP-ready buckets and ECR repositories act as a staging zone for the FedRAMP engineering team to review and promote prior to deployment. A programmatic check using hashes, checksums, or equivalent technologies should be part of this script or function performing a pull into the FedRAMP boundary to verify integrity.
  - Note that this approach requires a deeper understanding of Amazon Elastic Kubernetes Service (Amazon EKS) and IAM roles for service accounts (IRSA) but provides a significant security benefit.

Figure 4 – Focused View on Steps 3 and 4.

With the code staged in-boundary in the code repositories zone and artifacts pulled into the FedRAMP boundary, the FedRAMP engineering team is able to review the changes alongside the artifacts generated in Step 2c.
- These changes are submitted to the CCB for approval, and artifacts are promoted to their respective FedRAMP-Prod Amazon S3 buckets and ECR repositories for deployment.
- Code changes synced from the commercial code repository are integrated with in-boundary live code that contains sensitive metadata (such as IPs, hostnames, and identifying system information).

Figure 5 – Focused view on Step 5.

With all of this consolidated and approved, the FedRAMP in-boundary deployment tool(s) go into staging and production environments with the latest features from commercial.

Conclusion

In this post, Coalfire demonstrated how customers can optimize DevOps practices while adhering to FedRAMP compliance, supporting faster and more frequent staging in the FedRAMP boundary with less team overhead.

By providing an automated, repeatable deployment pipeline with review stage gates, the failure rate of changes can be reduced, and rollbacks are simpler. The time to deployment for a bug fix spanning both commercial and FedRAMP environments is also reduced with less redundant code and more streamlined pipelines.

With the proper frameworks and guardrails in place, organizations can maintain positive DevOps metrics in their FedRAMP environment, including deployment frequency and lead time for changes, lower change failure rates, and an overall increase in cloud deployment speed.

If you want to strengthen your security posture and secure your digital transformations, contact the Coalfire support team. If you’re curious about beginning the FedRAMP journey without breaking the bank, RAMP/pak+ offers a free, open-source starter pack paired with expert advisory support to help you navigate the initial stages of the process.

The framework discussed in this post is a path to both AWS GovCloud (US) and AWS commercial region(s). Customers need to be aware of their compliance requirements that require data to reside solely in AWS GovCloud (US) when using this framework.

.

.

Coalfire – AWS Partner Spotlight

Coalfire is an AWS Specialization Partner and cybersecurity advisor providing insights to reduce risk, simplify compliance, and help keep businesses secure. Coalfire delivers expert advice, assessments, technical testing, and cyber engineering services.

Contact Coalfire | Partner Overview | AWS Marketplace