Sentinel SDLC

In late 2020 I identified a need for both a ‘shift-left’ approach and enhanced Policy-as-Code (“PaC”) in our cloud deployment pipelines. I then architected, developed, and implemented our PaC solution using Hashicorp Sentinel such that our auditability, usability, security, and developer experience were all greatly enhanced.

Tools/Products used:

Project Details

This project needed to satisfy the requirements of a highly regulated global bank with a large development team.

Training colleagues:

  • Created & delivered in-house training 15 engineers in 3 countries to do Sentinel policy development
  • Organized & facilitated vendor training to supplement in-house training
  • Participation in Hashicorp Financial Services User Group for ideas exchange and cross-company development

Templatized policy creation: Enable rapid & repeatable workflows across global teams

  • Uses cookiecutter
  • Pre-commit hooks validate naming and branching
  • Post-commit hooks create & switch to the feature branch, create the policy, test template files, insert policy record into sentinel.hcl, and create a slow-roll deployment

Git Pre-Commit Hooks: Catch often-missed things before running a pipeline, saving time and improving the developer experience

  • Using git’s custom git hooks feature
  • Runs sentinel fmt on all the policies prior to committing
  • Runs custom internal linting on sentinel.hcl prior to committing

CI Testing: Ensure policy sets are thoroughly validated and tested prior to deployment

  • Runs a check to ensure directory & file structure is as intended
  • Ensures at least one passing and one failing unit test are declared for each policy
  • Parses and lints the sentinel.hcl file to ensure policies will properly load and run
  • Runs sentinel test on each of the policies to ensure that all unit tests are passing

Continuous Delivery: Allow tailored policy deployment 24x7 and incremental changes

  • Checks for valid exceptions which are still in their authorized date range
  • Creates custom policy sets with exceptions for each Terraform Org/Workspace
  • Builds/uploads the artifact to the artifact repository for fast rollback and auditability
  • Deploys policy sets to all the deployment targets
  • Nightly job runs to re-evaluate validity of each exception and re-deploys as needed
    • No exception is accidentally left in place!
  • In Progress - Download & store Sentinel mocks from all workspaces for integration testing
  • In Progress - Integration testing via Sentinel Mocks to avoid running 1000’s of plans for validation
    • Offers representative coverage at a fraction of the cost & time

Continuous Monitoring: In Progress - Ensure policies are consistently applied effectively & correctly, detect outages, and provide data insights for data-driven decision making

  • Poll for policy set hits, individual policy hits, and the results of the evaluations
  • Track data for rollup metrics, trend data, and allowing data-driven policy promotions
  • Use insights for training, identifying areas to strengthen, and threat modeling
  • Date

    01 Apr, 2021
  • Categories

    Cloud, Security, Devops
  • Position/Employer

    VP, Cloud Native Security Engineering, Citigroup
  • Sentinel Product Page

  • Policy as Code