Skip to main content

Automation for Developers

In a Nutshell
What will I learn?In this tutorial we will generate CI workflow for sample python app.
Difficulty

Developing with worX​

What you'll need​

Software & Services​

Pre-commit​

What is | Using Pre-commit for projects​

Pre-commit is an open-source project that helps managing Git hooks, enabling automation running various code quality checks before changes are committed, ensuring the codebase remains consistent and error-free. It is a powerful Git hooks management tool that can be easily integrated into various projects, including Python, Terraform, JavaScript, and more, to run linters, formatters, and other code quality checks.

Installing pre-commit​

Note: Pre-commit is already installed on AWSH and BLOX.

Before you can start using pre-commit, you need to install it. You can install pre-commit using pip or Homebrew:

# Using pip
pip install pre-commit

# Using Homebrew (macOS)
brew install pre-commit
Setting up pre-commit for your project​

  1. Create a .pre-commit-config.yml file in the root directory of your project. This file will contain the configuration for pre-commit, specifying which hooks to run and their settings.

    Here's an example .pre-commit-config.yml file:

    repos:
    - repo: https://github.com/pre-commit/pre-commit-hooks
      rev: v4.1.0
      hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yml
      - id: check-json

  2. Install the Git hooks by running the following command in your project directory:

    pre-commit install

    This command will set up the pre-commit hooks to run automatically before each commit.

Running pre-commit​

Once pre-commit is set up, it will run automatically before each commit. If any issues are found, the commit will be aborted, and you'll need to fix the problems before retrying the commit.

You can also run pre-commit manually by executing the following command in your project directory:

pre-commit run --all-files

This command will run all the configured hooks against all the files in your project.

Updating pre-commit hooks​

To update the pre-commit hooks to their latest versions, run the following command:

pre-commit autoupdate

This command will update the hooks in your .pre-commit-config.yml file to the latest available versions.

Further reading​

For more information about pre-commit and its usage, check out the official documentation:

GitLab repository​

Before we begin, ensure that you have the following:

  1. A GitLab account and repository to set up your CI/CD pipeline.
  2. A GitLab repository with sample Python code, a Dockerfile, and pytest files. You can fork the Autoworx CI/CD examples repository.
  3. In your new repository, create a .gitlab-ci.yml file, which we will be configuring with Autoworx jobs in the next steps of this tutorial.

Tutorial​

Simple Application for autoworX Integration​

In this tutorial, we demonstrate a basic Flask API written in Python, which can be used as a starting point for integrating with autoworX CI/CD workflows.

The API features two endpoints:

  • The root endpoint ('/'), which returns a plain text greeting message: "Hello, World!"
  • The /api/v1/users endpoint, which returns a JSON object containing a list of user data.

To test and deploy this API using autoworX, you can build your CI/CD pipeline by importing relevant jobs from autoworX's modular library. These jobs may include building a Docker container, running tests, and deploying to a staging or production environment.

We have added several CI jobs to the Autoworx modular library, such as security scans for both the Python code and Terraform infrastructure, and container security scans using Aqua Trivy.

Moreover, in this tutorial, we will also provide a sample Terraform code that can be deployed to expose our sample-api Python code on AWS using an ECS service with an ALB on top of it. This will demonstrate how to automate the deployment process of a containerized application to a production environment using autoworX.

This sample API serves as a foundation for developers who wish to create more complex applications while leveraging the power and flexibility of autoworX's CI/CD solutions.

In this Autoworx tutorial, we will build a complete CI workflow for the Python-based application using a variety of "Lego" blocks that represent individual jobs. We will use the sample Python script named sample_api.py, which implements a simple Flask API with two endpoints, as mentioned above.

Application code​

sample_api.py
from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

@app.route('/api/v1/users')
def get_users():
    users = [
        {
            'id': 1,
            'name': 'John Doe',
            'email': '[email protected]'
        },
        {
            'id': 2,
            'name': 'Jane Doe',
            'email': '[email protected]'
        }
    ]
    return jsonify(users)

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=8080)

Unit tests​

To ensure the proper functioning of the API, we will use a set of unit tests written in the test_sample_api.py file. These tests cover the various operations provided by the API and help verify that the code is working correctly.

test_sample_api.py
import pytest
from sample_api import app

@pytest.fixture
def client():
    app.config["TESTING"] = True
    with app.test_client() as client:
        yield client

def test_hello_world(client):
    response = client.get("/")
    assert response.status_code == 200
    assert response.data == b"Hello, World!"

def test_get_users(client):
    response = client.get("/api/v1/users")
    assert response.status_code == 200
    assert response.json == [
        {
            "id": 1,
            "name": "John Doe",
            "email": "[email protected]",
        },
        {
            "id": 2,
            "name": "Jane Doe",
            "email": "[email protected]",
        },
    ]

The CI pipeline will provide linting, testing, building, and scanning Docker images for security vulnerabilities in the Python project with a Dockerfile. Each step of the pipeline follows best practices to ensure high-quality code, security, and proper functioning.

Main steps of the CI pipeline​

  1. Linting: Check the Python code and Dockerfile for syntax and style issues. This step helps maintain consistent and readable code, which is crucial for collaborative projects. We use flake8 for Python linting and hadolint for Dockerfile linting, as they are widely used and well-maintained tools with a comprehensive set of rules and plugins.

    linting linting

  2. Security:

    • Bandit scans Python code for potential vulnerabilities, helping protect applications from security risks and ensuring compliance with standards. securitysecurity
    • TFSec and Terrascan scan Terraform code for security vulnerabilities and compliance issues, ensuring infrastructure as code (IaC) adheres to best practices. terraform_tfsecterraform_tfsecterraform_terrascanterraform_terrascan
    • Checkov analyzes IaC files across multiple languages for security vulnerabilities and compliance, providing detailed reports. terraform_checkovterraform_checkov
    • Gitleaks scans Git repositories for leaked secrets, protecting sensitive data from accidental exposure and ensuring compliance with data protection regulations. terraform_gitleaksterraform_gitleaks

  3. Testing: Execute unit tests on the Python code to ensure it behaves as expected. This step verifies that the code is working correctly and helps catch regressions early in the development process. We use pytest, a popular testing framework, which provides a rich set of features and plugins for effective testing. Pytest is also integrated with GitLab CI to report test results and code coverage, helping you monitor your code quality over time.

    tests tests

  4. Building: Create a Docker image from the provided Dockerfile. This step packages your application and its dependencies into a portable container, which can be deployed and run consistently across different environments. We use Docker and GitLab CI's native Docker support for building and managing images.

  5. Container Security: Scan the built Docker image for security vulnerabilities. This step ensures that the application's runtime environment, including the base OS and third-party packages, are free from known security issues. We use Aqua Trivy, a comprehensive vulnerability scanner for containers and other artifacts, to scan the Docker image.

    container_security container_security

  6. Operations: Calculate the infrastructure costs using Infracost. This step provides an estimate of the cloud infrastructure costs before provisioning it, helping teams make informed decisions about their cloud resources and optimize costs. Infracost is a tool that works with Infrastructure as Code (IaC) files, such as Terraform, and integrates with GitLab CI to report the cost breakdown.

    operations

Sample GitLab CI/CD configuration​

.gitlab-ci.yml
# This GitLab CI/CD configuration demonstrates a complete CI workflow for a Python-based application
# with a Dockerfile and Infrastructure as Code (IaC) using Terraform. The pipeline provides
# linting, testing, building, and scanning Docker images for security vulnerabilities.
#
# The pipeline consists of the following stages:
# - lint: Lint the Python code, Dockerfile, and Terraform code
# - security: Run security checks on the Python code and IaC
# - test: Run unit tests on the Python code
# - build: Build the Docker image
# - container-security: Scan the Docker image for security vulnerabilities
# - operations: Calculate the infrastructure costs using Infracost
#
# The pipeline includes several jobs from external templates, such as flake8,
# bandit, pytest, and Docker-related jobs, as well as Terraform and IaC jobs.
#
# You need to provide the following variables:
# - DOCKERFILE_PATH: Path to the Dockerfile
# - PYTHON_CODE_PATH: Path to the Python code files (*.py)
# - PYTHON_REQUIREMENTS_PATH: Path to the requirements.txt file
# - PYTHON_CODE_DIR: Path to the directory containing the Python code
# - PYTHON_TEST_LIB: Name of the test module (e.g., "test_sample_api" for "test_sample_api.py")
# - PYTHON_LIB: Name of the module you want to cover with tests (e.g., "sample_api" for "sample_api.py")
# - INFRACOST_API_KEY: Infracost API key (mandatory). Best practice is to set this variable as a protected
#                      variable in your GitLab CI/CD repository settings to avoid exposing #sensitive information.
#
# The pipeline produces various artifacts, such as code quality reports, test
# results, and security scan results. These artifacts expire after 1 week.
#
# Sample Python code, Dockerfile, pytest files, and Terraform files used in this tutorial can
# be found in the public repository at:
# https://gitlab.com/hestio-community/autoworx/autoworx-cicd-examples

variables:
  DOCKERFILE_PATH: "Dockerfile"

stages:
  - lint
  - security
  - test
  - build
  - container-security

include:
  - project: 'hest-io/hestio-product/hestio-worx/autoworx/ci/templates'
    file:
      - 'Jobs/Python/python-flake8.gitlab-ci.yml'
      - 'Jobs/Python/python-bandit.gitlab-ci.yml'
      - 'Jobs/Python/python-pytest.gitlab-ci.yml'
      - 'Jobs/Docker/dockerfile-lint.gitlab-ci.yml'
      - 'Jobs/Docker/docker-build.gitlab-ci.yml'
      - 'Jobs/Docker/docker-container-security.gitlab-ci.yml'
      - 'Jobs/Terraform/terraform-terraascan.gitlab-ci.yml'
      - 'Jobs/Terraform/terraform-tfsec.gitlab-ci.yml'
      - 'Jobs/IaC/iac-checkov.gitlab-ci.yml'
      - 'Jobs/IaC/iac-secret-detection.gitlab-ci.yml'
      - 'Jobs/Terraform/terraform-infracost.gitlab-ci.yml'

flake8-lint:
  stage: lint
  variables:
    PYTHON_CODE_PATH: "python/*.py"

dockerfile_lint:
  stage: lint
  variables:
    DOCKERFILE_PATH: "python/Dockerfile"

bandit-security:
  stage: security
  variables:
    PYTHON_CODE_PATH: "python/*.py"

pytest:
  stage: test
  variables:
    PYTHON_REQUIREMENTS_PATH: "python/requirements.txt"
    PYTHON_CODE_DIR: "python/"
    PYTHON_TEST_LIB: "test_sample_api"
    PYTHON_LIB: "sample_api"

build_docker_image:
  stage: build
  variables:
    DOCKERFILE_PATH: "python/Dockerfile"

container-security:
  stage: container-security

tfsec_scan:
  stage: security
  variables:
    TERRAFORM_CODE_PATH: "terraform/"
  allow_failure: true

terrascan:
  stage: security
  variables:
    TERRAFORM_CODE_PATH: "terraform/"
  allow_failure: true

iac_checkov:
  stage: security
  variables:
    IAC_CODE_PATH: "terraform/"
  allow_failure: true

gitleaks_scan:
  stage: security
  allow_failure: true

infracost:
  stage: operations

ci jobs tests

Deploying the Terraform Infrastructure​

Now that we have created and tested our Flask API, we can deploy it to a development environment using Terraform. In this tutorial, we will use a sample Terraform code that creates an Amazon Elastic Container Service (ECS) cluster and deploys our sample API to it using a Docker container.

To simplify the deployment process, we recommend using the BLOX container, which provides a simple and efficient way to deploy and manage cloud resources on AWS.

Prerequisites​

Before we begin, make sure that you have the following prerequisites:

Deploying the Terraform Code​

To deploy the sample Terraform code, follow these steps:

  1. Clone the autoworx-examples repository and navigate to the terraform directory:

    BLOX>
    git clone https://gitlab.com/hestio-community/autoworx/autoworx-cicd-examples.git
    cd autoworx-cicd-examples/terraform

  2. Initialize the Terraform environment and download the required providers:

    BLOX>
    terraform init

  3. Review the resources that will be created by running a Terraform plan:

    BLOX>
    terraform plan
    data.aws_availability_zones.available: Reading...
    data.aws_region.current: Reading...
    data.aws_region.current: Read complete after 0s [id=eu-west-1]
    data.aws_availability_zones.available: Read complete after 0s [id=eu-west-1]

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      <resources-hidden-due-security-purposes>

    Plan: 16 to add, 0 to change, 0 to destroy.

    Changes to Outputs:
      + alb_dns_name = (known after apply)

  1. If everything looks good, apply the Terraform code to create the resources:

    BLOX>
    terraform apply
    ...

    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

    Outputs:

    alb_dns_name = "sample-alb-1781135928.eu-west-1.elb.amazonaws.com"

    This will create an ECS cluster, a task definition, a service, and an application load balancer (ALB) for our Flask API.

Verify​

Verify that the sample API is running by navigating to the ALB URL in a web browser or using a tool like curl:

BLOX>
curl sample-alb-1781135928.eu-west-1.elb.amazonaws.com/api/v1/users
[{"email":"[email protected]","id":1,"name":"John Doe"},{"email":"[email protected]","id":2,"name":"Jane Doe"}]

Congratulations! You have successfully deployed your Flask API to a development environment using Terraform and BLOX.

TL;DR​

With the help of autoworX snippets for best practices, we efficiently built, tested, examined security issues, and deployed our sample_api application to an AWS development environment. This streamlined process enabled us to concentrate on developing a robust Flask API while taking advantage of autoworX's powerful and flexible CI/CD solutions.


Want to find out more?

At Hestio, we have taken our experience with designing and building on cloud to codify these patterns and made them available as a low-code pattern library for AWS. Why spend time and effort on reinventing the wheel when it's already a solved problem? Would you start developing office productivity software in a world where Microsoft Office already exists?

If you'd like to find out about worX, our low-code patterns library for AWS you can read more here or get in touch today to schedule a demo.

If you'd like to find out more about the products and services Hestio has to offer, select one of the options below.

Logo
Terms of Service - Privacy Policy - Cookie Policy

Copyright © 2023 Hestio Limited.
EU VAT Registered: IE3619670DH, Registered in Ireland No 651646