Automation for Developers
|What will I learn?||In this tutorial we will generate CI workflow for sample python app.|
Developing with worX
What you'll need
Software & Services
- Sample code for this tutorial. You can obtain the code by forking the Autoworx CI/CD Examples repository.
What is | Using Pre-commit for projects
Note: Pre-commit is already installed on AWSH and BLOX.
Before you can start using pre-commit, you need to install it. You can install pre-commit using pip or Homebrew:
# Using pip
pip install pre-commit
# Using Homebrew (macOS)
brew install pre-commit
Setting up pre-commit for your project
.pre-commit-config.ymlfile in the root directory of your project. This file will contain the configuration for pre-commit, specifying which hooks to run and their settings.
Here's an example
- repo: https://github.com/pre-commit/pre-commit-hooks
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yml
- id: check-json
Install the Git hooks by running the following command in your project directory:
This command will set up the pre-commit hooks to run automatically before each commit.
Once pre-commit is set up, it will run automatically before each commit. If any issues are found, the commit will be aborted, and you'll need to fix the problems before retrying the commit.
You can also run pre-commit manually by executing the following command in your project directory:
pre-commit run --all-files
This command will run all the configured hooks against all the files in your project.
Updating pre-commit hooks
To update the pre-commit hooks to their latest versions, run the following command:
This command will update the hooks in your .pre-commit-config.yml file to the latest available versions.
For more information about pre-commit and its usage, check out the official documentation:
Before we begin, ensure that you have the following:
- A GitLab account and repository to set up your CI/CD pipeline.
- A GitLab repository with sample Python code, a
pytestfiles. You can fork the Autoworx CI/CD examples repository.
- In your new repository, create a
.gitlab-ci.ymlfile, which we will be configuring with Autoworx jobs in the next steps of this tutorial.
Simple Application for autoworX Integration
In this tutorial, we demonstrate a basic Flask API written in Python, which can be used as a starting point for integrating with autoworX CI/CD workflows.
The API features two endpoints:
- The root endpoint (
'/'), which returns a plain text greeting message: "Hello, World!"
/api/v1/usersendpoint, which returns a JSON object containing a list of user data.
To test and deploy this API using autoworX, you can build your CI/CD pipeline by importing relevant jobs from autoworX's modular library. These jobs may include building a Docker container, running tests, and deploying to a staging or production environment.
We have added several CI jobs to the Autoworx modular library, such as security scans for both the Python code and Terraform infrastructure, and container security scans using Aqua Trivy.
Moreover, in this tutorial, we will also provide a sample Terraform code that can be deployed to expose our sample-api Python code on AWS using an ECS service with an ALB on top of it. This will demonstrate how to automate the deployment process of a containerized application to a production environment using autoworX.
This sample API serves as a foundation for developers who wish to create more complex applications while leveraging the power and flexibility of autoworX's CI/CD solutions.
In this Autoworx tutorial, we will build a complete CI workflow for the Python-based application using a variety of "Lego" blocks that represent individual jobs. We will use the sample Python script named
sample_api.py, which implements a simple Flask API with two endpoints, as mentioned above.
from flask import Flask, jsonify
app = Flask(__name__)
return 'Hello, World!'
users = [
'name': 'John Doe',
'email': '[email protected]'
'name': 'Jane Doe',
'email': '[email protected]'
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=8080)
To ensure the proper functioning of the API, we will use a set of unit tests written in the
test_sample_api.py file. These tests cover the various operations provided by the API and help verify that the code is working correctly.
from sample_api import app
app.config["TESTING"] = True
with app.test_client() as client:
response = client.get("/")
assert response.status_code == 200
assert response.data == b"Hello, World!"
response = client.get("/api/v1/users")
assert response.status_code == 200
assert response.json == [
"name": "John Doe",
"email": "[email protected]",
"name": "Jane Doe",
"email": "[email protected]",
The CI pipeline will provide linting, testing, building, and scanning Docker images for security vulnerabilities in the Python project with a Dockerfile. Each step of the pipeline follows best practices to ensure high-quality code, security, and proper functioning.
Main steps of the CI pipeline
Linting: Check the Python code and Dockerfile for syntax and style issues. This step helps maintain consistent and readable code, which is crucial for collaborative projects. We use flake8 for Python linting and hadolint for Dockerfile linting, as they are widely used and well-maintained tools with a comprehensive set of rules and plugins.
- Bandit scans Python code for potential vulnerabilities, helping protect applications from security risks and ensuring compliance with standards.
- TFSec and Terrascan scan Terraform code for security vulnerabilities and compliance issues, ensuring infrastructure as code (IaC) adheres to best practices.
- Checkov analyzes IaC files across multiple languages for security vulnerabilities and compliance, providing detailed reports.
- Gitleaks scans Git repositories for leaked secrets, protecting sensitive data from accidental exposure and ensuring compliance with data protection regulations.
Testing: Execute unit tests on the Python code to ensure it behaves as expected. This step verifies that the code is working correctly and helps catch regressions early in the development process. We use pytest, a popular testing framework, which provides a rich set of features and plugins for effective testing. Pytest is also integrated with GitLab CI to report test results and code coverage, helping you monitor your code quality over time.
Building: Create a Docker image from the provided Dockerfile. This step packages your application and its dependencies into a portable container, which can be deployed and run consistently across different environments. We use Docker and GitLab CI's native Docker support for building and managing images.
Container Security: Scan the built Docker image for security vulnerabilities. This step ensures that the application's runtime environment, including the base OS and third-party packages, are free from known security issues. We use Aqua Trivy, a comprehensive vulnerability scanner for containers and other artifacts, to scan the Docker image.
Operations: Calculate the infrastructure costs using Infracost. This step provides an estimate of the cloud infrastructure costs before provisioning it, helping teams make informed decisions about their cloud resources and optimize costs. Infracost is a tool that works with Infrastructure as Code (IaC) files, such as Terraform, and integrates with GitLab CI to report the cost breakdown.
Sample GitLab CI/CD configuration
# This GitLab CI/CD configuration demonstrates a complete CI workflow for a Python-based application
# with a Dockerfile and Infrastructure as Code (IaC) using Terraform. The pipeline provides
# linting, testing, building, and scanning Docker images for security vulnerabilities.
# The pipeline consists of the following stages:
# - lint: Lint the Python code, Dockerfile, and Terraform code
# - security: Run security checks on the Python code and IaC
# - test: Run unit tests on the Python code
# - build: Build the Docker image
# - container-security: Scan the Docker image for security vulnerabilities
# - operations: Calculate the infrastructure costs using Infracost
# The pipeline includes several jobs from external templates, such as flake8,
# bandit, pytest, and Docker-related jobs, as well as Terraform and IaC jobs.
# You need to provide the following variables:
# - DOCKERFILE_PATH: Path to the Dockerfile
# - PYTHON_CODE_PATH: Path to the Python code files (*.py)
# - PYTHON_REQUIREMENTS_PATH: Path to the requirements.txt file
# - PYTHON_CODE_DIR: Path to the directory containing the Python code
# - PYTHON_TEST_LIB: Name of the test module (e.g., "test_sample_api" for "test_sample_api.py")
# - PYTHON_LIB: Name of the module you want to cover with tests (e.g., "sample_api" for "sample_api.py")
# - INFRACOST_API_KEY: Infracost API key (mandatory). Best practice is to set this variable as a protected
# variable in your GitLab CI/CD repository settings to avoid exposing #sensitive information.
# The pipeline produces various artifacts, such as code quality reports, test
# results, and security scan results. These artifacts expire after 1 week.
# Sample Python code, Dockerfile, pytest files, and Terraform files used in this tutorial can
# be found in the public repository at:
- project: 'hest-io/hestio-product/hestio-worx/autoworx/ci/templates'
Deploying the Terraform Infrastructure
Now that we have created and tested our Flask API, we can deploy it to a development environment using Terraform. In this tutorial, we will use a sample Terraform code that creates an Amazon Elastic Container Service (ECS) cluster and deploys our sample API to it using a Docker container.
To simplify the deployment process, we recommend using the BLOX container, which provides a simple and efficient way to deploy and manage cloud resources on AWS.
Before we begin, make sure that you have the following prerequisites:
- An AWS account with appropriate permissions to create ECS clusters and other resources
- An active BLOX session from Get Started with BLOX with some AWS credentials loaded
Deploying the Terraform Code
To deploy the sample Terraform code, follow these steps:
Clone the autoworx-examples repository and navigate to the
git clone https://gitlab.com/hestio-community/autoworx/autoworx-cicd-examples.git
Initialize the Terraform environment and download the required providers:BLOX>
Review the resources that will be created by running a Terraform plan:BLOX>
data.aws_region.current: Read complete after 0s [id=eu-west-1]
data.aws_availability_zones.available: Read complete after 0s [id=eu-west-1]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
Terraform will perform the following actions:
Plan: 16 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ alb_dns_name = (known after apply)
If everything looks good, apply the Terraform code to create the resources:BLOX>
Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
alb_dns_name = "sample-alb-1781135928.eu-west-1.elb.amazonaws.com"
This will create an ECS cluster, a task definition, a service, and an application load balancer (ALB) for our Flask API.
Verify that the sample API is running by navigating to the ALB URL in a web browser or using a tool like
Congratulations! You have successfully deployed your Flask API to a development environment using Terraform and BLOX.
With the help of autoworX snippets for best practices, we efficiently built, tested, examined security issues, and deployed our sample_api application to an AWS development environment. This streamlined process enabled us to concentrate on developing a robust Flask API while taking advantage of autoworX's powerful and flexible CI/CD solutions.
At Hestio, we have taken our experience with designing and building on cloud to codify these patterns and made them available as a low-code pattern library for AWS. Why spend time and effort on reinventing the wheel when it's already a solved problem? Would you start developing office productivity software in a world where Microsoft Office already exists?
If you'd like to find out more about the products and services Hestio has to offer, select one of the options below.