As a Python developer, you’ve probably struggled with the hassle of setting up and maintaining a Continuous Integration/Continuous Deployment (CI/CD) pipeline for your project. You might have manually created separate scripts for testing, building, and deploying, only to realize that these tasks need to be updated whenever your codebase changes. This can lead to frustration and wasted time.
You’ll build a custom CI/CD pipeline using GitHub Actions, automating the process of testing with Pytest and leveraging features of Python 8.2+, such as type hinting and improved error messages. By following this tutorial, you’ll also learn how to package your application using setuptools and twine, making it easier to deploy to production environments like Docker and AWS Elastic Beanstalk.
Prerequisites: Setting Up a Python Project and GitHub Repository
To build a custom CI/CD pipeline for your Python project, you’ll first need to set up a basic project structure and create a GitHub repository.
Create a New Python Project
Create a new directory for your project and navigate into it in your terminal. I’m using my-python-project as the example project name:
mkdir my-python-project
cd my-python-projectNext, create a virtual environment to isolate your dependencies. I’ll use python -m venv here, but you can also use conda or another package manager if preferred:
python -m venv venv
source venv/bin/activate # On Linux/Mac
venv\Scripts\activate # On WindowsOnce activated, install the required dependencies. For this tutorial, we’ll start with pytest, but you should also install any other packages needed for your project:
pip install pytestInitialize a New Git Repository
Now that your project is set up, initialize a new Git repository by running:
git add .
git commit -m "Initial commit"Create a new GitHub repository or navigate to an existing one. Follow the prompts to link it to your local repository.
This sets the stage for creating our custom CI/CD pipeline with GitHub Actions in the next section.
Creating a Custom CI/CD Pipeline with GitHub Actions
To create a custom CI/CD pipeline for our Python project, we’ll leverage GitHub Actions. First, let’s navigate to our repository on GitHub and click on Actions in the left-hand menu.
Next, we’ll create a new workflow by clicking on New workflow and selecting Python package as the template.
name: Build and deploy
on:
push:
branches:
- main
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4.1.0
with:
python-version: '3.10'
- name: Install dependencies
run: |
pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: |
pytest
- name: Build and deploy
run: |
# Your deployment script here (e.g., using `twine` or `setuptools`)This workflow checks out our code, sets up Python 3.10, installs dependencies, runs our tests with Pytest, and finally executes a build/deployment script.
We can customize this template to suit our project’s specific needs by modifying the steps section. This might involve adding or removing steps, updating dependency versions, or customizing the deployment process.
By following these steps, we’ve created a basic CI/CD pipeline that integrates with GitHub Actions and automates testing and deployment for our Python project.
Automating Testing with Pytest and Python 8.2+ Features
In this section, we’ll focus on automating testing for our Python project using Pytest. As a reminder, you should have already set up your GitHub repository with the necessary files.
To begin, let’s create a new file called tests/test_app.py in the root of our project directory:
// Not applicable (PHP) - we'll use PHP pseudocode to represent Python code
# tests/test_app.py
import pytest
from myapp import app # assuming myapp is your application package
def test_root_route(client):
response = client.get('/')
assert response.status_code == 200This is a basic example of how you can write unit tests using Pytest. Here, we’re testing the root route of our Flask application.
To run these tests, navigate to your project directory and execute:
python -m pytest tests/If all tests pass, you should see an output indicating so.
We’ll also leverage Python 8.2+ features, such as type hinting and the dataclasses module, to enhance our testing setup. To make use of these features, ensure your pyproject.toml file includes the necessary dependencies:
[tool.poetry.dependencies]
python = "^3.9"
pytest = "^6.2"
[tool.poetry.dev-dependencies]With this configuration in place, you can utilize Python’s built-in type hinting and dataclass features to write more efficient and readable tests.
By automating testing using Pytest and taking advantage of Python 8.2+ features, we can ensure our application is thoroughly tested before deployment. This provides a solid foundation for building reliable software.
Building and Packaging the Application with setuptools and twine
Now that our tests are passing, let’s focus on building a distributable package for our Python project.
First, install the required packages using pip:
pip install setuptools twineNext, we’ll modify our setup.py file to use setuptools. Here’s an updated version of the file:
import setuptools
with open("README.md", "r") as f:
long_description = f.read()
setuptools.setup(
name="my-python-project",
version="1.0.0",
author="Your Name",
author_email="your@email.com",
description="A brief description of my project",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/your-username/my-python-project",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
)This updated setup.py file includes metadata about our project, including the name, version, author, and description.
To build a source distribution, run:
python setup.py sdistThis will create a tarball (e.g., my-python-project-1.0.0.tar.gz) in a dist/ directory. To upload this package to PyPI, use twine. First, create a new user account on the PyPI website and install the twine package:
pip install twineThen, authenticate with your PyPI credentials using twine login, and upload the source distribution:
twine upload dist/*Your package should now be available on PyPI for installation by other users.
Deploying to Production Using Docker and AWS Elastic Beanstalk
Now that our application is packaged and built, it’s time to deploy it to production using Docker and AWS Elastic Beanstalk.
Step 1: Create a New Elastic Beanstalk Environment
First, navigate to the AWS Management Console and create a new Elastic Beanstalk environment. In this example, I’ll use my-python-app as the environment name.
aws elasticbeanstalk create-environment --environment-name my-python-app --application-name python-app --region us-west-2Step 2: Create a Dockerfile
In your project directory, create a new file named Dockerfile. This will define our application’s container image:
FROM public.ecr.aws/ aws-sam-cli /runtime-python:3.9
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "main.py"]Step 3: Build and Push the Docker Image
Build your Docker image using docker build and push it to Amazon ECR:
docker build -t my-python-app .
docker tag my-python-app:latest <account_id>.dkr.ecr.us-west-2.amazonaws.com/my-python-app:latest
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <account_id>.dkr.ecr.us-west-2.amazonaws.com
docker push <account_id>.dkr.ecr.us-west-2.amazonaws.com/my-python-app:latestStep 4: Configure Elastic Beanstalk
Finally, configure your Elastic Beanstalk environment to use the Docker image:
aws elasticbeanstalk update-environment --environment-name my-python-app --option-settings Namespace=EnvironmentVariables,OptionName=DockerImageName,Value=<account_id>.dkr.ecr.us-west-2.amazonaws.com/my-python-app:latestWith these steps complete, your Python application should now be deployed to production using Docker and AWS Elastic Beanstalk.
Integrating Environment Variables and Configuration Management
When working on a Python project with multiple environments (e.g., development, testing, production), it’s essential to manage environment variables securely and efficiently.
To start, let’s create a .env file in the root of our project using the python-dotenv library:
# .env.example
DB_HOST=localhost
DB_PORT=5432
DB_USER=myuser
DB_PASSWORD=mypasswordIn your docker-compose.yml, you can reference these environment variables with ${VARIABLE_NAME} syntax:
version: '3'
services:
web:
build: .
environment:
- DB_HOST=${DB_HOST}
- DB_PORT=${DB_PORT}
- DB_USER=${DB_USER}
- DB_PASSWORD=${DB_PASSWORD}However, this is still not ideal for a CI/CD pipeline. We need to securely store and inject these values into our containers.
For production environments, consider using AWS Secrets Manager or Hashicorp’s Vault. For development environments, you can use the python-dotenv library with GitHub Actions’ env feature:
# .github/workflows/ci-cd.yml
name: CI/CD Pipeline
on:
push:
branches: [ main ]
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set environment variables
env:
DB_HOST: ${{ secrets.DB_HOST }}
DB_PORT: ${{ secrets.DB_PORT }}
DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}Note that secrets are stored securely in the repository settings.
Monitoring and Reporting Pipeline Failures with Slack Notifications
To take our pipeline to the next level, we’ll set up notifications for pipeline failures using Slack. This way, the development team will be immediately notified whenever a build or deployment fails.
First, create an actions.yml file in the .github/workflows directory of your repository:
name: Build and Deploy
on:
push:
branches:
- main
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Run tests
run: |
python -m unittest tests/
- name: Build and deploy
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
docker build -t my-image .
aws ecr create-repository --repository-name my-repo
docker tag my-image:latest 123456789012.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest
docker push 123456789012.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest
- name: Send notification to Slack
uses: slack-dev/slack-notifier@v1.0.3
with:
webHookUrl: ${{ secrets.SLACK_WEBHOOK_URL }}
message: "Pipeline failed!"In the code above, we added a step to send a notification to Slack using the slack-notifier action. This will trigger whenever a pipeline fails.
Make sure to store your Slack webhook URL as a secret in your repository settings and then reference it in the actions.yml file. Now you’ll receive immediate notifications when something goes wrong during deployment!
Frequently Asked Questions
What is the difference between venv and conda, and which one should I use?
Both venv and conda are package managers, but they serve different purposes. venv is a built-in Python module for creating isolated environments, while conda is a separate package manager that can be used to create environments as well. For this tutorial, we recommend using venv, but you can choose the one that best fits your needs.
Why do I need to install pytest separately if GitHub Actions already has it installed?
GitHub Actions uses a specific version of Python and tools, which may not match the versions used in your local environment. By installing pytest locally, you ensure that your tests run consistently across different environments.
What happens if I forget to update my requirements.txt file after changing dependencies?
If you don’t update your requirements.txt file, your CI/CD pipeline may fail to install the updated dependencies. Make sure to run pip freeze > requirements.txt whenever you add or remove dependencies in your project.
Can I use this custom CI/CD pipeline with other version control systems like GitLab or Bitbucket?
Yes, GitHub Actions can be integrated with other version control systems as well. However, the setup process may vary depending on the platform and repository settings.
Why do I need to use pip install --upgrade pip in my workflow if I’ve already installed it locally?
The --upgrade flag ensures that the latest version of pip is used, which may not be the case if you’ve installed an older version locally. This helps maintain consistency across different environments and prevents potential issues with package installation.