
Managing a monorepo with multiple Python packages presents unique challenges, particularly in ensuring efficient CI/CD. In this article, I will share how I created CI/CD workflows for our monorepo of Python packages using GitHub Actions.

We will delve into the workflows, including matrix complexity, and scripts for checking changed paths, alongside other custom scripts and composite actions that facilitate this process.

Our primary CI workflow (`on-pr-build-push.yaml`) triggers on pull requests, ensuring that all changes are validated before merging. This workflow comprises several jobs: `ChangedPaths`, `Lint`, `Typecheck`, `Test`, and `Build-Push`.

__Job: ChangedPaths__

This job determines which packages have been modified and outputs a matrix that drives the subsequent jobs. So for example, if a PR modifies `package1` and `package2`, the matrix will contain two rows: `package1` and `package2`.

```yaml
name: Build & Push Python Package

on:
  pull_request:

jobs:
  ChangedPaths:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.get_changed_paths.outputs.matrix }}
    timeout-minutes: 10
    permissions:
      id-token: write
      contents: read
    
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Run init
        uses: ./.github/actions/init # Custom composite action for initializing the environment
      
      - name: Get Changed Paths
        id: get_changed_paths
        shell: bash
        run: |
          python $GITHUB_WORKSPACE/.github/scripts/internal/get_changed_paths.py
```

The `get_changed_paths.py` python script identifies the changed paths and filters them to include only those within the packages directory.
This script is quite straightforward and uses the `git diff` command by using the built-in `subprocess` module in order to identify the changed paths.

```python
import subprocess
import os

output_file = os.getenv('GITHUB_OUTPUT')


def get_changed_paths():
    cmd = 'git diff --name-only $(git rev-parse HEAD) HEAD~1'
    result = subprocess.run(cmd, shell=True, capture_output=True)
    if result.returncode != 0:
        raise Exception("Error getting changed paths")

    changed_paths = result.stdout.decode("utf-8").strip().split("\n")
    print(f"Changed paths: {changed_paths}")
    return changed_paths


def get_filtered_paths(changed_paths):
    filtered_paths = set()
    for path in changed_paths:
        if path.startswith("packages/"):
            filtered_paths.add(path.split("/")[1])
    print(f"Filtered paths: {filtered_paths}")
    return list(filtered_paths)


if __name__ == "__main__":
    changed_paths = get_changed_paths()
    filtered_paths = get_filtered_paths(changed_paths)
    print(filtered_paths)
    with open(output_file, "a") as file:
        file.write(f"matrix={filtered_paths}")
```

Note the `capture_output=True` argument allows us to capture the command's output. We then decode and split the output to get a list of changed paths. The `get_filtered_paths` function filters these paths to include only those that start with `packages/` which is my path in my repos case.

Also note that GitHub Actions uses environment files to pass data between steps and jobs. By writing the output matrix to a file (GITHUB_OUTPUT), we can ensure that the list of changed paths is available as an output in the ChangedPaths job. This allows subsequent jobs to access and iterate over the paths efficiently.

__Job: Lint, Typecheck, and Test__

These jobs runs composite actions for linting (flake8), type checking (mypy), and tests (pytest) on the changed packages using a matrix strategy. They depend on the ChangedPaths job.

```yaml
  Lint:
    needs: ChangedPaths
    if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
    strategy:
      matrix: 
        path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Get path
        run: echo ${{ matrix.path }}
      
      - name: Run init
        uses: ./.github/actions/init
        with: 
          change_path: ${{ matrix.path }}

      - name: Run Lint action
        uses: ./.github/actions/lint
        env: 
          paths: ${{ matrix.path }}
```

__Job: Build-Push__

The final job in the CI pipeline builds and pushes the packages if all previous jobs succeed.

```yaml
  Build-Push:
    needs: [ChangedPaths, Lint, Typecheck, Test]
    if: ${{ needs.ChangedPaths.outputs.matrix != '[]' }}
    strategy:
      matrix:
        path: ${{ fromJson(needs.ChangedPaths.outputs.matrix) }}
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        id: checkout

      - name: Run init
        id: run-init
        uses: ./.github/actions/init
        with: 
          change_path: ${{ matrix.path }}

      - name: Build & Publish Python Packages
        uses: ./.github/actions/build-push-package
        env: 
          paths: ${{ matrix.path }}
```

This job calls a custom composite action (`build-push-package`) that builds and upload the package to GitHub cache, so the user can install it from there.

```yaml
name: Build & Release Package
description: "Build & Push Python Package"
author: "Or Kazaz"

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install Python Dependencies
      shell: bash
      run: |
        pip install --upgrade pip
        pip install poetry twine check-wheel-contents

    - name: Run Poetry Check for pyproject.toml
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry check

    - name: Build Package Distribution
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: poetry build

    - name: Check Distribution Descriptions
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        twine check dist/*

    - name: Check Wheel Contents
      working-directory: ./packages/${{ env.paths }}
      shell: bash
      run: |
        check-wheel-contents dist/*.whl

    - name: Cache Package Distribution
      uses: actions/upload-artifact@v3
      with:
        name: ${{ env.paths }}
        path: ./packages/${{ env.paths }}/dist/
```

The secondary workflow (`tag-and-release.yaml`) is triggered on a manually on demand by using `workflow_dispatch`. It tags the package, updates its version is pyproject.toml.toml, and publish the packages into our internal PyPI repository (on Jfrog Artifactory). It consists of multiple jobs to ensure a smooth release process.

__Job: Tag and Release__

This job tags the repository and creates a release on GitHub by zipping it and using an action to create a release to be available for download.

```yaml
jobs:
  tag-and-release:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    outputs:
      package_version: ${{ steps.dynamic_version.outputs.package_version }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.commit_hash }}

      - name: Run Dynamic-Versioning
        id: dynamic_version
        uses: ./.github/actions/dynamic-versioning
        with:
          package_name: ${{ github.event.inputs.package_name }}

      - name: Archive package directory
        working-directory: ./packages
        run: |
          zip -r ${{ github.event.inputs.package_name }}.zip ./${{ github.event.inputs.package_name }}
          echo "ARTIFACT_FILE_PATH=./packages/${{ github.event.inputs.package_name }}.zip" >> $GITHUB_ENV
        env:
          ARTIFACT_FILE_PATH: ${{ env.ARTIFACT_FILE_PATH }}

      - name: Create Release
        uses: ncipollo/release-action@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          artifacts: "${{ env.ARTIFACT_FILE_PATH }}"
          commit: ${{ github.event.inputs.commit_hash }}
          tag: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          name: ${{ github.event.inputs.package_name }}-v${{ steps.dynamic_version.outputs.package_version }}
          token: ${{ secrets.GLOBAL_GITHUB_TOKEN }}
          skipIfReleaseExists: true
```

Note that the `dynamic-versioning` action is a custom action that updates the version in the `pyproject.toml` file.

```yaml
name: "Dynamic Versioning"
description: "Bump version in python package"
author: "Or Kazaz"

inputs:
  package_name:
    description: "Package name"
    required: true

outputs:
  package_version:
    description: 'bumped package version'
    value: ${{ steps.bump_version.outputs.package_version }}

runs:
  using: composite

  steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Bump version
      id: bump_version
      shell: bash
      run: |
          PACKAGE_VERSION=$($GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ inputs.package_name }})
          echo "${{ inputs.package_name }} package will be bumped to version: ${PACKAGE_VERSION}"
          echo "package_version=$PACKAGE_VERSION" >> $GITHUB_OUTPUT
```

Bump version is a very simple bash script the increments the package version in the `pyproject.toml` file.

```bash
#!/bin/bash

PACKAGE_NAME=$1

VERSION_LINE=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml)
OLD_VERSION=$(echo "$VERSION_LINE" | awk -F'"' '{print $2}')
NEW_VERSION=$(echo "$OLD_VERSION" | awk -F. -v OFS=. '{$3=$3+1}1')

sed -i "s/$VERSION_LINE/version = \"$NEW_VERSION\"/g" ./packages/$PACKAGE_NAME/pyproject.toml

VERSION=$(grep version ./packages/$PACKAGE_NAME/pyproject.toml | awk -F'\"' '{print $2}')
echo $VERSION
```

__Job: Build__

This job builds the package distribution using Poetry.

```yaml
  build:
    needs: tag-and-release
    name: Build Python Package
    runs-on: ubuntu-latest
    timeout-minutes: 30

    steps:

      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.commit_hash }}

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install Python Dependencies
        working-directory: ./packages/${{ github.event.inputs.package_name }}
        run: |
          pip install --upgrade pip
          pip install --upgrade poetry twine check-wheel-contents

      - name: Run Dynamic-Versioning
        id: dynamic_version
        uses: ./.github/actions/dynamic-versioning
        with:
          package_name: ${{ github.event.inputs.package_name }}

      - name: Build Package Distribution
        working-directory: ./packages/${{ github.event.inputs.package_name }}
        run: |
          poetry build
```

__Job: Publish to JFrog__

This job publishes the built package to JFrog by retrieving the cached package distribution and using the official `gh-action-pypi-publish` action.

```yaml
  publish-jfrog:
    needs: build
    name: Publish Package to JFrog
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - name: Retrieve Cached Package Distribution
        uses: actions/download-artifact@v3
        with:
          name: ${{ github.event.inputs.package_name }}
          path: ./packages/${{ github.event.inputs.package_name }}/dist

      - name: Publish to JFrog
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          packages-dir: ./packages/${{ github.event.inputs.package_name }}/dist
          user: ${{ secret.secret_user }}
          password: ${{ secret.secret_password }}
          repository-url: https://the.internal.repo.url/artifactory/api/pypi/pypi

```

__Job: Commit Updated Version__

Finally, this job commits the updated version to the repository while skipping CI.


```yaml
  commit-version:
    needs: [tag-and-release, build, publish-jfrog]
    name: Commit updated version
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Commit updated version
      shell: bash
      run: |
          git config --global user.name $GITHUB_USER
          git config --global user.email $GITHUB_EMAIL
          git pull
          bash $GITHUB_WORKSPACE/.github/scripts/internal/bump_version.sh ${{ github.event.inputs.package_name }}
          git add ./packages/${{ github.event.inputs.package_name }}/pyproject.toml
          git commit -m "Bump ${{ github.event.inputs.package_name }} to version ${{ needs.tag-and-release.outputs.package_version }} [skip ci]"
          git push origin ${GITHUB_REF#refs/heads/}
```

![CI-ON-PR](https://res.cloudinary.com/dtt2usukl/image/upload/v1718703406/Image_18-06-2024_at_12.36_ge6oku.jpg)*On Pull request CI of 3 packages modified*

This setup ensures that only the affected packages are processed, reducing build times and resource usage. The use of matrix strategies, custom scripts, and composite actions provides a flexible, generic and scalable approach to managing CI/CD workflows of such use case.


In this article i'll share how I handheld the CI/CD in a mono repo with multiple python packages using GitHub Actions.

Handling CI/CD in a Mono Repo With Multiple Python Packages


Ensuring consistent application performance across multiple platforms is a cornerstone of quality software delivery. In this post, I'll share my approach for using GitHub Actions to automate testing across a variety of platforms, drawing from a real use case at my workplace.

Our solution utilizes GitHub Actions, a CI/CD service that enables automation of build, test, and deployment workflows directly within GitHub. The highlight of the approach is the dynamic matrix strategy, which handles both scheduled runs and on-demand tests across specified platforms.

__Scheduled and On-demand Testing__

In this example, the workflow is being triggered automatically at 22:00 every day for nightly tests. Additionally, we enabled manual triggers for specific platforms, allowing for targeted testing as needed. This flexibility is crucial for maintaining a fast-paced development cycle without compromising on quality.

```yaml
on:
  schedule:
    - cron: '0 22 * * *'
  workflow_dispatch:
    inputs:
      platform:
        description: 'Choose platform to run the test on'
        required: true
        type: choice
        options:
          - webos
          - tizen
          - playstation4
          - playstation5
          - xbox
```

This configuration leverages the `schedule` and `workflow_dispatch` events in GitHub Actions. The `cron` syntax schedules automated test runs, while `workflow_dispatch` allows for manual execution with platform-specific parameters.

__Dynamic Test Matrix Configuration__

The core of our workflow dynamically adjusts the test matrix based on the context of the trigger—whether it's a scheduled job covering all platforms or a manual job focusing on a specific platform.

```yaml
jobs:
  setup-test:
    runs-on: ubuntu-latest
    outputs:
      platform: ${{ steps.set-matrix.outputs.platform }}
    steps:
      - id: set-matrix
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'platform=["webos", "tizen", "playstation4", "playstation5", "xbox"]' >> $GITHUB_ENV
          else
            echo 'platform=["${{ github.event.inputs.platform }}"]' >> $GITHUB_ENV
```

In this snippet, we employ conditional logic to populate the platform variable differently based on the trigger type. For scheduled runs, we test across all platforms, while manual triggers allow for focused testing on a selected platform. This strategy ensures that resources are allocated efficiently, focusing our testing efforts where they are most needed.
Of course, you could always do that with an external script or with a composite action, but this is a simple and effective way to achieve the same result.

__Executing the Tests with Dynamic Configuration__

Our approach to executing tests in this workflow benefits significantly from the dynamic nature of GitHub Actions, especially when it comes to handling multiple platforms which can be quite different from each other. This flexibility is further extended into our application's `package.json` file, where we define specific scripts for our testing needs. The `package.json` plays a crucial role in integrating the GitHub environment variables directly into our test execution process.

Without getting into the tests code, you can imagine that each platform has its own specific tests and configurations, and that's why we need to run the tests separately for each platform. This is where the dynamic matrix comes into play.


```yaml
- name: Run tests
  working-directory: ./src/lr-launcher-e2e
  env:
    npm_config_env: ${{ matrix.platform }}
  run: npm test
```

Here, the `npm test` command is executed within the context of each platform specified by the dynamic matrix. This ensures that our E2E tests are relevant and efficient, automatically adapting to the complexities of multi-platform support.

Here's a look at the relevant part of our `package.json` file:

```json
{
  "name": "lr-launcher",
  "version": "1.0.0",
  "scripts": {
    "pre-test": "node ./utils/setupTest.js --",
    "test": "mocha \"tests/*.test.js\" \"tests/$npm_config_env/*.test.js\" --no-timeouts --retries 1 --exit"
  },
  "dependencies": {
    "axios": "^1.6.2",
    "mocha": "^10.2.0",
  }
}
```
This script utilizes the `npm_config_env` environment variable, which we set dynamically based on the selected platform in our GitHub Actions workflow. By incorporating `$npm_config_env` into the mocha command, we ensure that the test runner picks up not only the general tests from tests/*.test.js but also platform-specific tests located in directories named after each platform (e.g., tests/webos/*.test.js).

This setup allows us to maintain a clear and organized test structure where platform-agnostic tests are separated from those that are platform-specific. It ensures that when our GitHub Actions workflow runs, it only executes the relevant tests for the specified platform, making our testing process both efficient and comprehensive.


Automate cross-platform testing with GitHub Actions, using dynamic matrices for efficient, targeted tests.

GitHub Actions for Dynamic Cross-Platform Testing


At work, I manage a repository for a front-end application, which includes Markdown files that are compiled into a documentation portal.

Documentation, especially in Markdown format, is more than just words on a page. In my case, it's a user's guide, a developer's reference, and sometimes, even a product's first impression. Therefore, ensuring its quality, consistency, and accuracy is critical, and this is where Markdown linters come into play.

### Essential Tools for Markdown Linting I Used

Our approach involves three key tools, each serving a unique purpose:

__Markdownlint:__ For stylistic consistency and syntax correctness.

__Markdown-link-check:__ To validate all hyperlinks are functional.

__cspell__ (Code Spell Checker): For ensuring correct spelling throughout your documentation.

In my case, we used to run these linters on each pull request/commit that was uploaded to the repository.

### Linters

__Markdownlint:__

```bash
npm install markdownlint-cli
```

Ensures Markdown files follow best practices and maintain a consistent style. Of course, as anything the following different rules, it can be customized to your needs.

Markdown lint follows its defined rules, you can find it [here](https://github.com/markdownlint/markdownlint/blob/main/docs/RULES.md).

To modify it, you can use a configuration file, `.markdownlint.json`, which can be placed in the root of your repository.

For example:

```json
{
    "MD001": false,
    "MD013": false,
    "MD028": false,
    "MD029": false,
    "MD036": false,
    "MD033": {
        "allowed_elements": [
            "iframe",
            "url"
        ]
    }
}
```

This config file ignore certain rules defined by the tool, and allows specific elements that are not allowed by default.

__Markdown-link-check:__

```bash
npm install markdown-link-check
```

Checks for broken links in your Markdown files, ensuring all external references are valid. Note that this tools fails to check internal links from obvious reasons, therefore you can configure it to ignore them.

You can use `mlc_config.json` file to configure the tool, and place it in the root of your repository.

```json
{
    "ignorePatterns": [
        { 
            "pattern": "^https://github.com/myorganization/" 
        }
    ]
}
```

myorganization is the name of your organization in GitHub in this example.

__cspell:__

```bash
npm install cspell
```

Catches spelling mistakes, a critical aspect of maintaining professional and clear documentation. Although sometimes you might have words which aren't really a typo, but are not in the dictionary, therefore you can add them to the dictionary.

You can use `cspell.json` file to configure the tool, and place it in the root of your repository.

```json
{
    "words": [
        "tizen",
        "webos",
    ],
    "ignoreRegExpList": [
        "\\!\\[[^\\]]*\\]\$[^)]*\$",
        "(?<=```[\\s\\S]*?)\\S+",
        "\\[.*?\\]\$https?://[^\$]*\\)"
    ]
}
```

Note that I also added regex expression to ignore certain patterns, such as links, code blocks, and images alt text.


### Setting Up GitHub Actions Workflow

Integrating these tools into a GitHub Actions workflow ensures that every pull request adheres to your documentation standards. Here’s how you can configure the workflow for each tool:

```yaml
name: Documentation Checks

on:
  pull_request:
    paths:
      - 'docs'
      - 'docs/**'

jobs:
  markdown-lint:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
  
    - name: Setup-env
      uses: ./.github/actions/setup-env
      with:
        node-version: '18.16'

    - name: Run markdownlint
      run: npm run lint

  link-checker:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Setup-env
      uses: ./.github/actions/setup-env
      with:
        node-version: '18.16'

    - name: Run markdown-link-check
      run: npm run link-check

  spell-checking:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Setup-env
      uses: ./.github/actions/setup-env
      with:
        node-version: '18.16'

    - name: Run cspell
      run: npm run spell-check
```

This is a simple workflow that runs on every pull request, and runs each linter separately.
In this case I specified the paths to the docs folder, but you can change it to your needs.

In the beginning of the workflow, I used a very simple composite github action to install node and npm, and then run the linters so i won't have to duplicate the same steps in each job. It's a simple action, but it's very useful, you can improve it to your needs.

If your'e interested in the composite action, here it is:

```yaml
name: 'Setup Environment'
description: 'Checkout and setup Node.js'
inputs:
  node-version:
    description: 'NodeJS version'
    required: true
    default: '18.16'

runs:
  using: "composite"
  steps:

    - name: Setup Node
      uses: actions/setup-node@v3
      with:
        node-version: ${{ inputs.node-version }}
    
    - name: install dependencies
      shell: bash
      run: npm install
```

![markdown-linters-workflow](/images/markdown-workflows.jpeg)

By integrating these Markdown linters into your GitHub Actions workflow, you elevate the standard of your project's documentation. This setup not only ensures consistency and correctness but also encourages a culture of quality and attention to detail within your development team. Remember, good documentation reflects the quality of your code and your commitment as a developer.


Do you have a repo that contains important documentation? Streamline your documentation with Markdown linters to enhance documentation quality.

All Blog Posts in the GitHub Actions Category

Handling CI/CD in a Mono Repo With Multiple Python Packages

GitHub Actions for Dynamic Cross-Platform Testing

Markdown Linters for Your Docs Repo