Lightning-AI/ecosystem-ci

Lightning-AI/ecosystem-ci

Description: Automate issue discovery for your projects against Lightning nightly and releases.

Language: Python

License: Apache-2.0

Stars: 46

Forks: 15

Open issues: 14

Created: 2021-10-30T23:12:41Z

Pushed: 2025-05-06T08:26:32Z

Default branch: main

Fork: no

Archived: yes

README: 

Automated Testing for Lightning EcoSystem Projects

  

______________________________________________________________________

How do I add my own Project?

Pre-requisites

Here are pre-requisites for your project before adding to the Lightning EcoSystem CI:

• Your project already includes some Python tests with PyTorch Lightning as a dependency
• You'll be a contact/responsible person to resolve any issues that the CI finds in the future for your project

Adding your own project config

1. First, fork this project (with CLI or in browser) to be able to create a new Pull Request, and work within a specific branch.

gh repo fork Lightning-AI/ecosystem-ci
cd ecosystem-ci/

1. Copy the [template file](_actions/_config.yaml) in configs folder and call it .yaml.

cp configs/template.yaml configs/.yaml

1. At the minimum, modify the HTTPS variable to point to your repository. See Configuring my project for more options.

source_repository:
HTTPS: https://github.com/MyUsername/MyProject.git
...

If your project tests multiple configurations or you'd like to test against multiple Lightning versions such as master and release branches, create a config file for each one of them. As an example, have a look at [metrics master](configs/Lightning-AI/metrics_pl-develop.yaml) and [metrics release](configs/Lightning-AI/metrics_pl-release.yaml) CI files. 1. Define your runtimes (OS and Python version) in your config file to be executed on CPU and/or add the config filename in the [Azure GPU CI file](.azure/ci-testig-parameterized.yml).

• For CPU integration, specify the OS and Python version combinations inside your config file:

runtimes:
- {os: "ubuntu-20.04", python: "3.9"}
- {os: "macOS-12", python: "3.7"}
- {os: "windows-2019", python: "3.8"}
...

• For GPU integration, add your config filename in the [Azure GPU CI file](.azure/ci-testig-parameterized.yml) file:

...
jobs:
- template: testing-template.yml
parameters:
configs:
- "Lightning-AI/metrics_pl-develop.yaml"
- "Lightning-AI/metrics_pl-release.yaml"
- "MyUsername/my_project-master.yaml"

1. Add the responsible person(s) to [CODEOWNERS](.github/CODEOWNERS) for your organization folder or just the project.

# MyProject
/configs/Myusername/MyProject* @Myusername

1. Finally, create a draft PR to the repo!

Additional suggestions and engagement rules

• To qualify for GPU machines we require your project to have 100+ GitHub stars (please note that this is for capacity reasons and may change in the future)
• (Optional) Join our Slack channel #alerts-ecosystem-ci to be notified if your project is breaking
• (Kind request) include Lightning badge in your readme:

Configuring my project

The config include a few different sections:

• source_repository include your project
• env (optional) define any environment variables required when running tests
• dependencies listing all dependencies which are taken outside pip
• testing defines specific pytest arguments and what folders shall be tested

All dependencies as well as the target repository is sharing the same template with the only required field HTTPS and all others are optional:

source_repository:
HTTPS: https://github.com/Lightning-AI/metrics.git
username: my-nick # Optional, used when checking out private/protected repo
password: dont-tell-anyone # Optional, used when checking out private/protected repo
token: authentication-token # Optional, overrides the user/pass when checking out private/protected repo
checkout: master # Optional, checkout a particular branch or a tag
install_extras: all # Refers to standard pip option to install some additional dependencies defined with setuptools, typically used as `[]`.

# Optional, if any installation/tests require some env variables
env:
MY_ENV_VARIABLE: "VAR"

copy_tests:
- integrations # copied folder from the original repo into the running test directory
# this is copied as we use the helpers inside integrations as regular python package
- tests/__init__.py
- tests/helpers

# Optional, additional pytest arguments and control which directory to test on
testing:
dirs:
- integrations
pytest_args: --strict

Note: If you define some files as done above, and they are using internal-cross imports, you need to copy the __init__.py files from each particular package level.

The testing section provides access to the pytest run args and command.

testing:
# by default pytest is called on all copied items/tests
dirs:
- integrations
# OPTIONAL, additional pytest arguments
pytest_args: --strict