microsoft/dstoolkit-devcontainers

microsoft/dstoolkit-devcontainers

Description: A machine learning project template that makes it easy to work with multiple Docker based VSCode Dev Containers. Easily transition projects to the cloud and production with automated code quality checks, pytest configuration, CI pipeline templates & AML cloud example

Language: Python

License: MIT

Stars: 51

Forks: 12

Open issues: 18

Created: 2023-03-24T01:02:08Z

Pushed: 2026-06-17T04:49:47Z

Default branch: main

Fork: no

Archived: no

README:

Dev Containers for ML feasibility study with VS Code


A machine learning and data science project template that makes it easy to work with multiple Docker based VSCode Dev Containers in the same repository. The template leverages uv, an extremely fast Python package and project manager as a base for better productivity. The template also makes it easy to transition projects to the cloud and production by including automated code quality checks, pytest configuration, CI pipeline templates and a sample for running on Azure Machine Learning.

Contents

• [Dev Containers for ML feasibility study with VS Code](#dev-containers-for-ml-feasibility-study-with-vs-code)
• [Contents](#contents)
• [Introduction and Overview](#introduction-and-overview)
• [Features](#features)
• [Getting Started](#getting-started)
• [How to setup dev environment?](#how-to-setup-dev-environment)
• [How to create a new directory under src with a new environment](#how-to-create-a-new-directory-under-src-with-a-new-environment)
• [How to update python packages in the dev container](#how-to-update-python-packages-in-the-dev-container)
• [Directory Structure](#directory-structure)
• [notebooks directory vs src directory](#notebooks-directory-vs-src-directory)
• [AML Example](#aml-example)
• [CI Pipeline](#ci-pipeline)
• [Running all unit tests with ci-tests.sh](#running-all-unit-tests-with-ci-testssh)
• [How to Configure Azure DevOps CI Pipeline](#how-to-configure-azure-devops-ci-pipeline)
• [Choosing between Azure DevOps Microsoft-hosted vs Self-hosted CI Pipeline](#choosing-between-azure-devops-microsoft-hosted-vs-self-hosted-ci-pipeline)
• [How to Configure Github Actions CI Pipeline](#how-to-configure-github-actions-ci-pipeline)
• [Using SSH Keys in Dev Containers](#using-ssh-keys-in-dev-containers)
• [Future Roadmap](#future-roadmap)
• [Contributing](#contributing)
• [Trademarks](#trademarks)

Introduction and Overview

This repository provides a VSCode Dev Container based project template that can help accelerate your Machine Learning inner-loop development phase. The template covers the phases from early ML experimentation (local training/testing) until production oriented ML model training (cloud based training/testing with bigger CPUs and GPUs).

During the early phase of Machine Learning project, you may face challenges such as each data scientist creating various different python environments that span across CPU and GPU that tend to have different setup procedures. With the power of Dev Containers, you can automate environment setup process across the team and every data scientist will get the exact same environment automatically. This template provides both CPU and GPU Dev Container setup as examples. To support multiple different ML approaches with different python environments to be experimented in one project, this solution allows multiple different Dev Containers to be used in one repository.

Another challenge you may face is each data scientist creating a low quality codebase. That is fine during the experimentation stage to keep the team agility high and maximize your team’s experimentation throughput. But when you move to the model productionization stage, you experience the burden of bringing code quality up to production level. With the power of python tools and VSCode extensions configured for this template on top of Dev Containers, you can keep the code quality high automatically without losing your team’s agility and experimentation throughput and ease the transition to the productionization phase.

Features

• Multiple Dev Container samples (both CPU and GPU) with many common config steps already configured as following:
• Automated code quality checks (linter and auto formatter) and automated fix when possible with ruff on VSCode on save
• Automated code quality checks (linter and auto formatter) with ruff as precommit hook
• Zero effort transition from local env to Azure Machine Learning (cloud based env) by leveraging the same Dockerfile
• Pre-configured VSCode extensions installed such as python, jupyter, shellcheck, code-spell-checker, git tools etc
• [Github Actions and Azure DevOps CI pipelines](#ci-pipeline) that run linter (ruff) and pytest with test result reporting and coverage reporting
• Pull Request templates that helps you to write a good PR description for both Github and Azure DevOps

This template automates all tedious setup process as much as possible and saves time and reduces setup errors for the entire data scientist team.

Getting Started

This section provides a comprehensive guide on how to set up a development environment using Dev Containers in Visual Studio Code with step-by-step instructions.

How to setup dev environment?

1. Install Visual Studio Code 1. If your team has a commercial license for Docker Desktop, follow VS Code Remote Containers | Docker Desktop. Otherwise, go to VS Code Remote Containers | Rancher Desktop Docs and finish the first step (Install and launch Rancher Desktop. Select dockerd (moby) as the Container Runtime from the Kubernetes Settings menu.) 1. Install VSCode Remote - Containers extension on vscode 1. If you forget this step, you will get an error when you try to build the container so make sure you have `.env` at root of this directory before you move on to the next step.. Copy .env.example...