basetenlabs/avatar-generation-app-tutorial

basetenlabs/avatar-generation-app-tutorial

Description: A thorough tutorial + working code for building your own Lensa-like app using Blueprint

Language: TypeScript

Stars: 22

Forks: 3

Open issues: 0

Created: 2023-02-27T08:53:34Z

Pushed: 2023-03-01T18:16:50Z

Default branch: main

Fork: no

Archived: no

README:

Building a Lensa-like app with Blueprint

Background

Blueprint is a fine-tuning and serving infrastructure toolkit for software developers who are comfortable with backend and frontend engineering but lack expertise in model development and hosting.

Blueprint works great for building user-facing applications with fine-tuned models. One popular type of app using fine-tuning is AI avatar generation apps. This project implements a simple version of an avatar generation app like Lensa using Blueprint.

!screenshot-lensa-app

This repo contains:

• A tutorial on building your own avatar generation application
• Code snippets for a lightweight front-end and back-end application that allows users to authenticate, upload some data, and kick-off a fine-tuning job before allowing them to use their models to generate new photos

Tutorial

Prerequisites

• A Blueprint account For fine-tuning, plus a valid API key. https://docs.blueprint.baseten.co/getting-started
• A Supabase account for storage and user state management
• A Vercel account for deploying your NextJS application

Creating your Next.js App and Setup

We'll be using Next.js for building the front-end, and Supabase for authentication.

To get set up:

git clone git@github.com:tuhins/fine-tuning-app.git
npm install
npm run dev

Once you clone this repo, you can see the relevant parts of the application:

• Supabase client
• Login page
• Main application page

Follow the instructions here to create a basic Supabase project.

Add your NEXT_PUBLIC_SUPABASE_URL and NEXT_PUBLIC_SUPABASE_ANON_KEY to the .env.local in the project root folder. The file's contents should look like:

NEXT_PUBLIC_SUPABASE_URL=https://XXXXXXXX.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=YYYYYYYYY

Collecting user data

We're using Supabase storage to collect files from users which we zip up, upload to Supabase and then pass that URL to Blueprint.

First set up a Supabase bucket for storing data. You can follow the great guide from Supabase to get started (call it something like fine-tuning-bucket; if you call it something else you'll have to update the `index.tsx`). Make sure that the bucket is marked as public and set the policy as open for reading, writing, deleting and updating data; here is what your policy should look like. The bulk of the work here is done by the upload handler, which collects images from the user via an input component, zips them up and uploads the blob to Supabase. Supabase provides a publicly accessible URL to pass through to the Blueprint API.

Next, create a new table in Supabase called finetuningruns. We'll be using this table to link users with fine tuning runs created through Blueprint. For this purpose of this demo, disable Row-Level-Security. This table only needs a few columns:

• user_id (linked to users table, join on id) see screenshot
• run_id (_varchar_)
• dataset (_text_)

Fine tuning routes

Blueprint provides a simple SDK — given a set of images, it's straightforward to programmatically schedule fine-tuning jobs, deploy fine-tuned models, and provides an inference API for the deployed model. It also provides a dataset API for validating uploaded user data, but we won't be using that in this demo.

Blueprint also provides a serverless backend to be able to build on top of its fine-tuning SDK. Think of these like Flask routes. We'll set up some serverless routes that interact with the fine-tuning SDK. We'll write this code in the Blueprint web IDE.

Create a new Blueprint project

First, we'll get set up on Blueprint.

1. Go to https://app.baseten.co/blueprint/projects 2. Click New Project 3. Copy URL with the project ID 4. Back in this repository, go to index.tsx and update the Blueprint project route

!Create New Blueprint Project

Quick overview of Blueprint endpoints

Blueprint endpoints are like serverless functions. They are defined in the main.py file accessible via the sidebar in your Blueprint project. They run in a fully-managed environment — its requirements can be defined and managed in the requirements.txt file also accessible via the sidebar in your Blueprint project. You can read more about Blueprint endpoints.

All the code referenced below should be used via Blueprint endpoints. The full-code can be found in the main.py, and in the requirements file.

Endpoint for triggering a fine-tuning job

We'll create a endpoint that takes the url of a fine-tuning job and a user identifier to trigger a fine-tuning job. Names must be unique, and ensure that permissions on the file are set to open or whatever you can configure from the backend. We'll store a link between the user, the dataset and the fine-tuning run in a Supabase…