Run Code Llama 70B with an API

Run Code Llama 70B with an API – Replicate blog

Replicate Blog

Run Code Llama 70B with an API

Posted January 30, 2024 by cbh123

Code Llama is a code generation model built on top of Llama 2. It can generate code and natural language about code in many programming languages, including Python, JavaScript, TypeScript, C++, Java, PHP, C#, Bash and more.

Today, Meta announced a more powerful new version of Code Llama with 70 billion parameters. It’s one of the highest performing open models. Meta reports a 67.8 on HumanEval, which beats zero-shot GPT-4 .

With Replicate, you can run Code Llama 70B in the cloud with one line of code.

Contents

Contents

Code Llama 70B variants

Run Code Llama 70B with JavaScript

Run Code Llama 70B with Python

Run Code Llama 70B with cURL

Keep up to speed

Code Llama 70B variants

There are three variants of Code Llama 70B. The code snippets in this guide use codellama-70b-instruct , but all three variants are available on Replicate:

Code Llama 70B Base is the foundation model.

Code Llama 70B Python is trained on Python code.

Code Llama 70B Instruct is fine-tuned for understanding natural language instructions.

Run Code Llama 70B with JavaScript

You can run Code Llama 70B with our official JavaScript client :

Copy

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

Copy

export REPLICATE_API_TOKEN =

Import and set up the client:

Copy

import Replicate from "replicate" ;

const replicate = new Replicate ({ auth: process.env. REPLICATE_API_TOKEN , });

Run meta/codellama-70b-instruct using Replicate’s API:

Copy

const output = await replicate. run ( "meta/codellama-70b-instruct:a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf" , input: { prompt: "In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?" , } ); console. log (output);

To learn more, take a look at the guide on getting started with Node.js .

Run Code Llama 70B with Python

You can run Code Llama 70B with our official Python client :

Copy

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

Copy

export REPLICATE_API_TOKEN =

Run meta/codellama-70b-instruct using Replicate’s API:

Copy

import replicate

output = replicate.run( "meta/codellama-70b-instruct:a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf" , input = { "prompt" : "In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?" , } ) print ( "" .join(output))

To learn more, take a look at the guide on getting started with Python .

Run Code Llama 70B with cURL

You can call the HTTP API directly with tools like cURL:

Set the REPLICATE_API_TOKEN environment variable:

Copy

export REPLICATE_API_TOKEN =

Run meta/codellama-70b-instruct using Replicate’s API:

Copy

curl -s -X POST \ -H "Authorization: Bearer $REPLICATE_API_TOKEN " \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ -d $'{ "version": "a279116fe47a0f65701a8817188601e2fe8f4b9e04a518789655ea7b995851bf", "input": { "top_k": 10, "top_p": 0.95, "prompt": "In Bash, how do I list all text files in the current directory (excluding subdirectories) that have been modified in the last month?", "max_tokens": 500, "temperature": 0.8, "system_prompt": "", "repeat_penalty": 1.1, "presence_penalty": 0, "frequency_penalty": 0 } }' \ https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

You can also run Code Llama 70B using other Replicate client libraries for Go, Swift, and others

Keep up to speed

Check out our Code Llama 70B model page

Follow us on Twitter X to get the latest from Replicate

Hop in our Discord and talk Code Llama 70B

Next: How to create an AI narrator for your life