togethercomputer/redpajama.cpp

togethercomputer/redpajama.cpp

Description: Extend the original llama.cpp repo to support redpajama model.

Language: C

License: MIT

Stars: 117

Forks: 14

Open issues: 3

Created: 2023-05-03T14:14:29Z

Pushed: 2024-09-03T22:47:38Z

Default branch: master

Fork: yes

Parent repository: ggml-org/llama.cpp

Archived: no

README:

llama.cpp

!llama


Inference of LLaMA model in pure C/C++

Hot topics:

• Roadmap May 2023
• New quantization methods

RedPajama Support

For RedPajama Models, see this example.

Description

The main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook

• Plain C/C++ implementation without dependencies
• Apple silicon first-class citizen - optimized via ARM NEON and Accelerate framework
• AVX, AVX2 and AVX512 support for x86 architectures
• Mixed F16 / F32 precision
• 4-bit, 5-bit and 8-bit integer quantization support
• Runs on the CPU
• OpenBLAS support
• cuBLAS and CLBlast support

The original implementation of llama.cpp was hacked in an evening. Since then, the project has improved significantly thanks to many contributions. This project is for educational purposes and serves as the main playground for developing new features for the ggml library.

Supported platforms:

• [X] Mac OS
• [X] Linux
• [X] Windows (via CMake)
• [X] Docker

Supported models:

• [X] LLaMA 🦙
• [X] Alpaca
• [X] GPT4All
• [X] Chinese LLaMA / Alpaca
• [X] Vigogne (French)
• [X] Vicuna
• [X] Koala
• [X] OpenBuddy 🐶 (Multilingual)

Bindings:

• Python: abetlen/llama-cpp-python
• Go: go-skynet/go-llama.cpp
• Node.js: hlhr202/llama-node
• Ruby: yoshoku/llama_cpp.rb

UI:

• nat/openplayground
• oobabooga/text-generation-webui

---

Here is a typical run using LLaMA-7B:

make -j && ./main -m ./models/7B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -n 512
I llama.cpp build info:
I UNAME_S: Darwin
I UNAME_P: arm
I UNAME_M: arm64
I CFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -pthread -DGGML_USE_ACCELERATE
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread
I LDFLAGS: -framework Accelerate
I CC: Apple clang version 14.0.0 (clang-1400.0.29.202)
I CXX: Apple clang version 14.0.0 (clang-1400.0.29.202)

make: Nothing to be done for `default'.
main: seed = 1678486056
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx = 512
llama_model_load: n_embd = 4096
llama_model_load: n_mult = 256
llama_model_load: n_head = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 11008
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size = 512.00 MB, n_mem = 16384
llama_model_load: .................................... done
llama_model_load: model size = 4017.27 MB / num tensors = 291

main: prompt: 'Building a website can be done in 10 simple steps:'
main: number of tokens in prompt = 15
1 -> ''
8893 -> 'Build'
292 -> 'ing'
263 -> ' a'
4700 -> ' website'
508 -> ' can'
367 -> ' be'
2309 -> ' done'
297 -> ' in'
29871 -> ' '
29896 -> '1'
29900 -> '0'
2560 -> ' simple'
6576 -> ' steps'
29901 -> ':'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000

Building a website can be done in 10 simple steps:
1) Select a domain name and web hosting plan
2) Complete a sitemap
3) List your products
4) Write product descriptions
5) Create a user account
6) Build the template
7) Start building the website
8) Advertise the website
9) Provide email support
10) Submit the website to search engines
A website is a collection of web pages that are formatted with HTML. HTML is the code that defines what the website looks like and how it behaves.
The HTML code is formatted into a template or a format. Once this is done, it is displayed on the user's browser.
The web pages are stored in a web server. The web server is also called a host. When the website is accessed, it is retrieved from the server and displayed on the user's computer.
A website is known as a website when it is hosted. This means that it is displayed on a host. The host is usually a web server.
A website can be displayed on different browsers. The browsers are basically the software that renders the website on the user's screen.
A website can also be viewed on different devices such as desktops, tablets and smartphones.
Hence, to have a website displayed on a browser, the website must be hosted.
A domain name is an address of a website. It is the name of the website.
The website is known as a website when it is hosted. This means that it is displayed on a host. The host is usually a web server.
A website can be displayed on different browsers. The browsers are basically the software that renders the website on the user’s screen.
A website can also be viewed on different devices such as desktops, tablets and smartphones. Hence, to have a website displayed on a browser, the website must be hosted.
A domain name is an address of a website. It is the name of the website.
A website is an address of a website. It is a collection of web pages that are formatted with HTML. HTML is the code that defines what the website looks like…