How to Build a $300 AI Computer for the GPU-Poor

Hi, I have not tried the 5700U myself. By looking at the specs, they might not differ that much in LLM inference tasks. Please try it out and let us know your results.

The Cheapskate’s Guide to Fine-Tuning LLaMA-2 and Running It on Your Laptop

When you convert the Lora model into general GGML model, it already merged both the Lora adapter (e.g. 100MB parameters) as well as the original “mother” LLM weights (7B parameters). So in the last step, you are running the fine-tuned large model.

A Practical 5-Step Guide to Do Semantic Search on Your Private Data With the Help of LLMs

I think the semantic search with vector DB can be cheaper than  fine-tuning existing LLM (e.g. LLaMA), less hallucination, faster, and more controllable. And there are many knobs to turn (chunking size, overlapping, hierarchical indexing, meta data search, etc) in order to get the best results you want.

At least for me, the demo’d solution works better than existing site-wide search.

I don’t think it is poor, but it is definitely slow. So I use HuggingFaceEmbeddings nowadays if I want to do everything local. Otherwise I use openAI embedding, if I don’t need to worry about privacy.

Let me clarify, the embedding module is called “huggingface”, but it is simply a wrapper for “sentence transformers embedding” and does not require any Internet access or cloud API.

You only need to change two lines of code, and it is much much faster to do embedding.

```javascript
from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings()
```

If you don’t trust me, you can find the doc here :-)

<https://python.langchain.com/en/latest/modules/models/text_embedding/examples/sentence_transformers.html>

Absolutely. After publishing the article, I did some experiments and changed the embeddings to huggingface embedding. It is super fast, and only took 15 minutes to do the embedding for 7500 chunks of 1000 word each.

I Conducted Experiments With the Alpaca/LLaMA 7B Language Model: Here Are the Results

This space is advancing faster than my typing. A few more tips since my last comment:

\
1) There are several projects doing similar things like Alpaca, namely Vicuna, GPT4All and Koala. They are all very good, all based on LLaMA (7B or 13B models), and perform reasonably close to chatGPT 3.5. And you can use any of them with confidence.

2) To download the weights for these models, your best friend is huggingface. When you search for example “vicuna”, make sure the model is 4 bit quantized and in GGML format for llama.cpp.

3) If you computer have enough RAM (e.g. more than 16GB), you might want to add --mlock switch to the end of the command. This will preload the entire weight db into RAM, which will be slower to start but faster for token generation.

\
Feel free to share in the comments if you find any other tips and tricks for the community!

Since the recent llama.cpp code update, readers have reported errors when loading the alpaca-7b-4bit model weight file. There are two solutions.

\
**1) Better solution: migrate the weight file to newer format**

`python migrate-ggml-2023-03-30-pr613.py ./models/ggml-alpaca-7b-q4.bin ./models/ggml-alpaca-7b-q4-updated.bin`

\
2) Revert to older version of llama.cpp by installing earlier releases, e.g. March 24th, 2023 when this article was published

<https://github.com/ggerganov/llama.cpp/releases>

\


Since I published the article, there are a few new open source AI projects focusing on the smaller LLM model training and fine-tuning. And more are popping up every day. We are indeed at the Cambrian explosion period of AI research and development.

* __[Colossal AI](https://github.com/hpcaitech/ColossalAI)__
* Together.xyz __[OpenChatKit](https://github.com/togethercomputer/OpenChatKit)__
* Nomic-ai __[GPT4All](https://github.com/nomic-ai/gpt4all)__

\


Bruce Li