Custom Local LLM: Fine-tune LLM by Llama-Factory on Jetson
Introduction
🚀Finetune LLM by Llama-Factory on Jetson! Now you can tailor a custom private local LLM to meet your requirements.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/run.gif)
Llama-Factory provides a highly convenient large language model fine-tuning tool that supports common large language models, datasets, and fine-tuning methods. With this platform, we can easily customize private large language models.
In this wiki, we will learn how to deploy Llama-Factory on Nvidia Jetson and use Llama-Factory to train a large language model that supports Chinese Q&A.
Prerequisites
- Jetson device with more than 16GB of memory.
- Monitor, mouse, keyboard and network. (not necessary)
We have already tested the feasibility of this wiki on reComputer Orin NX 16GB and AGX Orin 64GB Developer Kit.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/agx_orin.png)
Getting Started
Hardware Connection
- Connect the Ethernet cable to the reComputer (Powered by Jetson).
- Connect the mouse, keyboard, and monitor to the reComputer.
- Power on reComputer.
Install Jetson-Examples
The jetson-examples by Seeed Studio offers a seamless, one-line command deployment to run vision AI and Generative AI models on the NVIDIA Jetson platform.
To install the package, please open the terminal in Jetson and run:
pip3 install jetson-examples
sudo reboot
Install and Run Llama-Factory on Jetson
Deploy Llama-Factory
by jetson-examples in one-line:
reComputer run llama-factory
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/run_llama_factory.png)
We can then open a web browser and access the address to open the WebUI:
# http://<jetson-ip>:7860
http://127.0.0.1:7860
Start Training
Here, we use the alpaca_zh
dataset to fine-tune the Phi-1.5
model, enabling it to have Chinese conversational capabilities. Therefore, in the web UI, we only configure the training Model name
and Dataset
, keeping the other training parameters as default.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/run_train.png)
Finally, click the start
button to initiate the training.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/training.png)
The training process will take approximately 18 hours.
After completing the fine-tuning, you can find the fine-tuned model in the save directory.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/train_result.png)
Testing the Fine-tuned Model.
Finally, we can use Llama-Factory with the fine-tuned model to test if it indeed has acquired Chinese conversational capabilities. The specific steps are as follows.
Step1. Load the fine-tuned model by Llama-Factory WebUI.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/load_model.png)
Step2. Enter a Chinese prompt in the Input
text box, click the Submit
button, and check the output result of the large language model in the Chatbot
text box.
![](https://files.seeedstudio.com/wiki/reComputer-Jetson/Llama-Factory/test_model.png)
From the test results, we can see that the fine-tuned model already has the capability to talk with human in Chinese. If you want your model to have more advanced capabilities, try using a more diverse set of fine-tuning data to train your model!
Demonstration
References
- https://github.com/hiyouga/LLaMA-Factory
- https://github.com/dusty-nv/jetson-containers
- https://github.com/Seeed-Projects/jetson-examples
Tech Support & Product Discussion
Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.