Skip to main content

Custom Local LLM: Fine-tune LLM by Llama-Factory on Jetson

Introduction​

🚀Finetune LLM by Llama-Factory on Jetson! Now you can tailor a custom private local LLM to meet your requirements.

Llama-Factory provides a highly convenient large language model fine-tuning tool that supports common large language models, datasets, and fine-tuning methods. With this platform, we can easily customize private large language models.

In this wiki, we will learn how to deploy Llama-Factory on Nvidia Jetson and use Llama-Factory to train a large language model that supports Chinese Q&A.

Prerequisites​

  • Jetson device with more than 16GB of memory.
  • Monitor, mouse, keyboard and network. (not necessary)
note

We have already tested the feasibility of this wiki on reComputer Orin NX 16GB and AGX Orin 64GB Developer Kit.

Get One Now 🖱️

Getting Started​

Hardware Connection​

  1. Connect the Ethernet cable to the reComputer (Powered by Jetson).
  2. Connect the mouse, keyboard, and monitor to the reComputer.
  3. Power on reComputer.

Install Jetson-Examples​

note

The jetson-examples by Seeed Studio offers a seamless, one-line command deployment to run vision AI and Generative AI models on the NVIDIA Jetson platform.

To install the package, please open the terminal in Jetson and run:

pip3 install jetson-examples
sudo reboot

Install and Run Llama-Factory on Jetson​

Deploy Llama-Factory by jetson-examples in one-line:

reComputer run llama-factory

We can then open a web browser and access the address to open the WebUI:

# http://<jetson-ip>:7860
http://127.0.0.1:7860

Start Training​

Here, we use the alpaca_zh dataset to fine-tune the Phi-1.5 model, enabling it to have Chinese conversational capabilities. Therefore, in the web UI, we only configure the training Model name and Dataset, keeping the other training parameters as default.

Finally, click the start button to initiate the training.

note

The training process will take approximately 18 hours.

After completing the fine-tuning, you can find the fine-tuned model in the save directory.

Testing the Fine-tuned Model.​

Finally, we can use Llama-Factory with the fine-tuned model to test if it indeed has acquired Chinese conversational capabilities. The specific steps are as follows.

Step1. Load the fine-tuned model by Llama-Factory WebUI.

Step2. Enter a Chinese prompt in the Input text box, click the Submit button, and check the output result of the large language model in the Chatbot text box.

From the test results, we can see that the fine-tuned model already has the capability to talk with human in Chinese. If you want your model to have more advanced capabilities, try using a more diverse set of fine-tuning data to train your model!

Demonstration​

References​

Tech Support & Product Discussion​

Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.

Loading Comments...