Skip to main content

Control Motor by Voice LLM on Jetson

Introduction​

This wiki builds an end-to-end voice-controlled motor system based on the Jetson platform (reComputer Robotics J4012), integrating speech recognition, edge large language model (LLM) intent understanding, and hardware control to enable natural human-machine interaction—users can control MyActuator X Series Motors with simple voice commands (e.g., "Rotate 90 degrees clockwise").

Prerequisites​

  • reComputer Robotics J4012
  • reSpeaker XVF3800
  • MyActuator X Series Motors
reComputer Robotics J4012reSpeaker XVF3800MyActuator X Series Motors
Get One Now 🖱️
Get One Now 🖱️
Get One Now 🖱️

Hardware Connection​

Getting Started​

The workflow of the entire system involves three steps:

  1. ASR: The microphone captures audio commands from the user and converts them into text using Whisper.
  2. Function Calling: The large language model generates motor control parameters that meet the requirements based on the user's input instructions.
  3. Motor Control: The appropriate control program is called to drive the motor to rotate to the specified position.

Next, we will explain the implementation of each step in detail.

Install Whisper Server​

First, we need to install the Whisper service to enable ASR functionality. Please open a terminal window on the Jetson device and run the following command.

git clone https://github.com/jjjadand/whisper-stable4curl
cd whisper-stable4curl
export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
cmake --build build -j --config Release

If everything goes well, you will see the following logs in the terminal window.

Then we launch the Whisper inference service:

./build/bin/whisper-stream -m ./models/ggml-base.en-q5_1.bin -t 8 --step 0 --length 7000 -vth 0.7 --keep 1200

Install Ollama​

Ollama is an extremely user-friendly edge computing LLM inference framework that can be deployed on Jetson with just a single command. Open a new terminal window on your Jetson device and execute:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5
info

Here, we use the Qwen 2.5 large language model to understand the user's intent.

Install Motor Control Script​

Execute the following command to start the motor control script:

git clone https://github.com/yuyoujiang/voice_control.git
cd voice_control
sudo ip link set can0 type can bitrate 1000000
sudo ip link set can0 up
python app.py

Effect Demonstration​

Tech Support & Product Discussion​

Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.

Loading Comments...