Supported Software Overview

There are two methods enabling the Grove Vision AI Module V2:

• No code getting started with SenseCraft AI
  • What is SenseCraft AI?
  • Getting Started with SenseCraft AI Model Assistant
• Program on Arduino connecting with Seeed Studio XIAO Board
  • Arduino Library Introduction
  • Getting Started with Seeed Studio XIAO

No code getting started with SenseCraft AI

In this section, we'll walk you through using SenseCraft AI Model Assistant to quickly deploy and observe results, with no code and no other developemtn boards:

What is SenseCraft AI?

SenseCraft AI empowers users to effortlessly deploy a vast library of publicly available AI models onto their edge devices:

• reComputer (NVIDIA Jetson)
• Seeed Studio XIAO S3 Board
• Vision AI modules, and more

It provides a seamless and user-friendly experience, allowing you to deploy public AI models directly onto your edge devices with just a few clicks.

Learn More 📕

It includes:

• Model Assistant software service
• Device Workspace software service

In this section, we will use "Model Assistant" here to enable the module. Combined with the empowerment of SenseCraft AI Model Assistant, you can easily upload a wide variety of co-created models and directly observe the results.

Getting Started with SenseCraft AI Model Assistant

Now we will quickly get started the modules with SenseCraft AI, and this will only require the mnodule only.

Step 1. Connect the Grove Vision AI V2 to the SenseCraft AI Model Assistant

First, we need to open the main SenseCraft AI Model Assistant page.

Go to SenseCraft AI

Please use a Type-C type cable to connect Grove Vision AI V2 to your computer.

In the upper right corner of the SenseCraft AI Model Assistant page, you can select Grove Vision AI (WE2). Then click the Connect button on the far right.

In the new window that pops up, select the correct COM port for the device and click the Connect button.

Step 2. Upload a suitable model

Then, just select a model you want to use and click the Send button below. Here is an example of Gesture Detection.

Please remain on this page for 1-2 minutes until the model is successfully uploaded. Kindly note that switching to another page tab during this process may result in an unsuccessful upload (our team is actively working on resolving this issue, and it will be fixed soon).

Step 3. Observations

Once the model is uploaded successfully, you will be able to see the live feed from the Grove Vision AI V2 camera in the Preview on the right.

We can see that in the Preview Settings on the right hand side, there are two setting options that can be changed to optimise the recognition accuracy of the model.

• Confidence: Confidence refers to the level of certainty or probability assigned by a model to its predictions.

• IoU: IoU is used to assess the accuracy of predicted bounding boxes compared to truth bounding boxes.

tip

• Confidence: This tells you how sure the computer vision model is about its guess. Think of it like a percentage: higher confidence means the model is more certain.
• IoU (Intersection over Union): This measures how much overlap there is between two boxes. Imagine you drew a box around an object in a picture, and the model also drew a box. IoU tells you how well your boxes match up. Higher IoU means the boxes are closer together.

At this point, you have finished uploading the model and observing the results.

Program on Arduino connecting with Seeed Studio XIAO Board

Arduino Library Introduction

tip

If this is your first time using Arduino, we highly recommend you to refer to Getting Started with Arduino.

Download the Library

Grove Vision AI uses WiseEye2 HX6538 chip, image processing and model inference are processed locally in Grove Vision AI, and then the result is output to XIAO via IIC or UART, so the main purpose of this library is to process the data stream of Grove Vision AI, and it doesn't involve model inference or image processing. So the main purpose of this library is to process the data stream of Grove Vision AI, without involving model inference or image processing.

The Grove Vision AI communicates with the XIAO via the IIC, and the device's IIC address is 0x62. Image information transfer is via the USB serial port.

Function

Before we get started developing a sketch, let's look at the available functions of the library.

• bool begin(TwoWire *wire = &Wire, uint16_t address = I2C_ADDRESS, uint32_t wait_delay = 2, uint32_t clock = 400000) —— Initialise Grove Vision AI V2.

  Input Parameters:

  • TwoWire *wire —— This pointer points to a TwoWire object, typically used to communicate with I2C devices.
  • uint16_t address —— This is the address of the I2C device that identifies the particular device connected to the I2C bus.
  • uint32_t wait_delay —— The delay (in milliseconds) to wait for a response before sending a command.
  • uint32_t clock —— This is the clock rate of the I2C bus (in Hz).

  Return: True or False. Initialisation success is true, initialisation failure is false.

• int invoke(int times = 1, bool filter = 0, bool show = 0) —— Used to send INVOKE commands to the Grove Vision AI V2 to allow the Grove Vision AI to start calling models, inference and recognition.

  Input Parameters:

  • int times —— The number of Invoke.
  • fileter —— Means the event reply will only be sent if the last result is different from the previous result (compared by geometry and score).
  • bool show —— Means the event reply will only be sent if the last result is different from the previous result (compared by geometry and score).

note

For more information about Grove Vision AI's protocol definitions, you can read the protocol documentation.

Return: CMD_OK or CMD_ETIMEDOUT. Returns CMD_OK if the model was enabled successfully, otherwise returns CMD_ETIMEDOUT.

• int available() —— Check how many bytes of data are available to read from the connected device via the IIC.

  Input Parameters: None.

  Return: The number of data bytes that can be read from the device.

• int read(char *data, int length) —— Reads data from the Grove Vision AI via the IIC interface. The purpose of the function is to populate the array pointed to by the provided data pointer with the read data.

  Input Parameters:

  • char *data —— An array for storing data.
  • int length —— The length of the data to be read.

  Return: The length of the data to be read.

• int write(const char *data, int length) —— Writes data to the specified device via the I2C interface.

  Input Parameters:

  • const char *data —— The content of the data to be written.
  • int length —— The length of the data to be written.

  Return: The length of the data to be written.

• std::vector<boxes_t> &boxes() { return _boxes; } —— Output the results in the form of a box.

typedef struct
{
    uint16_t x;      // Horizontal coordinates of the centre of the box
    uint16_t y;      // Vertical coordinates of the centre of the box
    uint16_t w;      // Width of the identification box
    uint16_t h;      // Height of the identification box
    uint8_t score;   // Confidence in identifying as target
    uint8_t target;  // Target
} boxes_t;

• std::vector<classes_t> &classes() { return _classes; } —— Outputs the results as categories.

typedef struct
{
    uint8_t target;  // Target
    uint8_t score;   // Confidence in identifying as target
} classes_t;

• std::vector<point_t> &points() { return _points; } —— Outputs the result as a dot.

typedef struct
{
    uint16_t x;      // Horizontal coordinates of the identification point
    uint16_t y;      // Vertical coordinates of identification point
    uint16_t z;      // Relative depth coordinates of the identification point
    uint8_t score;   // Confidence in identifying as target
    uint8_t target;  // Target
} point_t;

note

The depth coordinate here is relative, it does not mean that Grove Vision AI is depth camera enabled, but rather the algorithm calculates a relative depth coordinate which is valid in some models (e.g., face-3d models).

• perf_t &perf() { return _perf; } —— Image processing and inference time.

typedef struct
{
    uint16_t prepocess;   // Pre-processing time
    uint16_t inference;   // inference time
    uint16_t postprocess; // Post-processing time
} perf_t;

note

The outputs do not always always output information that identifies boxes and points, depending on the model.

Installation

Since you have downloaded the zip Library, open your Arduino IDE, click on Sketch > Include Library > Add .ZIP Library. Choose the zip file you just downloaded，and if the library install correct, you will see Library added to your libraries in the notice window. Which means the library is installed successfully.

Getting Started with Seeed Studio XIAO

If you want to get a prototype of a sensor with machine vision capabilities, then you may want to consider adding an XIAO. Using XIAO and Grove Vision AI V2, you can use the model you want to use and quickly deploy it in your application area.

Preliminary

Step 1. Materials Required

In this tutorial, we will use the XIAO ESP32S3 as an example to introduce the use of Arduino programmes. So, we recommend you to prepare the following hardware.

Seeed Studio XIAO ESP32S3	Grove Vision AI V2	OV5647-62 FOV Camera Module for Raspberry Pi 3B+4B
Get One Now 🖱️	Get One Now 🖱️	Get One Now 🖱️

You then need to connect the XIAO and Grove Vision AI V2 via the row of pins (or use the expansion board and Grove interface).

caution

Please note the direction of the connection, Grove Vision AI's Type-C connector should be in the same direction as XIAO's Type-C connector.

Step 2. Launch the Arduino application.

Download Arduino IDE

Step 3. Select your development board model and add it to the Arduino IDE.

tip

Below is a list of supported boards for the Grove Vision AI V2 example, you can choose the board you want to use to complete the next example. In this tutorial, XIAO ESP32S3 will be used as an example.

• If you want to use Seeed Studio XIAO SAMD21 for the later routines, please refer to this tutorial to finish adding.

• If you want to use Seeed Studio XIAO RP2040 for the later routines, please refer to this tutorial to finish adding.

• If you want to use Seeed Studio XIAO nRF52840 for the later routines, please refer to this tutorial to finish adding.

• If you want to use Seeed Studio XIAO ESP32C3 for the later routines, please refer to this tutorial to finish adding.

• If you want to use Seeed Studio XIAO ESP32S3 for the later routines, please refer to this tutorial to finish adding.

• If you want to use Seeeduino V4.3 for the later routines, please refer to this tutorial to finish adding.

Step 4. Install the ArduinoJSON library.

Go to the Sketch menu, then select Include Library > Manage Libraries.... This will open the Library Manager. In the search bar at the top of the Library Manager, type in ArduinoJSON. The search results will list the ArduinoJSON library. There will be an Install button next to the library. Click the Install button. The Arduino IDE will automatically download and install the library into your Arduino development environment.

Demo 1. Use XIAO to get recognition results

The following procedure is a simple example to help you obtain and parse the recognition information reported by Grove Vision AI V2.

#include <Seeed_Arduino_SSCMA.h>

SSCMA AI;

void setup()
{
    AI.begin();
    Serial.begin(9600);
}

void loop()
{
    if (!AI.invoke())
    {
        Serial.println("invoke success");
        Serial.print("perf: prepocess=");
        Serial.print(AI.perf().prepocess);
        Serial.print(", inference=");
        Serial.print(AI.perf().inference);
        Serial.print(", postpocess=");
        Serial.println(AI.perf().postprocess);

        for (int i = 0; i < AI.boxes().size(); i++)
        {
            Serial.print("Box[");
            Serial.print(i);
            Serial.print("] target=");
            Serial.print(AI.boxes()[i].target);
            Serial.print(", score=");
            Serial.print(AI.boxes()[i].score);
            Serial.print(", x=");
            Serial.print(AI.boxes()[i].x);
            Serial.print(", y=");
            Serial.print(AI.boxes()[i].y);
            Serial.print(", w=");
            Serial.print(AI.boxes()[i].w);
            Serial.print(", h=");
            Serial.println(AI.boxes()[i].h);
        }
        for (int i = 0; i < AI.classes().size(); i++)
        {
            Serial.print("Class[");
            Serial.print(i);
            Serial.print("] target=");
            Serial.print(AI.classes()[i].target);
            Serial.print(", score=");
            Serial.println(AI.classes()[i].score);
        }
        for (int i = 0; i < AI.points().size(); i++)
        {
            Serial.print("Point[");
            Serial.print(i);
            Serial.print("] target=");
            Serial.print(AI.points()[i].target);
            Serial.print(", score=");
            Serial.print(AI.points()[i].score);
            Serial.print(", x=");
            Serial.print(AI.points()[i].x);
            Serial.print(", y=");
            Serial.println(AI.points()[i].y);
        }
    }
}

Explanation of procedure

This Arduino sketch utilizes the Seeed_Arduino_SSCMA library to interface with the Grove Vision AI Module V2. In the setup() function, the AI module is initialized, and serial communication is started.

The loop() function repeatedly calls the invoke() method to perform inference using the built-in algorithms of the Grove Vision AI Module V2. Upon a successful inference, the sketch prints out performance metrics to the serial monitor, including preprocessing, inference, and postprocessing times.

The sketch processes and prints out detailed information about the results of the inference, including:

• Bounding boxes (boxes()) that identify the locations and dimensions of detected objects in the form of x and y coordinates, width, and height.
• Classifications (classes()) that identify the categories of detected objects along with their confidence scores.
• Points (points()) that represent specific features or keypoints of detected objects, along with their x and y coordinates and confidence scores.

These results provide insights into the objects detected by the AI module, their positions, sizes, and the confidence level of each detection or classification. The output is printed to the serial monitor for further analysis or debugging.

Effect

We are still using the Gesture Detection model. After you have uploaded the application, please open the serial monitor and set the baud rate of the serial monitor to 9600. Please prepare your "rock-paper-scissors" and point it at the recognition area of the camera, and the serial monitor will output the recognition result.

Demo 2. Send control commands via XIAO

The Grove Vision AI V2 supports communication via the UART protocol, and by using the built-in protocol, it is possible to operate the Himax WE2 chip of the Grove Vision AI V2. The following sample program shows how to use the XIAO to communicate with the Grove Vision AI V2 over the serial port.

#include <Arduino.h>
#include <Wire.h>

#include <Seeed_Arduino_SSCMA.h>

SSCMA AI;

void setup()
{
    // put your setup code here, to run once:
    Wire.begin();
    Serial.begin(115200);
    while (!Serial)
        delay(1000);

    Serial.println("Proxy start");
}

char buf[512];
void loop()
{
    int s_len = Serial.available();
    if (s_len)
    {
        int len = Serial.readBytes((char *)buf, s_len);
        AI.write(buf, len);
    }
    int t_len = AI.available();
    if (t_len)
    {
        if (t_len > 512)
        {
            t_len = 512;
        }

        AI.read(buf, t_len);
        Serial.write(buf, t_len);
    }
}

Explanation of procedure

This Arduino sketch establishes a communication interface between the serial console and the Grove Vision AI V2 module, utilizing the Seeed Studio Machine Learning Framework SSCMA. The sketch is designed to pass data to and from the Grove Vision AI V2 module via serial commands.

Here's the modified description of the sketch:

In the setup() function:

• The I2C communication protocol is initialized with Wire.begin(), allowing for communication with the Grove Vision AI V2 module.
• The serial communication is set up at a baud rate of 115200.
• The sketch waits for the serial connection to be active before it outputs "Proxy start" to the serial monitor, indicating that the proxy communication channel is ready.

In the loop() function:

• The sketch checks for incoming serial data using Serial.available(). If there is data, it reads it into a buffer named buf.
• The data collected in the buffer is then sent to the Grove Vision AI V2 module using the AI.write() method.
• Next, the sketch checks if the Grove Vision AI V2 module has any data to send back using AI.available().
• If the Grove Vision AI V2 module has sent data, the sketch reads this data into the buffer buf with AI.read(), ensuring the data does not exceed the buffer size of 512 bytes.
• Finally, the received data from the Grove Vision AI V2 module is transmitted back to the serial console using Serial.write(), completing the two-way communication between the serial interface and the vision AI module.

This setup allows for interactive communication with the Grove Vision AI V2 module, enabling users to send instructions to and receive data from the module over a serial connection.

Resources

• SenseCraft AI
• SenseCraft AI Model Assistant
• Arduino Library for XIAO

Troubleshooting

Q1: Why can't I see the recognised result on the port after I successfully upload the code of Demo1 using XIAO ESP32C3?

If you are using Arduino IDE version 1.xx, please re-open the serial monitor after pressing Reset on C3 to see the result. If you are using Arduino version 2.xx, the message will be refreshed in the serial monitor after pressing the Reset button.

Tech Support & Product Discussion

Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.