Common Operations

Mastering the following operations will help you use the VR Teleoperation Device more efficiently.

Starting the HMI Interface

ROBOT-CONTROL-HMI (hereinafter referred to as HMI) is the graphical user interface for controlling and monitoring the robot system. It serves as the core tool for interacting with the robot when using the VR Teleoperation Device. The HMI consists of two separate programs: the user-side program and the robot-side program. Both must be started before use, following the procedures below.

Starting the User-Side Program

1. Check that Microsoft Edge is installed and set as the default browser.

2. On the industrial PC, open a terminal and enter the following commands.

   cd ~
   bash ~/user_docker_start.sh

3. After the script executes, the Microsoft Edge browser will launch the HMI interface automatically. If it does not open automatically, manually enter http://127.0.0.1:4090 in the browser’s address bar.

   

   Note

   • The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems. Therefore, when switching between starting the Exoskeleton Device, VR Control Device, and Active Exoskeleton Device, you must close all controls running under the previous device first. Ensure all buttons in the Robot Control panel are blue before switching devices.

   • To close the HMI interface, follow the procedure below:

     1. User-side: First, use Ctrl+C to terminate the user-side program. Then, close the corresponding terminal. Finally, close the Microsoft Edge browser.
     2. Robot-side: On the robot terminal, execute the sudo reboot command, enter the current user’s password, and wait 30 seconds for the reboot to complete.

Starting the Robot-Side Program

1. On the industrial PC, open a new terminal window and remotely connect to the robot terminal using the ssh command.

   ssh <username>@<robot_IP_address>

   Note

   Replace <username> and <robot_IP_address> with the actual username and IP address.

2. In the successfully connected terminal, enter the following commands.

   cd ~
   bash ~/robot_docker_start.sh

   Note

   • To close the HMI interface, follow the procedure below:

     1. User-side: First, use Ctrl+C to terminate the user-side program. Then, close the corresponding terminal. Finally, close the Microsoft Edge browser.
     2. Robot-side: On the robot terminal, execute the sudo reboot command, enter the current user’s password, and wait 30 seconds for the reboot to complete.

After Starting the HMI interface, you can use the VR device to control the robot.

Using the VR Device

Prerequisites

Before using the VR device, ensure that the following prerequisites have been completed.

Tip

Clicks or confirmations in the steps below can be performed by pressing the index trigger button on the left or right controller.

1. Ensure the VR headset has sufficient battery power, is powered on, and connected to the industrial PC via a stable data cable. After holding the controllers with both hands, the view will display as follows. For first-time connection or reconnection after disconnection, Click the Notification button (bell icon) on the Tab bar to enter the Notification Center.

   

2. In the Notification Center, click USB detected to allow access to the headset via USB.

   

3. In the pop-up window shown below, click Allow to enable USB debugging.

   

4. If the above operations are correct, open a new terminal on the industrial PC and enter the following command. The display below indicates the headset is ready.

   user@user:~$ adb devices
   List of devices attached
   XXXXXXXXXXXXXX  device

   Note

   XXXXXXXXXXXXXX is the serial number of the headset, which may vary by device. Please refer to the actual displayed serial number.

5. Press the Menu button on the right controller to redisplay the Tab bar.

   

6. Click APP in the Tab bar to enter the headset control program.

   

7. When the interface shown below appears, the prerequisites have been completed.

   

Starting the VR Control Device

1. After ensuring that all prerequisites have been completed, click the Start VR Control Device button in the HMI interface. Upon successful startup, the button will turn red and the text will change to Stop VR Control Device.

   

   Note

   • The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems. Therefore, when switching between starting the Exoskeleton Device, VR Control Device, and Active Exoskeleton Device, you must close all controls running under the previous device first. Ensure all buttons in the Robot Control panel are blue before switching devices.
   • If the robot malfunctions or the robotic arm moves abnormally or becomes uncontrollable, click the Stop VR Control Device button immediately. The robot will emergency stop and cease response.

2. Check the System Status panel. If VR Device Status displays Data Normal, 3 entries, the startup is successful.

Starting the Camera and the Video Stream

1. After ensuring that the VR Control Device has been successfully started, click the Start RealSense Camera button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the RealSense camera will enter a self-check state. The Start RealSense Camera blue button will change to a red Stop RealSense Camera button, and the RealSense Camera Status in the System Status panel will display Running.

   Note

   This feature is core to data acquisition. If the RealSense camera is not started, data acquisition cannot be performed.

   

4. Click the Start Video Stream button in the upper-right corner of the interface.

   

5. A window as shown below will pop up. This window will close automatically.

   

6. The video stream playback area will display the real-time video stream, and the Video Stream Status in the System Status panel will display Running.

   

Starting the Robotic Arm

1. After ensuring that the camera and the video stream have been successfully started, click the Start Robotic Arm button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the robotic arm will enter a self-check state. The Start Robotic Arm blue button will change to a red Stop Robotic Arm button, and the Robotic Arm Status in the System Status panel will display Running.

   

Starting the Torso

1. After ensuring that the camera and the video stream have been successfully started, click the Start Torso button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the torso will enter a self-check state. The Start Torso blue button will change to a red Stop Torso button, and the Torso Status in the System Status panel will display Running.

   

Starting the Chassis

1. Rotate the chassis knob to Auto Mode and disconnect the charging cable.

2. After ensuring that the camera and the video stream have been successfully started, click the Start Chassis button in the Robot Control panel.

   

3. A prompt message as shown below will appear; please wait patiently.

   

4. Upon successful startup, the chassis will enter a self-check state. The Start Chassis blue button will change to a red Stop Chassis button, and the Chassis Status in the System Status panel will display Running.

   

Using the VR Controllers

After starting the VR Control Device, you can use the VR controllers to control the robot.

Operation	Description
Move Chassis	Move Forward: Left hand middle finger trigger + left joystick forward Move Backward: Left hand middle finger trigger + left joystick backward Turn Left: Right hand middle finger trigger + left joystick left Turn Right: Right hand middle finger trigger + left joystick right
Follow Robotic Arm End-Effector	Follow Left Arm End-Effector: Left hand index finger trigger Follow Right Arm End-Effector: Right hand index finger trigger
Control Gripper	Left Arm Gripper: X button Right Arm Gripper: A button
Control Neck	Yaw: Right controller joystick left/right Pitch: Right controller joystick up/down
Control Legs	Body Rise: Right hand middle finger trigger + X button Body Lower: Right hand middle finger trigger + A button
Control Waist	Waist Turn Left: Left hand middle finger trigger + X button Waist Turn Right: Left hand middle finger trigger + A button
Reset Robotic Arm	Press the left controller joystick

Data Acquisition

After starting the VR Control Device, you can acquire data from the VR Teleoperation Device using the Robot Data Acquisition function in the HMI interface.

Note

The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems; do not mix them.

Creating a Task

1. In the Robot Data Acquisition panel, enter a Task ID and click the Create Task button.

   Note

   The Task ID can only contain numeric characters. Do not use illegal characters.

   

   • If it is a new task, a prompt message as shown below will pop up on the page.

     

     Click the blank area outside the pop-up window to close it.

   • If the Task ID already exists, a prompt message as shown below will pop up on the page. Click the OK button to resume the existing task.

     

     Click the blank area outside the pop-up window to close it.

     

2. Click the Send Task button. A pop-up prompt will appear after successful sending.

   Note

   Before clicking the Send Task button, ensure that the camera and robotic arm have been started. Otherwise, the button will not respond.

   

3. At this time, a prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.

   

Starting Data Acquisition

After successfully creating a task, you can start acquiring data.

1. Use any of the following three methods to start data acquisition.

   • Click the Start Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+3 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the middle button of the foot pedal.

2. Once data acquisition starts, the Acquisition Status in the Robot Data Acquisition Status panel will change from Waiting to Start to Acquiring, and the acquisition timer will begin counting.

   

Canceling Acquisition

During the acquisition process, you can cancel acquisition at any time.

1. Use any of the following three methods to cancel data acquisition.

   • Click the Cancel Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+4 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the right button of the foot pedal.

2. After acquisition is canceled, the Acquisition Status in the Robot Data Acquisition Status panel will change from Acquiring to Waiting to Start. The running time will stop counting, and the acquisition time will reset to zero.

   

Ending Acquisition

After data acquisition is complete, you need to manually end the acquisition process.

1. Use any of the following three methods to end data acquisition.

   • Click the End Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+3 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the middle button of the foot pedal.

2. After acquisition ends, the Acquisition Status in the Robot Data Acquisition Status panel will change from Acquiring to Waiting to Start. The running time will stop counting, the acquisition time will reset to zero, one data entry will be written to disk, and the acquisition entry count will increment by 1.

   

Compressing Data

After data acquisition, you need to compress data for easier subsequent upload and use. Before proceeding, ensure you have completed at least one data acquisition.

1. Click the Data Compression button.

   

2. In the pop-up window below, enter a task name for compression, then click Start Compression.

   

3. Wait for the compression process to finish. Click the blank area outside the window to close it.

   

Uploading Data

After data acquisition and compression, you can use the Data Upload function to upload data. Before proceeding, ensure you have completed at least one data acquisition and compression.

1. Click the Data Upload button.

   Note

   During upload, keep the HMI interface active and do not switch to other browser tabs, otherwise the upload may fail.

   
2. In the pop-up window below, please wait patiently for the data upload to complete and confirm that the number of uploaded files is correct. If you need to cancel the upload, click the Cancel Upload button.

   Note

   If the upload does not complete successfully, you can close the pop-up window and try clicking the Data Upload button again to continue the upload.

   

Using the Foot Pedal

Button	Description
Middle Button	Start or stop data acquisition
Right Button	Cancel acquisition
Left Button	Add markers at different phases of long-duration tasks to facilitate data segmentation and training verification.