Common Operations

Mastering the following operations will help you use the Exoskeleton Teleoperation Device more efficiently.

Starting the HMI Interface

ROBOT-CONTROL-HMI (hereinafter referred to as HMI) is the graphical user interface for controlling and monitoring the robot system. It serves as the core tool for interacting with the robot when using the Exoskeleton Teleoperation Device. The HMI consists of two separate programs: the user-side program and the robot-side program. Both must be started before use, following the procedures below.

Starting the User-Side Program

1. Check that Microsoft Edge is installed and set as the default browser.

2. On the industrial PC, open a terminal and enter the following commands.

   cd ~
   bash ~/user_docker_start.sh

3. After the script executes, the Microsoft Edge browser will launch the HMI interface automatically. If it does not open automatically, manually enter http://127.0.0.1:4090 in the browser’s address bar.

   

   Note

   • The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems. Therefore, when switching between starting the Exoskeleton Device, VR Control Device, and Active Exoskeleton Device, you must close all controls running under the previous device first. Ensure all buttons in the Robot Control panel are blue before switching devices.

   • To close the HMI interface, follow the procedure below:

     1. User-side: First, use Ctrl+C to terminate the user-side program. Then, close the corresponding terminal. Finally, close the Microsoft Edge browser.
     2. Robot-side: On the robot terminal, execute the sudo reboot command, enter the current user’s password, and wait 30 seconds for the reboot to complete.

   Tip

   To maintain communication, the user-side program requests and updates robot status data approximately every 10 seconds. With both programs running, periodic robot status retrieval may fail only in the following two scenarios:

   • HMI interface closed: The user-side program cannot retrieve robot status via the browser if the interface is manually closed with no other interface active.
   • HMI interface suspended by browser background: In multi-tab browsing, if the HMI interface loses focus, the browser may automatically suspend it, preventing the user-side program from sending status requests.

Starting the Robot-Side Program

1. On the industrial PC, open a new terminal window and remotely connect to the robot terminal using the ssh command.

   ssh <username>@<robot_IP_address>

   Note

   Replace <username> and <robot_IP_address> with the actual username and IP address.

2. In the successfully connected terminal, enter the following commands.

   cd ~
   bash ~/robot_docker_start.sh

   Note

   • To close the HMI interface, follow the procedure below:

     1. User-side: First, use Ctrl+C to terminate the user-side program. Then, close the corresponding terminal. Finally, close the Microsoft Edge browser.
     2. Robot-side: On the robot terminal, execute the sudo reboot command, enter the current user’s password, and wait 30 seconds for the reboot to complete.

After Starting the HMI interface, you can use the exoskeleton device to control the robot.

Using the Exoskeleton Device

Starting the Exoskeleton Control Device

1. After starting the HMI interface, click the Start Exoskeleton Device button. Once started successfully, the button will turn red and the text will change to Stop Exoskeleton Device.

   

   Note

   • The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems. Therefore, when switching between starting the Exoskeleton Device, VR Control Device, and Active Exoskeleton Device, you must close all controls running under the previous device first. Ensure all buttons in the Robot Control panel are blue before switching devices.
   • If the robot malfunctions or the robotic arm moves abnormally or becomes uncontrollable, click the Stop Exoskeleton Device button immediately. The robot will emergency stop and cease response.

2. The AI2-HMI window will pop up. This window displays real-time teleoperation data. Do not close this window during operation.

   

   Note

   • If there is only one monitor, minimize this window to ensure it does not affect the operation of the main HMI interface. If there are multiple monitors, drag this window to a non-main monitor for easy observation of data changes.
   • This window does not automatically exit when the HMI interface is closed. It can only be closed by clicking its own close button or the Stop Exoskeleton Device button on the HMI interface.
   • To prevent display abnormalities or data conflicts caused by multiple windows running simultaneously, ensure only one HMI interface and one corresponding AI2-HMI window are open before each startup.

Starting the Camera and the Video Stream

1. After ensuring that the Exoskeleton Device has been successfully started, click the Start RealSense Camera button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the RealSense camera will enter a self-check state. The Start RealSense Camera blue button will change to a red Stop RealSense Camera button, and the RealSense Camera Status in the System Status panel will display Running.

   Note

   This feature is core to data acquisition. If the RealSense camera is not started, data acquisition cannot be performed.

   

4. Click the Start Video Stream button in the upper-right corner of the interface.

   

5. A window as shown below will pop up. This window will close automatically.

   

6. The video stream playback area will display the real-time video stream, and the Video Stream Status in the System Status panel will display Running.

   

Starting the Robotic Arm

1. After ensuring that the camera and the video stream have been successfully started, click the Start Robotic Arm button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the robotic arm will enter a self-check state. The Start Robotic Arm blue button will change to a red Stop Robotic Arm button, and the Robotic Arm Status in the System Status panel will display Running.

   

   Note

   After the robotic arm is successfully started, you need to start the follow-up mode and ensure the robotic arm moves normally before using the exoskeleton device to control the robotic arm.

Starting Follow-Up

1. After ensuring the robotic arm has been started, click the Start Slow Follow-Up button in the Robot Control panel.

   

2. A prompt message as shown below will appear. The robotic arm will slowly move to the pose of the teleoperation arm. After confirming that the pose is normal, click the OK button to switch to fast follow-up.

   Tip

   In slow follow-up mode, the robotic arm follows the movement of the teleoperation arm at a slow speed, allowing the operator to observe the motion status of the robotic arm. After confirming that the motion pose is consistent with the teleoperation arm and there are no abnormalities, switch to fast follow-up for a smoother operation experience.

   

3. Click the OK button in the pop-up window below to switch to Fast Follow-Up.

   

4. Once started successfully, the blue Start Slow Follow-Up button will change to the red Stop Fast Follow-Up button. Also confirm that the follow-up status in the System Status panel is Fast Follow-Up.

Starting the Torso

1. After ensuring that the camera and the video stream have been successfully started, click the Start Torso button in the Robot Control panel.

   

2. A prompt message as shown below will appear; please wait patiently.

   

3. Upon successful startup, the torso will enter a self-check state. The Start Torso blue button will change to a red Stop Torso button, and the Torso Status in the System Status panel will display Running.

   

Starting the Chassis

1. Rotate the chassis knob to Auto Mode and disconnect the charging cable.

2. After ensuring that the camera and the video stream have been successfully started, click the Start Chassis button in the Robot Control panel.

   

3. A prompt message as shown below will appear; please wait patiently.

   

4. Upon successful startup, the chassis will enter a self-check state. The Start Chassis blue button will change to a red Stop Chassis button, and the Chassis Status in the System Status panel will display Running.

   

Using the Controller

After starting the exoskeleton device, you can use the controller to control the robot.

1. Ensure that the controller receiver is connected to the USB port of the industrial PC and that the controller has sufficient battery power.

   Note

   • Do not connect the controller to the industrial PC using a wired connection.

   • The controller charging specification is 5V/1-2A. Fast charging is not supported. Do not use a fast charger with a power output exceeding 66W. The charging status indicators are described below.

     

2. When using the controller for the first time, the controller mode must be configured to Android Mode -> Receiver Mode.

   Configuration method: Press and hold the HOME button for 3 seconds to enter the mode selection interface, then select Android Mode -> Receiver.

3. On the industrial PC, open a terminal and enter the following command:

   ros2 run joy joy_node

   Note

   Do not close this terminal window while using the controller, otherwise the controller will become unusable.

4. You can control the neck, waist-legs, and chassis respectively through the controller buttons.

   

   Operation	Description
   Move Chassis	Move forward/backward: LB + Left Stick Y Turn left/right: LB + Left Stick X
   Control Neck	Turn head left/right: Right Stick X Tilt head up/down: Right Stick Y
   Control Waist/Legs	Raise waist/legs: RT + A Lower waist/legs: RT + X Rotate waist right: LT + A Rotate waist left: LT + X Move waist forward/backward: Left Stick Y

Data Acquisition

After starting the exoskeleton device, you can acquire data from the Exoskeleton Teleoperation Device using the Robot Data Acquisition function in the HMI interface.

Note

The Exoskeleton Device, VR Control Device, and Active Exoskeleton Device are three independent systems; do not mix them.

Creating a Task

1. In the Robot Data Acquisition panel, enter a Task ID and click the Create Task button.

   Note

   The Task ID can only contain numeric characters. Do not use illegal characters.

   

   • If it is a new task, a prompt message as shown below will pop up on the page.

     

     Click the blank area outside the pop-up window to close it.

   • If the Task ID already exists, a prompt message as shown below will pop up on the page. Click the OK button to resume the existing task.

     

     Click the blank area outside the pop-up window to close it.

     

2. Click the Send Task button. A pop-up prompt will appear after successful sending.

   Note

   Before clicking the Send Task button, ensure that the camera and robotic arm have been started. Otherwise, the button will not respond.

   

3. At this time, a prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.

   

Starting Data Acquisition

After successfully creating a task, you can start acquiring data.

1. Use any of the following three methods to start data acquisition.

   • Click the Start Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+3 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the middle button of the foot pedal.

2. Once data acquisition starts, the Acquisition Status in the Robot Data Acquisition Status panel will change from Waiting to Start to Acquiring, and the acquisition timer will begin counting.

   

Canceling Acquisition

During the acquisition process, you can cancel acquisition at any time.

1. Use any of the following three methods to cancel data acquisition.

   • Click the Cancel Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+4 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the right button of the foot pedal.

2. After acquisition is canceled, the Acquisition Status in the Robot Data Acquisition Status panel will change from Acquiring to Waiting to Start. The running time will stop counting, and the acquisition time will reset to zero.

   

Ending Acquisition

After data acquisition is complete, you need to manually end the acquisition process.

1. Use any of the following three methods to end data acquisition.

   • Click the End Acquisition button. A prompt message as shown below will pop up on the page. Click the blank area outside the pop-up window to close it.
   • Press the Ctrl+3 keyboard shortcut. (Note: Only keep the HMI page open in the browser; do not open any other pages. The existing shortcut key conflicts with the browser’s tab-switching functionality.)
   • Press the middle button of the foot pedal.

2. After acquisition ends, the Acquisition Status in the Robot Data Acquisition Status panel will change from Acquiring to Waiting to Start. The running time will stop counting, the acquisition time will reset to zero, one data entry will be written to disk, and the acquisition entry count will increment by 1.

   

Compressing Data

After data acquisition, you need to compress data for easier subsequent upload and use. Before proceeding, ensure you have completed at least one data acquisition.

1. Click the Data Compression button.

   

2. In the pop-up window below, enter a task name for compression, then click Start Compression.

   

3. Wait for the compression process to finish. Click the blank area outside the window to close it.

   

Uploading Data

After data acquisition and compression, you can use the Data Upload function to upload data. Before proceeding, ensure you have completed at least one data acquisition and compression.

1. Click the Data Upload button.

   Note

   During upload, keep the HMI interface active and do not switch to other browser tabs, otherwise the upload may fail.

   

2. In the pop-up window below, please wait patiently for the data upload to complete and confirm that the number of uploaded files is correct. If you need to cancel the upload, click the Cancel Upload button.

   Note

   If the upload does not complete successfully, you can close the pop-up window and try clicking the Data Upload button again to continue the upload.

   

Using the Foot Pedal

Button	Description
Middle Button	Start or stop data acquisition
Right Button	Cancel acquisition
Left Button	Add markers at different phases of long-duration tasks to facilitate data segmentation and training verification.

Zero Calibration

The zero calibration operation is only required after disassembling or assembling the structure. When the pose of the robotic arm is inconsistent with that of the teleoperation arm, and no over-limit is displayed on the AI2-HMI interface, it may be a zero position abnormality. Please contact us for assistance.