Getting Started

This page records the hardware, software environment, ROS workspace assumptions, and preparation requirements for the FYP robotics navigation system.

This section does not describe the launch sequence yet. Launch commands and runtime order will be documented separately after the required environment and project assumptions are clear.

1. Hardware Used

The project is built around a mobile robot setup with lidar, RGB-D camera input, embedded computing, and a Unitree robot platform.

The hardware used in this FYP includes:

• NVIDIA Jetson Xavier NX 16GB
• Unitree Go2 EDU
• Livox MID-360 lidar
• Intel RealSense Depth Camera D455

The Jetson Xavier NX 16GB is used as the embedded computing platform. The Livox MID-360 provides lidar input for mapping and localization. The Intel RealSense D455 provides camera input for the FAST-LIVO2 workflow. The Unitree Go2 EDU is the robot base used for deployment and integration.

2. Software Environment

The project is intended for a ROS 1 environment.

Confirmed software environment:

• Ubuntu 20.04
• ROS Noetic
• catkin-based ROS workspace
• PCL 1.10 compatibility considerations
• C++17 build support for packages that require newer PCL compatibility

The most important compatibility rule is to keep Ubuntu and ROS consistent. This project should be reproduced on Ubuntu 20.04 with ROS Noetic before trying other operating system or ROS versions.

3. Install Intel RealSense SDK

The Intel RealSense D455 requires the RealSense SDK before the ROS wrapper can access the camera reliably.

Use Intel’s official SDK repository as the primary reference:

• Intel RealSense SDK 2.0 / librealsense
• Intel RealSense ROS wrapper

For this project, the required result is:

• the D455 is detected by the operating system
• RealSense SDK tools can read color and depth streams
• the realsense2_camera ROS package is available in the ROS environment
• the camera can be accessed from ROS Noetic on Ubuntu 20.04

On Jetson, RealSense installation can be more sensitive than on a desktop PC because of kernel, USB, JetPack, and ARM64 package differences. For the Jetson Xavier NX 16GB, building librealsense from source is the safer and more reproducible route.

Install build dependencies:

sudo apt update
sudo apt install -y \
  git cmake build-essential \
  libssl-dev libusb-1.0-0-dev libudev-dev pkg-config \
  libgtk-3-dev libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev

Clone the RealSense SDK:

cd ~
git clone https://github.com/IntelRealSense/librealsense.git
cd librealsense

Install udev rules so the camera can be accessed without manual device permission fixes:

./scripts/setup_udev_rules.sh

Build and install the SDK:

mkdir build
cd build
cmake .. \
  -DBUILD_EXAMPLES=true \
  -DCMAKE_BUILD_TYPE=Release \
  -DFORCE_RSUSB_BACKEND=true
make -j$(nproc)
sudo make install

The FORCE_RSUSB_BACKEND=true option avoids kernel patching and is usually easier to reproduce on Jetson. If a native backend is required, follow the Jetson-specific RealSense guide and use the L4T kernel patch workflow instead.

Reconnect the D455 and verify the SDK:

realsense-viewer

If realsense-viewer cannot open the D455, fix the SDK / USB / permission issue before moving to the ROS wrapper.

3.1 Install RealSense ROS Wrapper

For this FYP, the RealSense ROS wrapper was installed directly from the ROS Noetic apt package:

sudo apt update
sudo apt install -y ros-noetic-realsense2-camera ros-noetic-realsense2-description

Verify that ROS can find the wrapper:

rospack find realsense2_camera

The expected result is a valid package path instead of a package not found error.

3.1.1 Fallback: Build From Source

If the apt package is unavailable or incompatible with the target Jetson setup, build the wrapper from source:

mkdir -p ~/realsense_ws/src
cd ~/realsense_ws/src
git clone https://github.com/IntelRealSense/realsense-ros.git -b ros1-legacy
cd ~/realsense_ws
rosdep install --from-paths src --ignore-src -r -y
catkin_make -DCATKIN_ENABLE_TESTING=False -DCMAKE_BUILD_TYPE=Release
source devel/setup.bash

At this stage, the goal is only to confirm that the D455 SDK and ROS wrapper are installed correctly. The project launch sequence is documented separately.

4. Install Livox SDK and ROS Driver

The Livox MID-360 requires Livox SDK support and the ROS driver before point cloud data can be used by the mapping pipeline.

Use the official Livox repositories as the primary reference:

• Livox SDK2
• Livox ROS Driver 2

For this project, the required result is:

• Livox SDK2 is built and installed on the target machine
• livox_ros_driver2 is available in the ROS workspace
• MID-360 network configuration is correct for the Jetson
• ROS can receive Livox point cloud messages after the driver is configured

The repository already contains livox_ros_driver2, but the system-level SDK and network setup still need to be correct on the target machine.

4.1 Install Livox SDK2

Install the required build tools:

sudo apt update
sudo apt install -y git cmake build-essential

Clone, build, and install Livox SDK2:

cd ~
git clone https://github.com/Livox-SDK/Livox-SDK2.git
cd Livox-SDK2
mkdir build
cd build
cmake ..
make -j$(nproc)
sudo make install

Livox SDK2 installs its libraries and headers under system locations such as /usr/local/lib and /usr/local/include. If the system cannot find the installed library later, refresh the dynamic linker cache:

sudo ldconfig

4.2 Prepare Livox ROS Driver 2

This FYP repository already includes livox_ros_driver2, so do not clone another copy into the same workspace unless the package is missing.

This project is built by preparing livox_ros_driver2 first and then compiling the remaining workspace.

The checked-in build.sh script assumes the driver is located under a conventional catkin layout such as:

catkin_ws/src/livox_ros_driver2

Because the GitHub FYP repository contains package sources that belong under src/, the correct local layout is:

~/FYP_ws
~/FYP_ws/src
~/FYP_ws/src/fyp_src/livox_ros_driver2

In practice, your working method is to keep only livox_ros_driver2 in the package tree for the first pass, run its ROS1 build script, and then restore the other packages before the full workspace build.

The exact file-moving sequence is documented in Section 7. The essential Livox preparation command is:

cd ~/FYP_ws/src/fyp_src/livox_ros_driver2
source /opt/ros/noetic/setup.bash
./build.sh ROS1

The important part is still the order:

install Livox SDK2
-> create the catkin workspace
-> place the repository packages under src
-> keep only livox_ros_driver2 in the package tree
-> run ./build.sh ROS1
-> restore the other packages
-> catkin_make the full workspace
-> source devel/setup.bash

4.3 Configure MID-360 Network

The MID-360 communicates over Ethernet, so the Jetson network interface must be configured before ROS can receive data.

For this setup, the MID-360 IP follows the 192.168.1.1XX pattern, where XX comes from the last two digits of the device SN code. For example, if the SN code ends with 56, the MID-360 IP should be treated as:

192.168.1.156

After identifying the MID-360 IP, configure the Jetson Ethernet interface to the same subnet. In the current repository configuration, the Jetson host IP is set to:

192.168.1.222

Check the Jetson Ethernet interface name:

ip addr

Then assign an address in the same subnet:

sudo ip addr add 192.168.1.222/24 dev <ethernet_interface>

Replace <ethernet_interface> with the actual Ethernet interface name on the Jetson, such as eth0.

Test connectivity:

ping <MID360_IP>

Do not continue until the Jetson can ping the MID-360.

Check the Livox driver configuration files and make sure the host IP and LiDAR IP match the actual network setup:

livox_ros_driver2/config/

In this repository, the checked-in MID-360 driver configuration already has a concrete JSON structure. The important fields are the MID-360 IP under lidar_configs, the Jetson host IPs under host_net_info, and the data ports:

{
  "lidar_summary_info": {
    "lidar_type": 8
  },
  "MID360": {
    "lidar_net_info": {
      "cmd_data_port": 56100,
      "push_msg_port": 56200,
      "point_data_port": 56300,
      "imu_data_port": 56400,
      "log_data_port": 56500
    },
    "host_net_info": {
      "cmd_data_ip": "192.168.1.222",
      "cmd_data_port": 56101,
      "push_msg_ip": "192.168.1.222",
      "push_msg_port": 56201,
      "point_data_ip": "192.168.1.222",
      "point_data_port": 56301,
      "imu_data_ip": "192.168.1.222",
      "imu_data_port": 56401,
      "log_data_ip": "",
      "log_data_port": 56501
    }
  },
  "lidar_configs": [
    {
      "ip": "192.168.1.156",
      "pcl_data_type": 1,
      "pattern_mode": 0,
      "extrinsic_parameter": {
        "roll": 0.0,
        "pitch": 0.0,
        "yaw": 0.0,
        "x": 0,
        "y": 0,
        "z": 0
      }
    }
  ]
}

In this example:

• 192.168.1.156 is the MID-360 IP calculated from the SN code.
• 192.168.1.222 is the Jetson Ethernet IP on the same subnet.
• the *_ip fields inside host_net_info must match the Jetson Ethernet interface used to connect to the MID-360.
• the ip field inside lidar_configs must match the actual MID-360 IP.
• The MID-360 port values should stay consistent with the Livox driver configuration.

For this FYP, the required result is that the Jetson and MID-360 are on the same subnet and the Livox driver configuration matches the real device.

5. Clone The FYP Repository

This GitHub repository contains the packages that belong under the src/ directory of a catkin workspace. It is not the full workspace root by itself.

Create a workspace first:

mkdir -p ~/FYP_ws/src
cd ~/FYP_ws
ln -s /opt/ros/noetic/share/catkin/cmake/toplevel.cmake CMakeLists.txt

Then clone the FYP repository into src/:

cd ~/FYP_ws/src
git clone https://github.com/FFraankk/FYP.git fyp_src

At this stage, ~/FYP_ws/src/fyp_src contains the package directories that belong inside the workspace src/ tree.

The important idea is:

~/FYP_ws           -> catkin workspace root
~/FYP_ws/src       -> catkin source directory
~/FYP_ws/src/fyp_src/* -> the packages from this repository

6. Install ROS Dependencies

Before building, install the ROS packages required by the workspace.

Start with the standard ROS dependency workflow:

cd ~/FYP_ws
rosdep update
rosdep install --from-paths src --ignore-src -r -y

Some dependencies may still need to be installed manually, especially sensor SDKs, vendor drivers, or packages built from source. If rosdep cannot resolve a package, check whether that package is already included in the workspace or must be installed separately.

7. Build The Workspace

The repository packages should be brought into the workspace in two stages.

7.1 Build livox_ros_driver2 First

Your reported working workflow is:

• first place only livox_ros_driver2 under src/
• run ./build.sh ROS1
• then bring the other packages into src/
• finally run catkin_make

The safest way to follow that workflow is:

cd ~/FYP_ws/src

mkdir -p _staging

mv fyp_src/FAST-LIVO2 _staging/
mv fyp_src/fast_gicp _staging/
mv fyp_src/fastlivo2_bringup _staging/
mv fyp_src/hdl_global_localization _staging/
mv fyp_src/hdl_localization _staging/
mv fyp_src/navigation _staging/
mv fyp_src/navigation_manager _staging/
mv fyp_src/ndt_omp _staging/
mv fyp_src/robot_bringup _staging/
mv fyp_src/rpg_vikit _staging/
mv fyp_src/unitree _staging/

After that, only livox_ros_driver2 remains in the package tree that will be compiled by its own build script.

Run the Livox ROS1 build:

cd ~/FYP_ws/src/fyp_src/livox_ros_driver2
source /opt/ros/noetic/setup.bash
./build.sh ROS1

7.2 Restore The Remaining Packages

After livox_ros_driver2 has been prepared, move the remaining packages back:

cd ~/FYP_ws/src

mv _staging/FAST-LIVO2 fyp_src/
mv _staging/fast_gicp fyp_src/
mv _staging/fastlivo2_bringup fyp_src/
mv _staging/hdl_global_localization fyp_src/
mv _staging/hdl_localization fyp_src/
mv _staging/navigation fyp_src/
mv _staging/navigation_manager fyp_src/
mv _staging/ndt_omp fyp_src/
mv _staging/robot_bringup fyp_src/
mv _staging/rpg_vikit fyp_src/
mv _staging/unitree fyp_src/

rmdir _staging

7.3 Build The Full Workspace

After the SDKs, ROS dependencies, and Livox ROS1 preparation step are complete, build the remaining workspace:

cd ~/FYP_ws
catkin_make
source devel/setup.bash

After sourcing the workspace, confirm that ROS can find the project packages:

rospack find navigation
rospack find navigation_manager
rospack find robot_bringup
rospack find unitree

The build should be fixed before any runtime debugging. If the workspace does not compile, do not continue to localization or navigation testing yet.

The expected build order for this FYP is:

install Livox SDK2
-> create a catkin workspace
-> clone the FYP repository into src
-> temporarily keep only livox_ros_driver2 in the package tree
-> run livox_ros_driver2/build.sh ROS1
-> restore the remaining packages
-> run catkin_make for the full FYP workspace
-> source devel/setup.bash

8. Camera Calibration

This project uses LiDAR-camera calibration between the Livox MID-360 and the Intel RealSense D455. The calibration result is the extrinsic relationship between the LiDAR frame and the camera frame.

Temporarily Not Fully Confirmed

The workflow below matches the current repository more closely than earlier versions, but the final calibration conversion procedure should still be verified on the target robot before treating it as the only correct method.

In particular, do not blindly copy the raw output of livox_camera_calib into FAST-LIVO2 without checking the camera frame convention used by the calibration output.

The tool used for this FYP is:

• hku-mars/livox_camera_calib

The setup here uses one RealSense D455 camera. The calibration tool can also work with multi-scene data, but this page focuses on the single-camera workflow used for this project.

8.1 Create A Calibration Workspace

Keep the calibration tool separate from the main FYP workspace:

mkdir -p ~/livox_camera_calib_ws/src
cd ~/livox_camera_calib_ws/src
git clone https://github.com/hku-mars/livox_camera_calib.git

Keeping it under the home directory avoids mixing calibration-only dependencies with the main FYP workspace.

8.2 Install Calibration Dependencies

Install the ROS and system dependencies:

sudo apt update
sudo apt install -y \
  ros-noetic-cv-bridge \
  ros-noetic-pcl-conversions \
  libeigen3-dev \
  libpcl-dev

Install Ceres Solver from source. Do not rely on libceres-dev for this setup.

Use the official Ceres installation guide as the reference:

• Ceres Solver Installation

Install Ceres dependencies:

sudo apt update
sudo apt install -y \
  cmake \
  libgoogle-glog-dev \
  libgflags-dev \
  libatlas-base-dev \
  libsuitesparse-dev

Build and install Ceres:

cd ~
git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver
mkdir build
cd build
cmake ..
make -j$(nproc)
sudo make install
sudo ldconfig

After installation, Ceres headers and libraries should be available to the calibration workspace.

8.3 Build livox_camera_calib

Build the calibration workspace:

cd ~/livox_camera_calib_ws
catkin_make
source devel/setup.bash

If the build cannot find Eigen, Ceres, PCL, cv_bridge, or pcl_conversions, fix the missing dependency before continuing.

8.4 Prepare The PCD File

The calibration tool needs a point cloud file from the Livox MID-360.

For this FYP, the .pcd file is generated through FAST-LIVO2 instead of manually exporting a random point cloud. This keeps the point cloud consistent with the same LiDAR-camera setup used by the robot.

In the FAST-LIVO2 MID-360 configuration file, enable PCD saving:

pcd_save:
  pcd_save_en: true
  colmap_output_en: false # need to set interval = -1
  filter_size_pcd: 0.15
  interval: -1
  # how many LiDAR frames saved in each pcd file;
  # -1 : all frames will be saved in ONE pcd file, may lead to memory crash when having too much frames.

The original value may be:

pcd_save:
  pcd_save_en: false

Change it to true only when collecting calibration data or saving a map. After the required .pcd file is generated, it can be changed back to false to avoid unnecessary disk usage.

Use FAST-LIVO2 to record a calibration scene. The scene should include clear structures such as wall corners, door frames, table edges, or other strong geometric edges. These edges are important because the calibration quality is checked by comparing projected LiDAR edges against camera image edges.

After FAST-LIVO2 finishes saving the point cloud, copy the selected .pcd file into the calibration workspace:

mkdir -p ~/livox_camera_calib_ws/data/calib_scene
cp /path/to/generated/file.pcd ~/livox_camera_calib_ws/data/calib_scene/0.pcd

The calibration scene should contain clear edge information. The livox_camera_calib method is targetless, so the quality of scene edges matters.

Example final file location:

~/livox_camera_calib_ws/data/calib_scene/0.pcd

8.5 Prepare The Image File

The calibration tool also needs an image from the Intel RealSense D455.

The current FAST-LIVO2 repository does not expose an image_save block like the one used earlier in these notes, so the image should be saved directly from the RealSense side instead of assuming FAST-LIVO2 will write image files automatically.

Use realsense-viewer or another capture method from the D455 color stream, and save a clear image from the same physical scene used for the .pcd file. A practical workflow is:

1. Keep the robot and sensors mounted exactly as they are during normal operation.
2. Run the sensors in the same scene used for calibration.
3. Save the .pcd file from FAST-LIVO2.
4. Save a color image from the D455 that shows the same walls, corners, door frames, and structural edges.

Then copy the chosen image into the calibration workspace:

cp /path/to/generated/image.png ~/livox_camera_calib_ws/data/calib_scene/0.png

The image should contain the same walls, corners, edges, or structural features as the .pcd file. Avoid blurry images, motion blur, or images from a different camera resolution.

Example final file location:

~/livox_camera_calib_ws/data/calib_scene/0.png

For this project, use one RealSense D455 camera for the single-camera calibration path. The tool also supports multi-scene calibration if multiple image / point cloud pairs are prepared, but that is not the main setup used here.

8.6 Get Camera Intrinsics From RealSense Viewer

Use realsense-viewer to read the D455 camera intrinsics and distortion parameters.

Open the viewer:

realsense-viewer

Then:

1. Select the Intel RealSense D455.
2. Enable the same camera stream used for calibration.
3. Set the same resolution as the saved calibration image.
4. Read the camera intrinsics from the stream information.
5. Record the intrinsic matrix and distortion coefficients.

The resolution must match. For example, if the image used by livox_camera_calib is captured at 1280x720, the intrinsics must also be read for 1280x720. Do not use intrinsics from another resolution such as 640x480.

Record these values:

fx, fy, cx, cy
distortion coefficients
image width
image height

These values are required in the calibration YAML file.

The current FAST-LIVO2 repository already contains one D455 camera file:

cam_model: Pinhole
cam_width: 1280
cam_height: 720
cam_fx: 648.752624511719
cam_fy: 648.752624511719
cam_cx: 639.716735839844
cam_cy: 362.857482910156
cam_d0: 0.0
cam_d1: 0.0
cam_d2: 0.0
cam_d3: 0.0

These values come from the current FAST-LIVO2/config/camera_D455.yaml. If the deployed camera or resolution changes, update this file and keep the calibration data consistent with the same resolution.

8.7 Edit calib.yaml

Open the calibration configuration:

cd ~/livox_camera_calib_ws/src/livox_camera_calib/config

For single-camera calibration, edit calib.yaml.

The configuration should point to:

• the .pcd file from the MID-360
• the image file from the D455
• the D455 intrinsic matrix
• the D455 distortion coefficients
• the image width and height
• the initial extrinsic estimate if required by the config

The exact key names should follow the calib.yaml file in the cloned repository. The important rule is that file paths, camera intrinsics, distortion parameters, and image resolution must describe the same captured dataset.

The file paths should point to the files prepared above:

~/livox_camera_calib_ws/data/calib_scene/0.pcd
~/livox_camera_calib_ws/data/calib_scene/0.png

8.8 Run Calibration

Source the calibration workspace:

source ~/livox_camera_calib_ws/devel/setup.bash

Run the single-scene calibration:

roslaunch livox_camera_calib calib.launch

For multi-scene calibration, the repository also provides multi_calib.yaml and multi_calib.launch, but that is not the main workflow documented here.

8.9 Save The Calibration Result

After calibration, save the resulting LiDAR-camera extrinsic parameters.

The calibration result must be copied back into the FAST-LIVO2 configuration. In the current repository, the relevant FAST-LIVO2 section is:

extrin_calib:
  extrinsic_T: [0.011, 0.02329, -0.04412]
  extrinsic_R: [1, 0, 0, 0, 1, 0, 0, 0, 1]
  Rcl: [-0.00946018, -0.999897, 0.0108095,
         0.0148908, -0.0109497, -0.999829,
         0.999844, -0.0092976, 0.0149929]
  Pcl: [-0.0366384, -0.0507661, 0.0366028]

In this section:

• extrinsic_T and extrinsic_R are the LiDAR-IMU extrinsic parameters.
• Rcl and Pcl are the LiDAR-camera extrinsic parameters used by FAST-LIVO2.
• The values above are the parameters currently checked into this repository.

The FAST-LIVO2 source code reads Rcl and Pcl and passes them into a function named setLidarToCameraExtrinsic, so the current code path treats them as LiDAR-to-camera extrinsics.

There are two checks before copying values into FAST-LIVO2:

1. Check transform direction.
2. Check camera frame convention.

If the calibration output is already LiDAR-to-camera and uses the same camera frame convention as FAST-LIVO2, it can be written into Rcl and Pcl directly.

If the only difference is transform direction, and the calibration output is LiDAR-to-camera:

P_camera = R_lidar_to_camera * P_lidar + t_lidar_to_camera

and FAST-LIVO2 needs camera-to-LiDAR, convert it with:

R_camera_to_lidar = R_lidar_to_camera^T
t_camera_to_lidar = -R_lidar_to_camera^T * t_lidar_to_camera

This direction-only conversion can be done with Python:

import numpy as np

R_lidar_to_camera = np.array([
    [r11, r12, r13],
    [r21, r22, r23],
    [r31, r32, r33],
])

t_lidar_to_camera = np.array([[tx], [ty], [tz]])

R_camera_to_lidar = R_lidar_to_camera.T
t_camera_to_lidar = -R_camera_to_lidar @ t_lidar_to_camera

print("Rcl:")
print(R_camera_to_lidar)

print("Pcl:")
print(t_camera_to_lidar.reshape(-1))

However, this inverse transform is not always the full conversion. If livox_camera_calib outputs a camera optical frame transform but FAST-LIVO2 expects another camera frame convention, an additional fixed axis conversion is required.

For example, RealSense commonly exposes frames such as:

camera_link
camera_color_frame
camera_color_optical_frame

The calibration result and the FAST-LIVO2 configuration must refer to the same camera frame convention. If one side uses camera_color_optical_frame and the other side expects a different camera body frame, the result needs an extra frame conversion before being written into Rcl and Pcl.

The checked-in source now makes the LiDAR-to-camera intent of Rcl/Pcl much clearer, but the final calibration still has to be validated visually on the real dataset before replacing the repository values.

Check the calibration quality visually. The result is acceptable when LiDAR points projected onto the camera image align with real image edges. In practice, look at wall corners, door frames, table edges, and other clear structures:

• If the colored LiDAR edge overlaps the camera image edge, the calibration is likely usable.
• If the colored LiDAR edge is consistently shifted away from the camera edge, the extrinsic result, camera intrinsics, image resolution, or data pairing is wrong.

Keep a record of:

• the .pcd file used
• the image file used
• image resolution
• camera intrinsics
• distortion coefficients
• final LiDAR-camera extrinsic result

This makes the calibration reproducible and prevents confusion when the camera resolution or sensor mounting changes later.

9. Build Verification

After building the workspace, confirm the basic environment before trying to run the full system:

echo $ROS_DISTRO
rospack find livox_ros_driver2
rospack find realsense2_camera
rospack find navigation
rospack find navigation_manager

Expected result:

• ROS reports noetic
• project packages can be found by rospack
• RealSense and Livox ROS packages are available
• the workspace can be sourced without errors

This page stops at the build and preparation stage. Runtime launch order, map loading, localization startup, and navigation startup are documented separately.