publications | Junlei Hu

2026

IJRR

Autonomous Robotic Exploration on Unknown Soft Object

Junlei Hu, Dominic Jones, Gerardo Loza Galindo, Shoudong Huang, and Pietro Valdastri

2026

The International Journal of Robotics Research (IJRR)

Abs HTML Website

Robotic exploration of unknown soft objects presents significant challenges for autonomous systems due to unpredictable deformations and shape changes during manipulation. To address this, we propose a framework that integrates topology-aware 3D reconstruction with a topology-guided motion planner, enabling the discovery and reconstruction of previously hidden or concave regions. This topology-aware 3D reconstruction employs a novel representation of deformable objects by combining Cylinder Čech Complexes with point clouds, enabling rapid tracking of significant topology changes and detection of non-manifold boundaries.The topology analysis and canonical reconstruction guide motion planning by optimising grasp points and planning trajectories to reveal previously unseen surfaces through two actions: turning over and stretching. We validated our algorithm through simulations and experiments using the \textitda Vinci Research Kit, demonstrating successful exploration with two or three manipulators. We showed it can fully explore surfaces of two everyday objects, a beanie and a rubber glove, and two cadaveric organs, a liver and a colon, within seven manipulations. Our method achieved a 45.6% improvement in 3D reconstruction accuracy compared to state-of-the-art point-cloud-based methods while also demonstrating the capability to detect and fix non-manifold geometry.
IJRR

Multiscale Deformable Objects Manipulation via Wavelet-Decomposed Boundary Element Method

Junlei Hu, Dominic Jones, Majed Melibary, Jiannan Liu, and Pietro Valdastri

2026

The International Journal of Robotics Research (IJRR)

Abs Code Website

Robotic deformable object manipulation faces critical challenges in surgical applications due to under-actuation, unpredictable tissue deformation, and limited intraoperative visibility. Traditional model-free methods often encounter unstable Jacobians caused by ill-conditioned observations, while physics-based models require precise tissue parameters and volumetric meshes, limiting their practicality in real surgeries. We propose a wavelet-Boundary Element Method (BEM) framework that encodes deformations from partial 3D shape feedback using multiscale wavelet descriptors. Coupling wavelets with BEM allows computation of the nonlinear motion-deformation mapping via boundary integrals, eliminating the need for volumetric meshes and enabling real-time control under occluded views. We validate our approach in simulation and on the da Vinci Research Kit (dVRK) using phantom organs and ex vivo animal tissue, achieving millimetre-level accuracy with contour, curve, and surface feedback. Comparative studies with Fourier-based and model-free methods show superior stability in dexterous tissue manipulation, addressing ill-conditioning through spatial-frequency localization. This framework enhances robotic manipulation in unstructured surgical environments, providing robust control and accurate tissue interaction despite partial observability. Our results demonstrate its potential to improve precision and safety in minimally invasive procedures.

2025

T-MI

NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3D Gaussian Reconstruction

Tianle Zeng, Junlei Hu, Gerardo Loza Galindo, Sharib Ali, Duygu Sarikaya, Pietro Valdastri, and Dominic Jones

2025

IEEE Transactions on Medical Imaging (T-MI)

Abs HTML

Computer vision-based technologies significantly enhance surgical automation by advancing tool tracking, detection, and localization. However, Current data-driven approaches are data-voracious, requiring large, high-quality labeled image datasets. Our Work introduces a novel dynamic Gaussian Splatting technique to address the data scarcity in surgical image datasets. We propose a dynamic Gaussian model to represent dynamic surgical scenes, enabling the rendering of surgical instruments from unseen viewpoints and deformations with real tissue backgrounds. We utilize a dynamic training adjustment strategy to address challenges posed by poorly calibrated camera poses from real-world scenarios. Additionally, automatically generate annotations for our synthetic data. For evaluation, we constructed a new dataset featuring seven scenes with 14,000 frames of tool and camera motion and tool jaw articulation, with a background of an exvivo porcine model. Using this dataset, we synthetically replicate the scene deformation from the ground truth data, allowing direct comparisons of synthetic image quality. Experimental results illustrate that our method generates photo-realistic labeled image datasets with the highest PSNR (29.87). We further evaluate the performance of medical-specific neural networks trained on real and synthetic images using an unseen real-world image dataset. Our results show that the performance of models trained on synthetic images generated by the proposed method outperforms those trained with state-of-the-art standard data augmentation by 10%, leading to an overall improvement in model performances by nearly 15%.
IEEE T-MRB
Instantaneous and Markerless Registration for Mandible Reconstruction Robot with Fibular Holder

Junlei Hu, Xiaoyan Jiang, Gang Zhu, Zhenggang Cao, Yao Gao, Zhaodong Zhang, Jing Han, and Jiannan Liu

IEEE Transactions on Medical Robotics and Bionics (T-MRB), 2025

Abs Bib HTML

Robotic-assisted mandible reconstruction surgery is reported to achieve a more precise fibula osteotomy. However, intraoperative registration of the fibula with preoperative imaging is the key to achieving precision. Although methods such as probe scanning, optical navigation, and point cloud recognition are available, the reliability and efficiency of registration are low due to the sparse anatomical features and bone membrane covering on the fibular surface. Fibular shaping involves multiple bone segment osteotomies, and therefore, the design of fibular holder is necessary to achieve stable shaping. Based on these facts, an instantaneous and markerless registration based on the fibular holder is proposed, consisting of the initial alignment and refinement. The initial alignment using the point cloud or the holder to locate the fibula. The refinement, based on mechanical constraints, is developed and solved iteratively, utilizing the mechanical constraints between the fibula and the V-shaped mounts of the fibular holder. The accuracy of the approach is evaluated through simulations, model experiments and animal experiments. The results indicated that this algorithm provides higher accuracy and efficiency. This method can achieve non-invasive, efficient, and highly precise fibular registration in mandibular reconstruction surgery.
@article{hu2025Instantaneous, author = {Hu, Junlei and Jiang, Xiaoyan and Zhu, Gang and Cao, Zhenggang and Gao, Yao and Zhang, Zhaodong and Han, Jing and Liu, Jiannan}, title = {Instantaneous and Markerless Registration for Mandible Reconstruction Robot with Fibular Holder}, journal = {IEEE Transactions on Medical Robotics and Bionics (T-MRB)}, year = {2025}, }
arXiv
Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision

Gerardo Loza, Junlei Hu, Dominic Jones, Sharib Ali, and Pietro Valdastri

arXiv preprint arXiv:2508.09681 (Submitted to IEEE Transactions on Medical Imaging), 2025

Under Review

Abs Bib HTML

We proposed a novel test-time optimisation (TTO) approach framed by a NeRF-based architecture for long-term 3D point tracking. Most current methods in point tracking struggle to obtain consistent motion or are limited to 2D motion. TTO approaches frame the solution for long-term tracking as optimising a function that aggregates correspondences from other specialised state-of-the-art methods. Unlike the state-of-the-art on TTO, we propose parametrising such a function with our new invertible Neural Radiance Field (InvNeRF) architecture to perform both 2D and 3D tracking in surgical scenarios. Our approach allows us to exploit the advantages of a rendering-based approach by supervising the reprojection of pixel correspondences. It adapts strategies from recent rendering-based methods to obtain a bidirectional deformable-canonical mapping, to efficiently handle a defined workspace, and to guide the rays’ density. It also presents our multi-scale HexPlanes for fast inference and a new algorithm for efficient pixel sampling and convergence criteria. We present results in the STIR and SCARE datasets, for evaluating point tracking and testing the integration of kinematic data in our pipeline, respectively. In 2D point tracking, our approach surpasses the precision and accuracy of the TTO state-of-the-art methods by nearly 50% on average precision, while competing with other approaches. In 3D point tracking, this is the first TTO approach, surpassing feed-forward methods while incorporating the benefits of a deformable NeRF-based reconstruction.
@article{loza2025surg, title = {Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision}, author = {Loza, Gerardo and Hu, Junlei and Jones, Dominic and Ali, Sharib and Valdastri, Pietro}, journal = {arXiv preprint arXiv:2508.09681 (Submitted to IEEE Transactions on Medical Imaging)}, year = {2025}, note = {Under Review}, }

2024

MICCAI 2024
Realistic surgical image dataset generation based on 3d gaussian splatting

Tianle Zeng, Gerardo Loza Galindo, Junlei Hu, Pietro Valdastri, and Dominic Jones

In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2024

Abs Bib HTML

Computer vision technologies markedly enhance the automation capabilities of robotic-assisted minimally invasive surgery (RAMIS) through advanced tool tracking, detection, and localization. However, the limited availability of comprehensive surgical datasets for training represents a significant challenge in this field. This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We propose a method for extracting and combining 3D Gaussian representations of surgical instruments and background operating environments, transforming and combining them to generate high-fidelity synthetic surgical scenarios. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality (27.796\pm1.796 PSNR). As a further validation, we compared two YOLOv5 models trained on the synthetic and real data, respectively, and assessed their performance in an unseen real-world test dataset. Comparing the performances, we observe an improvement in neural network performance, with the synthetic-trained model outperforming the real-world trained model by 12%, testing both on real-world data.
@inproceedings{zeng2024realistic, title = {Realistic surgical image dataset generation based on 3d gaussian splatting}, author = {Zeng, Tianle and Loza Galindo, Gerardo and Hu, Junlei and Valdastri, Pietro and Jones, Dominic}, booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)}, pages = {510--519}, year = {2024}, organization = {Springer}, }

2023

IEEE T-RO
Occlusion-Robust Autonomous Robotic Manipulation of Human Soft Tissues With 3-D Surface Feedback

Junlei Hu, Dominic Jones, Mehmet R Dogar, and Pietro Valdastri

IEEE Transactions on Robotics (T-RO), 2023

Abs Bib HTML Website

Robotic manipulation of 3-D soft objects remains challenging in the industrial and medical fields. Various methods based on mechanical modeling, data-driven approaches or explicit feature tracking have been proposed. A unifying disadvantage of these methods is the high computational cost of simultaneous imaging processing, identification of mechanical properties, and motion planning, leading to a need for less computationally intensive methods. We propose a method for autonomous robotic manipulation with 3-D surface feedback to solve these issues. First, we produce a deformation model of the manipulated object, which estimates the robots’ movements by monitoring the displacement of surface points surrounding the manipulators. Then, we develop a 6-degree-of-freedom velocity controller to manipulate the grasped object to achieve a desired shape. We validate our approach through comparative simulations with existing methods and experiments using phantom and cadaveric soft tissues with the da Vinci research kit. The results demonstrate the robustness of the technique to occlusions and various materials. Compared to state-of-the-art linear and data-driven methods, our approach is more precise by 46.5% and 15.9% and saves 55.2% and 25.7% manipulation time, respectively.
@article{hu2023occlusion, title = {Occlusion-Robust Autonomous Robotic Manipulation of Human Soft Tissues With 3-D Surface Feedback}, author = {Hu, Junlei and Jones, Dominic and Dogar, Mehmet R and Valdastri, Pietro}, journal = {IEEE Transactions on Robotics (T-RO)}, volume = {40}, pages = {624--638}, year = {2023}, publisher = {IEEE}, }
ICRA 2023
Coordinate Calibration of a Dual-Arm Robot System by Visual Tool Tracking

Junlei Hu, Dominic Jones, and Pietro Valdastri

In IEEE International Conference on Robotics and Automation (ICRA), 2023

Abs Bib HTML

The calibration of a vision-guided dual-arm robotic system, including the robot-robot and hand-eye calibration, requires the tracked positions of markers in different postures. However, in many cases, using markers to calibrate is impractical. Only some markerless features can be obtained rather than the rigid transform matrix; for example, the shaft of a markerless robotic tool can be tracked. Therefore, we proposed a Kronecker-Product-based method to calibrate the dual-arm system with a tracked robotic tool by decoupling the translation and rotation. The simulation and experiment results on a da Vinci Research Kit show that the proposed method is robust and accurate under different noise levels and various sample robot movements, compared with two state-of-the-art methods for dual-arm calibration with complete homogeneous transformations.
@inproceedings{hu2023coordinate, title = {Coordinate Calibration of a Dual-Arm Robot System by Visual Tool Tracking}, author = {Hu, Junlei and Jones, Dominic and Valdastri, Pietro}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, pages = {11468--11473}, year = {2023}, organization = {IEEE}, }

2020

IJCARS
Automatic robot-world calibration in an optical-navigated surgical robot system and its application for oral implant placement

Yang Li^*, Junlei Hu^*, Baoxin Tao, Dedong Yu, Yihan Shen, Shengchi Fan, Yiqun Wu, and Xiaojun Chen

International Journal of Computer Assisted Radiology and Surgery (IJCARS), 2020

Abs Bib HTML

Robot-world calibration, used to precisely determine the spatial relation between optical tracker and robot, is regarded as an essential step for optical-navigated surgical robot system to improve the surgical accuracy. However, these methods are complicated with numerous computation. Therefore, a more efficient method of a robot-world calibration is necessary. A fully automatic robot-world calibration was proposed and applied in a surgical robot system for oral implant placement. Making full usage of the movement characteristics of a tandem robot, the least square fitting algorithm was implemented to calculate the relationship between the tool center point of the robot and the robot reference frame, with the robot-world calibration matrix obtained as result.The experiment was designed to verify the accuracy of the robot-world calibration. The average distance deviation was 1.11 mm, and the average angle deviation was 0.99°. From the animal experiment on the pig maxilla, the entry, apical and angle deviation of the surgical robot system were 1.44±1.01 mm, 1.68±0.76 mm, 1.01±1.06°, respectively. The surgical robot system for oral implant placement with our robot-world calibration maintains a high precision. Besides, the operation range of the surgical tool is no longer limited by the visual range of the optical tracking device. Hence, it is unnecessary to adjust the optical tracking device for the planned implant trajectories to different positions and directions.
@article{li2020automatic, title = {Automatic robot-world calibration in an optical-navigated surgical robot system and its application for oral implant placement}, author = {Li, Yang and Hu, Junlei and Tao, Baoxin and Yu, Dedong and Shen, Yihan and Fan, Shengchi and Wu, Yiqun and Chen, Xiaojun}, journal = {International Journal of Computer Assisted Radiology and Surgery (IJCARS)}, volume = {15}, pages = {1685--1692}, year = {2020}, publisher = {Springer}, }
IJCARS
An approach to automated measuring morphological parameters of proximal femora on three-dimensional models

Junlei Hu, Liyu Xu, Mengjie Jing, Henghui Zhang, Liao Wang, and Xiaojun Chen

International Journal of Computer Assisted Radiology and Surgery (IJCARS), 2020

Abs Bib HTML

Analyses of the morphology of proximal femora are essential for preoperative planning and designing customized femoral stems in total hip arthroplasty as well as intramedullary nailing fixation. Various studies reported measurements and analyses on the general geometry of proximal femora three-dimensionally. However, the modeling and measurements are time-consuming and unfriendly to surgeons. Thus, automated measurement and modeling of the femoral medullary canal are critical to promote the clinical application. An approach to automated measuring morphological parameters of proximal femur was proposed, and a software allowing importing femur models and manually locating the related anatomic landmarks was developed in the current study. 3D modeling of the femoral medullary canal was created by the semispherical iterative searching algorithm, and 16 key anatomic parameters of the proximal femur were automatically calculated by the least-squares fitting algorithm. By experimenting on 196 femur STL models, the average execution time of single measurement was 0.88 (SD = 0.13) s, and the intra-class correlation coefficient of 10 anatomic parameters was between 0.880 and 0.996, showing high intra-rater and inter-rater reliability. The parameters of proximal femur can be easily measured, and the 3D modeling of the femoral medullary canal can be rapidly achieved. The approach will be easily applicable to clinical practice and has the potential to be applied in the design of customized femoral stems.
@article{hu2020approach, title = {An approach to automated measuring morphological parameters of proximal femora on three-dimensional models}, author = {Hu, Junlei and Xu, Liyu and Jing, Mengjie and Zhang, Henghui and Wang, Liao and Chen, Xiaojun}, journal = {International Journal of Computer Assisted Radiology and Surgery (IJCARS)}, volume = {15}, number = {1}, pages = {109--118}, year = {2020}, publisher = {Springer}, }