Robotic Surgery for Rectal Cancer with Augmented Reality of 3D Model During Operation: A Case Report

Rie Hayashi 1,2*, Norikatsu Miyoshi 1,2, Shiki Fujino 1,2,3

1Osaka University, Graduate School of Medicine, Department of Gastroenterological Surgery, Japan.

2Osaka International Cancer Institute, Department of Innovative Oncology Research and Regenerative Medicine, Japan.

3Monash University, Central Clinical School, Japan.

*Corresponding Author:Rie Hayashi, Osaka University, Graduate School of Medicine, Department of Gastroenterological Surgery, 2-2-E2 Yamadaoka, Suita, Osaka 565-0871, Japan, Tel: +81-6-6879-3251; Fax: +81-6-6879-3259

Citation: Rie Hayashi, Norikatsu Miyoshi, Shiki Fujino (2023)Robotic surgery for rectal cancer with augmented reality of 3D model during operation: A case report. Cancer Prog Diagn 7: 141.

Received: May 18, 2023; Accepted: May 20, 2023; Published: May 24, 2023.


Recently, robotic surgery for rectal cancer has become common as minimally invasive surgery. Aside from this topic, the technology of augmented and mixed reality is applied in various living environments, and also medicine. We report to perform a robotic surgery for rectal cancer successfully with three-dimensional (3D) images as mixed reality (MR) by HoloLens2 during operation. The patient was diagnosed with rectal cancer by colonoscopy PET-CT scan, and we performed robot-assisted anterior resection. The operator used HoloLens2 and performed the surgery while visualizing 3D image of pelvic anatomy with the location of the rectal cancer as hologram. The operation was performed completely and safely, and she was discharged 11 days after surgery with no postoperative complications. This case is very informative to discuss the usefulness of MR system visualized as hologram during surgery.

Key Words

Key Words: Rectal Cancer, Robotic Surgery, Mixed Reality, Holograms.


Colorectal cancer is one of the most common cancers in the world [1], and surgical resection is commonly performed for curative treatment. Some randomized clinical trials have confirmed that laparoscopic surgery resulted in equivalent long-term oncologic outcomes and improved short-term results compared with traditional open surgery [2-5]. Further technological advances have led to the development of robotic surgery. Robotic surgery has many technical advantages such as superior stability, curved flexible instruments, increased dexterity, fixed stable traction and improved accuracy of surgery [6–8]. Previous studies have reported that robotic surgery for colorectal cancer is associated with a shorter length of hospital stay, lower conversion rate, and lower overall complication rate than laparoscopic surgery [9–11]. Surgical methods have gradually transitioned from traditional laparotomy to laparoscopic surgery and robotic surgery. Although surgical equipment has advanced, especially in the case of rectal cancer, high skill is still required to perform the surgery while correctly understanding the pelvic anatomy. Therefore, supporting tools are useful to improve the surgical advantages. Recently, augmented and mixed reality technology has been applied to surgery [12–14]. Mixed reality (MR) enables the simultaneous experience of the real world and virtual reality by precisely superimposing images in coordinate spaces. The application of mixed reality to surgery, specifically the use of head mount display (HMD), allows surgeons to use intraoperative mixed reality technology to visualize 3D computer graphics models (holograms) of each patient during operation [14].

Here, we report a surgical case, robotic surgery performed with 3D images by HoloLens2 (Microsoft, Microsoft Corporation, Redmond, WA) as HMD during operation. The holograms were created by preoperative positron emission tomography-computed tomography (PET-CT).

Case Presentation

The patient was a 41-year-old woman with a history of abdominal surgery. She was not prescribed any medicine. The patient’s family medical history was unremarkable. The patient visited a nearby physician for positive fecal occult blood and was performed colonoscopy. She was diagnosed with rectal cancer and was referred to our hospital for curative treatment. Her height was 160cm, weight was 52kg, BMI was 20.3.  Her preoperative laboratory data were as follows: leukocyte 4,770 /dL, hemoglobin 12.5 g/dL, platelet 20.1×104 /µL, AST 14 U/L, ALT 8 U/L, ALP 43 IU/L, BUN 16 mg/dL, creatinine 0.63 mg/dL, CRP 0.04 mg/dL, electrolytes were within normal limits, CA19-9 9.5 U/mL, and CEA 2 ng/mL. A PET-CT and magnetic resonance imaging (MRI) revealed the localization of the cancer. Robot-assisted anterior resection was performed. The operator used HoloLens2 and performed the surgery while visualizing 3D image of pelvic anatomy with the location of the rectal cancer as hologram (Fig. A-C). The operation time was 261 minutes, and blood loss was 30ml. The patient’s postoperative course was uneventful, and was discharged 11 days after surgery, who is still relapse-free under follow-up.

Figure (A-C):

A: 3D reconstruction by preoperative PET-CT scan data (Green: rectal cancer)

B: Robotic Surgery was performed for rectal cancer (TilePro showing endoscopic images during operation)

C: Operator can visualize the 3D image as hologram by HoloLens2 during operation. (Upper)

3D model can be magnified in the hologram view. (Lower) 3D model can be moved by the operator.


This case showed the utility and safety of intraoperative hologram support for rectal cancer. The pelvic cavity is surrounded by blood vessels and nerves, which should not be injured, so it is important to develop the support tools for improving the surgical safety. The hologram image can support and provide surgeons with a better understanding of the individual patient’s pelvic anatomy. The information can’t be obtained from the CT images or operation field on planar monitor during operation. Furthermore, the 3D model can be magnified in the hologram view, and the surgeon can move to observe the hologram from various angles. The hologram was created by preoperative PET-CT and reflected the anatomy. Thus, it is considered that intraoperative simulations can improve the anatomical understanding compared with preoperative simulations. And if all surgeons on the surgery team wear the HoloLens2 to share the same hologram, their discussion and communication will be more active and improve the understanding of the positional relationship of patient-specific organs. In conclusion, it suggests that the use of intraoperative hologram support can improve the understanding of the individual’s anatomy and surgical safety. It also suggests that intraoperative holograms could be useful surgical tools to educate next-generation surgeons. Further study is needed to accumulate more cases in the future.


  1. Ferlay J, Soerjomataram I, Dikshit R, et al. (2015) Cancer incidence and mortality worldwide: sources, methods and major patterns in GLOBOCAN 2012. Int J Cancer 136: E359-386.
  2. Jayne DG, Guillou PJ, Thorpe H, et al. (2007) Randomized trial of laparoscopic-assisted resection of colorectal carcinoma: 3-year results of the UK MRC CLASICC Trial Group. J Clin Oncol 25: 3061–3068.
  3. Veldkamp R, Kuhry E, Hop WC, et al. (2005) Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomised trial. Lancet Oncol 6: 477–484.
  4. Buunen M, Veldkamp R, Hop WC, et al. (2009) Survival after laparoscopic surgery versus open surgery for colon cancer: long-term outcome of a randomised clinical trial. Lancet Oncol 10: 44–52.
  5. Nelson H, Sargent DJ, Wieand HS, et al. (2004) A comparison of laparoscopically assisted and open colectomy for colon cancer. N Engl J Med 350: 2050–2059.
  6. Aly EH (2014) Robotic colorectal surgery: summary of the current evidence. Int J Colorectal Dis 29:1–8.
  7. Kim J, Baek SJ, Kang DW, et al. (2017) Robotic resection is a good prognostic factor in rectal cancer compared with laparoscopic resection: long-term survival analysis using propensity score matching. Dis Colon Rectum 60: 266–273.
  8. Kuo LJ, Ngu JC, Tong YS, et al. (2017) Combined robotic transanal total mesorectal excision (R-taTME) and single-site plus one-port (R-SSPO) technique for ultra-low rectal surgery-initial experience with a new operation approach. Int J Colorectal Dis 32: 249–254.
  9. Wang X, Cao G, Mao W, et al. (2020) Robot-assisted versus laparoscopic surgery for rectal cancer: A systematic review and meta-analysis. J Can Res Ther 16: 979–989.
  10. Zhang Y, Feng H, Wang S, et al. (2022) Short- and long-term outcomes of robotic- versus laparoscopic-assisted right hemicolectomy: A propensity score-matched retrospective cohort study. Int J Surg 105: 106855.
  11. Feng Q, Yuan W, Li T, et al. (2022) Robotic versus laparoscopic surgery for middle and low rectal cancer (REAL): short-term outcomes of a multicentre randomised controlled trial. Lancet Gastroenterol Hepatol 7: 991–1004.
  12. Müller F, Roner S, Liebmann F, et al. (2020) Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J 20: 621–628.
  13. Neves CA, Vaisbuch Y, Leuze C, et al. (2020) Application of holographic augmented reality for external approaches to the frontal sinus. Int Forum Allergy Rhinol 10: 920–925.
  14. Saito Y, Sugimoto M, Imura S, et al. (2020) Intraoperative 3D hologram support with mixed reality techniques in liver surgery. Ann Surg 271: 4–7.