Automated analysis of operative video in surgical training: scoping review

General study characteristics

Study	Country	Study type	Specialty	Operation	Approach	Number of videos in study
Azari et al.³¹	USA	PCS	Pan-specialty	Multiple	Open	103
Balal et al.³²	UK	PCS	Ophthalmology	Cataract surgery	Open	120
Din et al.³³	UK	PCS	Ophthalmology	Cataract surgery	Open	22
Frasier et al.³⁴	USA	PCS	Pan-specialty	Multiple	Open	138
Glarner et al.³⁵	USA	PCS	Plastic surgery	Reduction mammoplasty	Open	6
Hameed et al.³⁶	Canada	Qualitative	Hepatobiliary	Cholecystectomy	Laparoscopic	Not applicable
Humm et al.³⁷	UK	RCS	Hepatobiliary	Cholecystectomy	Laparoscopic	159
Kitaguchi et al.³⁸	Japan	RCS	Colorectal	Transanal total mesorectal excision	Endoscopic	45
Lee et al.³⁹	South Korea	RCS	Endocrinology	Thyroidectomy	Robotic	40
Smith et al.⁴⁰	UK	PCS	Ophthalmology	Cataract surgery	Open	20
Wawrzynski et al.⁴¹	UK	PCS	Otolaryngology	Dacryocystorhinostomy	Endoscopic	20
Wu et al.⁴²	China	PCS	Hepatobiliary	Cholecystectomy	Laparoscopic	1107
Yang et al.⁴³	USA	PCS	Colorectal	Rectopexy	Robotic	92

Study	Country	Study type	Specialty	Operation	Approach	Number of videos in study
Azari et al.³¹	USA	PCS	Pan-specialty	Multiple	Open	103
Balal et al.³²	UK	PCS	Ophthalmology	Cataract surgery	Open	120
Din et al.³³	UK	PCS	Ophthalmology	Cataract surgery	Open	22
Frasier et al.³⁴	USA	PCS	Pan-specialty	Multiple	Open	138
Glarner et al.³⁵	USA	PCS	Plastic surgery	Reduction mammoplasty	Open	6
Hameed et al.³⁶	Canada	Qualitative	Hepatobiliary	Cholecystectomy	Laparoscopic	Not applicable
Humm et al.³⁷	UK	RCS	Hepatobiliary	Cholecystectomy	Laparoscopic	159
Kitaguchi et al.³⁸	Japan	RCS	Colorectal	Transanal total mesorectal excision	Endoscopic	45
Lee et al.³⁹	South Korea	RCS	Endocrinology	Thyroidectomy	Robotic	40
Smith et al.⁴⁰	UK	PCS	Ophthalmology	Cataract surgery	Open	20
Wawrzynski et al.⁴¹	UK	PCS	Otolaryngology	Dacryocystorhinostomy	Endoscopic	20
Wu et al.⁴²	China	PCS	Hepatobiliary	Cholecystectomy	Laparoscopic	1107
Yang et al.⁴³	USA	PCS	Colorectal	Rectopexy	Robotic	92

PCS, prospective cohort study; RCS, retrospective cohort study.

Table 1

General study characteristics

Study	Country	Study type	Specialty	Operation	Approach	Number of videos in study
Azari et al.³¹	USA	PCS	Pan-specialty	Multiple	Open	103
Balal et al.³²	UK	PCS	Ophthalmology	Cataract surgery	Open	120
Din et al.³³	UK	PCS	Ophthalmology	Cataract surgery	Open	22
Frasier et al.³⁴	USA	PCS	Pan-specialty	Multiple	Open	138
Glarner et al.³⁵	USA	PCS	Plastic surgery	Reduction mammoplasty	Open	6
Hameed et al.³⁶	Canada	Qualitative	Hepatobiliary	Cholecystectomy	Laparoscopic	Not applicable
Humm et al.³⁷	UK	RCS	Hepatobiliary	Cholecystectomy	Laparoscopic	159
Kitaguchi et al.³⁸	Japan	RCS	Colorectal	Transanal total mesorectal excision	Endoscopic	45
Lee et al.³⁹	South Korea	RCS	Endocrinology	Thyroidectomy	Robotic	40
Smith et al.⁴⁰	UK	PCS	Ophthalmology	Cataract surgery	Open	20
Wawrzynski et al.⁴¹	UK	PCS	Otolaryngology	Dacryocystorhinostomy	Endoscopic	20
Wu et al.⁴²	China	PCS	Hepatobiliary	Cholecystectomy	Laparoscopic	1107
Yang et al.⁴³	USA	PCS	Colorectal	Rectopexy	Robotic	92

Study	Country	Study type	Specialty	Operation	Approach	Number of videos in study
Azari et al.³¹	USA	PCS	Pan-specialty	Multiple	Open	103
Balal et al.³²	UK	PCS	Ophthalmology	Cataract surgery	Open	120
Din et al.³³	UK	PCS	Ophthalmology	Cataract surgery	Open	22
Frasier et al.³⁴	USA	PCS	Pan-specialty	Multiple	Open	138
Glarner et al.³⁵	USA	PCS	Plastic surgery	Reduction mammoplasty	Open	6
Hameed et al.³⁶	Canada	Qualitative	Hepatobiliary	Cholecystectomy	Laparoscopic	Not applicable
Humm et al.³⁷	UK	RCS	Hepatobiliary	Cholecystectomy	Laparoscopic	159
Kitaguchi et al.³⁸	Japan	RCS	Colorectal	Transanal total mesorectal excision	Endoscopic	45
Lee et al.³⁹	South Korea	RCS	Endocrinology	Thyroidectomy	Robotic	40
Smith et al.⁴⁰	UK	PCS	Ophthalmology	Cataract surgery	Open	20
Wawrzynski et al.⁴¹	UK	PCS	Otolaryngology	Dacryocystorhinostomy	Endoscopic	20
Wu et al.⁴²	China	PCS	Hepatobiliary	Cholecystectomy	Laparoscopic	1107
Yang et al.⁴³	USA	PCS	Colorectal	Rectopexy	Robotic	92

PCS, prospective cohort study; RCS, retrospective cohort study.

Participant details

There were a total of 107 trainees in the nine studies that reported on numbers of participants (mean(s.d.) 12(9.2)). The majority of studies (8 studies) did not report the grade of training. A total of five studies reported the baseline skills of trainees, using the total prior number of operations performed as the sole marker of skill. Of the studies, one used baseline skills to differentiate between novices, intermediates, and experts.

All of the included studies had a comparator group, with a total of 152 attendings or experts in the ten studies reporting these numbers (mean(s.d.) 15(17.5)). In contrast to trainees, the baseline skills of attendings and experts were more frequently reported (7 studies), using the prior number of operations performed as the marker of skill. Full participant characteristics are available in Table 2.

Table 2

Characteristics of study participants

Study	Number of trainees	Level of training	Baseline reported*	Number of experts for comparison	Baseline reported
Azari et al.³¹	3	Resident (unspecified)	Unknown	6	Unknown
Balal et al.³²	20	Resident (unspecified)	<200 operations performed	20	>1000 operations performed
Din et al.³³	31	Resident (unspecified)	<200 operations performed	31	>1000 operations performed
Frasier et al.³⁴	3	PGY3–PGY5	Not reported	6	Unknown
Glarner et al.³⁵	3	PGY3–PGY5	Unknown	3	8–30 years of experience
Hameed et al.³⁶	13	Fellow, Senior resident, Junior resident	Unknown	7	Mean of 57 laparoscopic cholecystectomies per year
Humm et al.³⁷	Unknown	Resident (unspecified)	Unknown	1	Unknown
Kitaguchi et al.³⁸	Unknown	Unknown	Novice <10 operations, intermediate 10–30 operations	Unknown	>30 operations performed
Lee et al.³⁹	Unknown	Fellow	Unknown	Unknown	Unknown
Smith et al.⁴⁰	10	Resident (unspecified)	<200 operations performed	10	>1000 operations performed
Wawrzynski et al.⁴¹	10	Resident (unspecified)	<20 operations performed	10	>100 operations performed
Wu et al.⁴²	14	Resident (unspecified)	Unknown	58	Unknown
Yang et al.⁴³	Unknown	PGY3–PGY7	Unknown	Unknown	Unknown

Study	Number of trainees	Level of training	Baseline reported*	Number of experts for comparison	Baseline reported
Azari et al.³¹	3	Resident (unspecified)	Unknown	6	Unknown
Balal et al.³²	20	Resident (unspecified)	<200 operations performed	20	>1000 operations performed
Din et al.³³	31	Resident (unspecified)	<200 operations performed	31	>1000 operations performed
Frasier et al.³⁴	3	PGY3–PGY5	Not reported	6	Unknown
Glarner et al.³⁵	3	PGY3–PGY5	Unknown	3	8–30 years of experience
Hameed et al.³⁶	13	Fellow, Senior resident, Junior resident	Unknown	7	Mean of 57 laparoscopic cholecystectomies per year
Humm et al.³⁷	Unknown	Resident (unspecified)	Unknown	1	Unknown
Kitaguchi et al.³⁸	Unknown	Unknown	Novice <10 operations, intermediate 10–30 operations	Unknown	>30 operations performed
Lee et al.³⁹	Unknown	Fellow	Unknown	Unknown	Unknown
Smith et al.⁴⁰	10	Resident (unspecified)	<200 operations performed	10	>1000 operations performed
Wawrzynski et al.⁴¹	10	Resident (unspecified)	<20 operations performed	10	>100 operations performed
Wu et al.⁴²	14	Resident (unspecified)	Unknown	58	Unknown
Yang et al.⁴³	Unknown	PGY3–PGY7	Unknown	Unknown	Unknown

PGY, postgraduate year. *Definition used in each study to quantify experience of participants. In all studies, except one, reporting baseline, the number of operations previously performed was used.

Table 2

Characteristics of study participants

Study	Number of trainees	Level of training	Baseline reported*	Number of experts for comparison	Baseline reported
Azari et al.³¹	3	Resident (unspecified)	Unknown	6	Unknown
Balal et al.³²	20	Resident (unspecified)	<200 operations performed	20	>1000 operations performed
Din et al.³³	31	Resident (unspecified)	<200 operations performed	31	>1000 operations performed
Frasier et al.³⁴	3	PGY3–PGY5	Not reported	6	Unknown
Glarner et al.³⁵	3	PGY3–PGY5	Unknown	3	8–30 years of experience
Hameed et al.³⁶	13	Fellow, Senior resident, Junior resident	Unknown	7	Mean of 57 laparoscopic cholecystectomies per year
Humm et al.³⁷	Unknown	Resident (unspecified)	Unknown	1	Unknown
Kitaguchi et al.³⁸	Unknown	Unknown	Novice <10 operations, intermediate 10–30 operations	Unknown	>30 operations performed
Lee et al.³⁹	Unknown	Fellow	Unknown	Unknown	Unknown
Smith et al.⁴⁰	10	Resident (unspecified)	<200 operations performed	10	>1000 operations performed
Wawrzynski et al.⁴¹	10	Resident (unspecified)	<20 operations performed	10	>100 operations performed
Wu et al.⁴²	14	Resident (unspecified)	Unknown	58	Unknown
Yang et al.⁴³	Unknown	PGY3–PGY7	Unknown	Unknown	Unknown

Study	Number of trainees	Level of training	Baseline reported*	Number of experts for comparison	Baseline reported
Azari et al.³¹	3	Resident (unspecified)	Unknown	6	Unknown
Balal et al.³²	20	Resident (unspecified)	<200 operations performed	20	>1000 operations performed
Din et al.³³	31	Resident (unspecified)	<200 operations performed	31	>1000 operations performed
Frasier et al.³⁴	3	PGY3–PGY5	Not reported	6	Unknown
Glarner et al.³⁵	3	PGY3–PGY5	Unknown	3	8–30 years of experience
Hameed et al.³⁶	13	Fellow, Senior resident, Junior resident	Unknown	7	Mean of 57 laparoscopic cholecystectomies per year
Humm et al.³⁷	Unknown	Resident (unspecified)	Unknown	1	Unknown
Kitaguchi et al.³⁸	Unknown	Unknown	Novice <10 operations, intermediate 10–30 operations	Unknown	>30 operations performed
Lee et al.³⁹	Unknown	Fellow	Unknown	Unknown	Unknown
Smith et al.⁴⁰	10	Resident (unspecified)	<200 operations performed	10	>1000 operations performed
Wawrzynski et al.⁴¹	10	Resident (unspecified)	<20 operations performed	10	>100 operations performed
Wu et al.⁴²	14	Resident (unspecified)	Unknown	58	Unknown
Yang et al.⁴³	Unknown	PGY3–PGY7	Unknown	Unknown	Unknown

PGY, postgraduate year. *Definition used in each study to quantify experience of participants. In all studies, except one, reporting baseline, the number of operations previously performed was used.

Method of automated video analysis

The most common method of automated video analysis was computer vision tracking (7 studies). This was used in all of the studies that assessed open surgery and one of the studies that assessed endoscopic procedures; three of these studies required manual identification of the region of interest (ROI) before computer vision tracking could be used for the completion of the assessment automatically^31,34,35. More advanced AI-based approaches to video analysis were used in the remaining studies, with convolutional neural networking (CNN) being the most common method (4 studies) (Table 3). CNN relies on recognizing patterns to process images. Pyramid scene parsing networking (PSPN), based on image segmentation, was also used to generate heat maps in laparoscopic cholecystectomy³⁶.

Table 3

Summary of the included studies

Study	Method of analysis	Metric(s) produced	Application to training
Azari et al.³¹	Computer vision tracking	Fluidity of motion Motion economy Tissue handling	Metric correlated with alternative assessment tool
Balal et al.³²	Computer vision tracking	Intraoperative phase duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Din et al.³³	Computer vision tracking	Instrument distance travelled Instrument use	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Frasier et al.³⁴	Computer vision tracking	Speed and acceleration of movement	To differentiate between different skill sets
Glarner et al.³⁵	Computer vision tracking	Instrument use Bimanual dexterity Displacement, velocity, and acceleration	To differentiate between different skill sets
Hameed et al.³⁶	PSPN	Heat maps	Qualitative study
Humm et al.³⁷	CNN	Total operative duration Intraoperative phase duration	To differentiate between different skill sets
Kitaguchi et al.³⁸	CNN	Generated suturing score	To differentiate between different skill sets
Lee et al.³⁹	CNN	Classification of skill (novice, skilled, and expert)	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Smith et al.⁴⁰	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wawrzynski et al.⁴¹	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wu et al.⁴²	TSM	CVS achievement	Metric correlated with years of experience
Yang et al.⁴³	CNN	Instrument distance travelled Bimanual dexterity Depth perception	Metric correlated with alternative assessment tool + To differentiate between different skill sets

Study	Method of analysis	Metric(s) produced	Application to training
Azari et al.³¹	Computer vision tracking	Fluidity of motion Motion economy Tissue handling	Metric correlated with alternative assessment tool
Balal et al.³²	Computer vision tracking	Intraoperative phase duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Din et al.³³	Computer vision tracking	Instrument distance travelled Instrument use	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Frasier et al.³⁴	Computer vision tracking	Speed and acceleration of movement	To differentiate between different skill sets
Glarner et al.³⁵	Computer vision tracking	Instrument use Bimanual dexterity Displacement, velocity, and acceleration	To differentiate between different skill sets
Hameed et al.³⁶	PSPN	Heat maps	Qualitative study
Humm et al.³⁷	CNN	Total operative duration Intraoperative phase duration	To differentiate between different skill sets
Kitaguchi et al.³⁸	CNN	Generated suturing score	To differentiate between different skill sets
Lee et al.³⁹	CNN	Classification of skill (novice, skilled, and expert)	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Smith et al.⁴⁰	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wawrzynski et al.⁴¹	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wu et al.⁴²	TSM	CVS achievement	Metric correlated with years of experience
Yang et al.⁴³	CNN	Instrument distance travelled Bimanual dexterity Depth perception	Metric correlated with alternative assessment tool + To differentiate between different skill sets

PSPN, pyramid scene parsing networking; CNN, convolutional neural networking; TSM, temporal shifting module; CVS, critical view of safety.

Table 3

Summary of the included studies

Study	Method of analysis	Metric(s) produced	Application to training
Azari et al.³¹	Computer vision tracking	Fluidity of motion Motion economy Tissue handling	Metric correlated with alternative assessment tool
Balal et al.³²	Computer vision tracking	Intraoperative phase duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Din et al.³³	Computer vision tracking	Instrument distance travelled Instrument use	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Frasier et al.³⁴	Computer vision tracking	Speed and acceleration of movement	To differentiate between different skill sets
Glarner et al.³⁵	Computer vision tracking	Instrument use Bimanual dexterity Displacement, velocity, and acceleration	To differentiate between different skill sets
Hameed et al.³⁶	PSPN	Heat maps	Qualitative study
Humm et al.³⁷	CNN	Total operative duration Intraoperative phase duration	To differentiate between different skill sets
Kitaguchi et al.³⁸	CNN	Generated suturing score	To differentiate between different skill sets
Lee et al.³⁹	CNN	Classification of skill (novice, skilled, and expert)	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Smith et al.⁴⁰	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wawrzynski et al.⁴¹	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wu et al.⁴²	TSM	CVS achievement	Metric correlated with years of experience
Yang et al.⁴³	CNN	Instrument distance travelled Bimanual dexterity Depth perception	Metric correlated with alternative assessment tool + To differentiate between different skill sets

Study	Method of analysis	Metric(s) produced	Application to training
Azari et al.³¹	Computer vision tracking	Fluidity of motion Motion economy Tissue handling	Metric correlated with alternative assessment tool
Balal et al.³²	Computer vision tracking	Intraoperative phase duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Din et al.³³	Computer vision tracking	Instrument distance travelled Instrument use	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Frasier et al.³⁴	Computer vision tracking	Speed and acceleration of movement	To differentiate between different skill sets
Glarner et al.³⁵	Computer vision tracking	Instrument use Bimanual dexterity Displacement, velocity, and acceleration	To differentiate between different skill sets
Hameed et al.³⁶	PSPN	Heat maps	Qualitative study
Humm et al.³⁷	CNN	Total operative duration Intraoperative phase duration	To differentiate between different skill sets
Kitaguchi et al.³⁸	CNN	Generated suturing score	To differentiate between different skill sets
Lee et al.³⁹	CNN	Classification of skill (novice, skilled, and expert)	Metric correlated with alternative assessment tool + To differentiate between different skill sets
Smith et al.⁴⁰	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wawrzynski et al.⁴¹	Computer vision tracking	Total operative duration Instrument distance travelled Instrument use	To differentiate between different skill sets
Wu et al.⁴²	TSM	CVS achievement	Metric correlated with years of experience
Yang et al.⁴³	CNN	Instrument distance travelled Bimanual dexterity Depth perception	Metric correlated with alternative assessment tool + To differentiate between different skill sets

PSPN, pyramid scene parsing networking; CNN, convolutional neural networking; TSM, temporal shifting module; CVS, critical view of safety.

Metrics produced

Metrics generated varied between studies and can be classified as either process or outcome based. Process-based metrics uniformly analysed motion and, in all but one study, were generated using computer vision tracking. Distance travelled and total movement of hands/instruments were the most common motions measured, although bimanual dexterity, speed, acceleration, and fluidity of movements were also reported. These metrics were generated for entire procedures, specific intraoperative phases, or individual tasks.

Outcome-based metrics were more diverse, largely generated from advanced AI-based analysis, and assessed either during specific phases of the operation or for individual tasks. Intraoperative phase durations were assessed in laparoscopic cholecystectomy and cataract surgery, whilst performance scores were generated for the quality of suturing in transanal total mesorectal excision (TaTME). Of the studies, two generated metrics that aimed to assess critical safety elements in laparoscopic cholecystectomy; these generated scores, to determine critical view of safety achievement, and heat maps, to demonstrate areas of safe and unsafe dissection. The full list of metrics produced is available in Table 3.

Application to training

The most common application to training was the use of metrics to discriminate between different skill levels (10 studies) for distinguishing trainees and attendings/experts. Of the studies, one compared different groups of trainees (novice versus intermediate). All studies were able to distinguish between groups with high levels of accuracy. In addition, five studies compared the metrics produced with an alternative assessment tool. These were based on expert assessment and included OSATS, Global Evaluative Assessment of Robotic Skills (GEARS), Objective Structured Assessment of Cataract Surgical Skill (OSACSS), and a performance rubric created by the study authors that had previously been validated using GEARS. All studies reported that the metrics produced from analysis correlated with the validated tool used. The number of human assessors ranged from one to six and the assessors were defined as being expert (2 studies) or board-certified surgeons (2 studies) (or no information was given (1 study)) and, of the studies relying on board-certified surgeons, one reported that the surgeons were experienced in the use of the AI model created to perform the video analysis. No study produced metrics for the assessment of non-technical skills or reported on how the metrics produced could be applied to assess non-technical skills. Furthermore, none of the included studies explored the impact on training outcomes (for example learning curves).

A qualitative study aimed to assess the educational value of using AI-generated heat maps to display areas of safe and unsafe dissection in laparoscopic cholecystectomy. Trainees and trainers strongly agreed (73%) that their use after surgery for coaching and feedback could be effective. Most suggested that heat maps should be incorporated in the surgical training curriculum, particularly for those at an earlier stage in training. However, 40% reported that, even if readily available, they would not use it routinely. Some raised concerns over potential dependence on it for making intraoperative decisions. Full results of the included studies are available in Table S3.

Discussion

This scoping review explores the current methods of automated analysis of intraoperative video in the context of surgical training. This is a rapidly evolving field, with most of the included studies being published within the last 5 years.

Several methods of automating the assessment of surgical video were identified. Computer vision, the broad term used to describe an algorithm’s ability to process and understand visual data²⁰, was used to generate kinematic data. Early techniques utilized tracking of objects between individual frames extracted from video^31,34,35 and others relied on the principle of point feature tracking^32,33,40,41. The included studies were effective at tracking movements of the surgeon’s hand during suturing and knot tying³¹ and tracking surgical instruments during cataract surgery⁴⁰. The need for manual identification of the ROI before automated tracking was a limitation in several studies^31,34,35. Human-dependent steps in these studies presented potential barriers to the scalability of the methods used. Minimizing inputs required from trainees, and trainers, to collect and analyse operative video not only improves engagement but also likely reduces the impact on cognitive load during the operation.

Newer approaches, utilizing AI, aim to overcome some of these barriers. CNN was able to automatically detect and track surgical instruments to assess motion^38,39,43. Surgical tool recognition is fundamental to this approach, with several studies reporting high accuracy with regard to identifying the presence or absence of instruments, as well as differentiating between instrument types^44,45. Furthermore, tool usage determined using AI correlates strongly with that determined by human assessment⁴⁶. Limitations of AI-based video analysis include the need for accurately annotated video for training AI models. However, the introduction of open data sets has diminished this obstacle⁴⁷.

It is not surprising that the most commonly produced metrics reflect kinematic data. Traditionally collected using sensors applied to the surgeon or instrument, kinematic data have long been used as markers of technical proficiency, due to strong correlation with expert-based assessment scores^48,49, even when deployed during complex operations⁵⁰. Insights into operative performance can be gained by analysing motion throughout an entire operation (for example total distance travelled) or within an individual task. Of the five studies that assessed individual tasks, four were based on motion analysis and included critical steps, such as suturing of bowel anastomoses and TaTME. Importantly, this granular technique is now being automatically assessed in both laparoscopic and robotic simulated settings^51,52.

Operative phase recognition is an emerging field of AI workflow analysis^53,54. The concept of surgical workflow analysis is based on the premise that an operation is rarely a singular event but rather a series of complex sequences that build upon one another. Variations in these sequences are more common for less experienced surgeons⁵⁵ and, thus, being able to efficiently analyse phases has great potential for improving post-hoc feedback. Several applications of phase analysis, such as duration of hepatocystic triangle dissection in laparoscopic cholecystectomy³⁷, tool usage in phacoemulsification during cataract surgery³², and skill determination during the first phase of robot-assisted thyroidectomy³⁹, were identified in the present review. Analysis of surgical gestures, defined as granular interactions between surgical instruments and human tissue⁵⁶, is another emerging field. Preliminary studies have identified that gesture type (for example to indicate ‘dissect’ or ‘coagulate’) and gesture efficiency (for example use of ineffective or erroneous gestures) can distinguish experts from novices^57–59. Most studies in this field are in the simulated setting; therefore, the potential application of surgical gestures to improve trainee surgical performance in the operating room remains unclear.

The majority of the included studies used the metrics produced from their analysis to differentiate between skill sets, a finding not consistent in the broader literature⁶⁰. A limitation of using this method in the included studies was that very few reported the level of training or the baseline skills of the participants. For example, the term ‘trainee’ is ambiguous and is equally applicable to someone in their first year of surgical training as to someone in their last. It may also be applied to fully certified surgeons learning a new procedure or approach (for example robotics).

Automatically generated metrics were also compared with validated, expert-based tools. Most studies compared metrics of similar properties (for example tool usage with bimanual dexterity); however, other extractable metrics from video, such as clearness of the operating field from blood, have also been shown to correlate with global expert-based assessments⁶¹. Although there is growing evidence to support the accuracy of automated assessments compared with expert-based methods^48,62, video-based assessments can only provide metrics from the material provided. In certain scenarios, wider contextual factors need consideration to guide accurate training and assessment. This has led some to advocate a hybrid approach, incorporating expert assessment with objective data generated from AI⁶³.

Currently evidence to explore the experiences of trainees and trainers using metrics generated from video analysis is lacking. Hameed et al.³⁶ report mixed opinions from trainees and trainers on an AI-based assessment of safe dissection planes in laparoscopic cholecystectomy. Most agreed that there was potential for use in surgical training; however, up to 40% said that, even if readily available, they would not routinely use it. Reservations are also shared by patients, with a recent survey reporting that, although most recognized the educational benefit of having their procedure recorded for training⁶⁴, concerns remained over non-maleficence, regarding data storage and access⁶⁵. Robust protocols are needed to satisfy these legitimate concerns and frameworks should be established to manage potential data breaches.

The present scoping review identifies the methods and metrics generated from automated video analysis of operative surgery. Establishing robust validity of automated video analytics is essential if this innovation is to help shape future training. It is also crucial that consensus be reached on which metrics of operative performance are most relevant for surgical trainees. No study reported longitudinal use of metrics by trainees to aid in their skill development. Training is a continuous process, requiring regular review of performance and goal setting. Understanding the longer-term impact of new technologies on training is a priority, so they can be applied in the most effective way. A second research priority is to address the lack of automated assessment of non-technical skills in surgery. Exploration of this in simulated and non-simulated settings is in progress^27,66.

The present review has some limitations. There were no restrictions with regard to surgical procedure or approach in the inclusion criteria, which likely increased heterogeneity in the findings. The metrics relevant for one procedure or skill are likely different from those required for another. It was surprising that MIS was not the most common surgical approach, given its video-based nature. Inclusion of only real-world operations may have resulted in under-representation of MIS, as research in MIS is still largely confined to the simulated setting, particularly in robotics.

The current landscape of automated video analytics in surgical training shows promise, both in terms of the metrics produced and the video analytics used. Additionally, there is a paucity of evidence with regard to understanding both the end user experience and the longitudinal benefit of automated video analysis for surgical trainees. To embrace this new technology, educators, curriculum developers, and professional bodies must support validation studies in training environments, before widespread adoption, for the benefit of trainees, trainers, and patients.

Funding

L.D. is supported by a Medical Education Fellowship funded by NHS Lothian. R.J.E.S. is funded by NHS Research Scotland (NRS) via a clinician post.

Acknowledgements

The authors would like to thank Marshall Dozier for assistance in generating the search criteria.

Author contributions

Lachlan Dick (Conceptualization, Data curation, Formal analysis, Methodology, Resources, Software, Visualization, Writing—original draft, Writing—review & editing), Connor P. Boyle (Data curation, Writing—review & editing), Richard J. E. Skipworth (Conceptualization, Supervision, Writing—review & editing), Douglas M. Smink (Writing—review & editing), Victoria R. Tallentire (Conceptualization, Supervision, Writing—review & editing), and Steven Yule (Conceptualization, Supervision, Writing—review & editing)

Disclosure

The authors declare no conflict of interest.

Supplementary material

Supplementary material is available at BJS Open online.

Data availability

Some data may be available upon reasonable request.

References

Haynes

Weiser

Berry

Lipsitz

Breizat

AHS

Dellinger

et al.

A surgical safety checklist to reduce morbidity and mortality in a global population

N Engl J Med

2009

;

360

491

–

499

Ou-Young

Boggett

El Ansary

Clarke-Errey

Royse

Bowyer

Identifying risk factors for poor multidimensional recovery after major surgery: a systematic review

Acta Anaesthesiol Scand

2023

;

1294

–

1305

Gawande

Zinner

Studdert

Brennan

Analysis of errors reported by surgeons at three teaching hospitals

Surgery

2003

;

133

614

–

621

Gawande

Thomas

Zinner

Brennan

The incidence and nature of surgical adverse events in Colorado and Utah in 1992

Surgery

1999

;

126

–

Birkmeyer

Finks

O’Reilly

Oerline

Carlin

Nunn

et al.

Surgical skill and complication rates after bariatric surgery

N Engl J Med

2013

;

369

1434

–

1442

Fecso

Bhatti

Stotland

Quereshy

Grantcharov

Technical performance as a predictor of clinical outcomes in laparoscopic gastric cancer surgery

Ann Surg

2019

;

270

115

–

120

Flin

Yule

Paterson-Brown

Maran

Rowley

Youngson

Teaching surgeons about non-technical skills

Surgeon

2007

;

–

Martin

Regehr

Reznick

Macrae

Murnaghan

Hutchison

et al.

Objective structured assessment of technical skill (OSATS) for surgical residents

Br J Surg

1997

;

273

–

278

PubMed

Niitsu

Hirabayashi

Yoshimitsu

Mimura

Taomoto

Sugiyama

et al.

Using the objective structured assessment of technical skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room

Surg Today

2013

;

271

–

275

Yule

Flin

Maran

Rowley

Youngson

Paterson-Brown

Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system

World J Surg

2008

;

548

–

556

St John

Caturegli

Kubicki

Kavic

The rise of minimally invasive surgery: 16 year analysis of the progressive replacement of open surgery with laparoscopy

JSLS

2020

;

e2020.00076

Friedl

Höppler

Ecard

Scholz

Hannekum

Stracke

Development and prospective evaluation of a multimedia teaching course on aortic valve replacement

Thorac Cardiovasc Surg

2006

;

–

Friedl

Höppler

Ecard

Scholz

Hannekum

Öchsner

et al.

Multimedia-driven teaching significantly improves students’ performance when compared with a print medium

Ann Thorac Surg

2006

;

1760

–

1766

Crawshaw

Steele

Lee

Delaney

Mustain

Russ

et al.

Failing to prepare is preparing to fail: a single-blinded, randomized controlled trial to determine the impact of a preoperative instructional video on the ability of residents to perform laparoscopic right colectomy

Dis Colon Rectum

2016

;

–

Singh

Aggarwal

Tahir

Pucher

Darzi

A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills

Ann Surg

2015

;

261

862

–

869

Driscoll

Paisley

Paterson-Brown

Video assessment of basic surgical trainees’ operative skills

Am J Surg

2008

;

196

265

–

272

Cauraugh

Martin

Modeling surgical expertise for motor skill acquisition

Am J Surg

1999

;

177

331

–

336

Andersen

SAW

Guldager

Mikkelsen

Sørensen

The effect of structured self-assessment in virtual reality simulation training of mastoidectomy

Eur Arch Otorhinolaryngol

2019

;

276

3345

–

3352

Varban

Thumma

Carlin

Ghaferi

Dimick

Finks

Evaluating the impact of surgeon self-awareness by comparing self versus peer ratings of surgical skill and outcomes for bariatric surgery

Ann Surg

2022

;

276

128

–

132

Hashimoto

Rosman

Rus

Meireles

Artificial intelligence in surgery: promises and perils

Ann Surg

2018

;

268

–

Hung

Chen

Che

Nilanon

Jarc

Titus

et al.

Utilizing machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict outcomes

J Endourol

2018

;

438

–

444

Lum

Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT

Clin Orthop Relat Res

2023

;

481

1623

–

1630

Rivas

Vázquez

Ruiz

Taratkin

Marenco

Cacciamani

et al.

Artificial intelligence and simulation in urology

Actas Urol Esp (Engl Ed)

2021

;

524

–

529

Arts

EEA

Leijte

Witteman

BPL

Jakimowicz

Verhoeven

Botden

SMBI

Face, content, and construct validity of the take-home eoSim augmented reality laparoscopy simulator for basic laparoscopic tasks

J Laparoendosc Adv Surg Tech

2019

;

1419

–

1426

Crossref

Hillemans

van de Mortel

Buyne

Verhoeven

Botden

SMBI

Objective assessment for open surgical suturing training by finger tracking can discriminate novices from experts

Med Ed

2023

;

2198818

Khalid

Goldenberg

Grantcharov

Taati

Rudzicz

Evaluation of deep learning models for identifying surgical actions and measuring performance

JAMA Netw Open

2020

;

e201664

Likosky

Yule

Mathis

Dias

Corso

Zhang

et al.

Novel assessments of technical and nontechnical cardiac surgery quality: protocol for a mixed methods study

JMIR Res Protoc

2021

;

e22536

Arksey

O’Malley

Scoping studies: towards a methodological framework

Int J Soc Res Methodol

2005

;

–

Crossref

Tricco

Lillie

Zarin

O’Brien

Colquhoun

Levac

et al.

PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation

Ann Intern Med

2018

;

169

467

–

473

Reed

Cook

Beckman

Levine

Kern

Wright

Association between funding and quality of published medical education research

JAMA

2007

;

298

1002

–

1009

Azari

Frasier

Quamme

Greenberg

Pugh

Greenberg

et al.

Modeling surgical technical skill using expert assessment for automated computer rating

Ann Surg

2019

;

269

574

–

581

Balal

Smith

Bader

Tang

Sullivan

Thomsen

et al.

Computer analysis of individual cataract surgery segments in the operating room

Eye

2019

;

313

–

319

Din

Smith

Emeriewen

Sharma

Jones

Wawrzynski

et al.

Man versus machine: software training for surgeons—an objective evaluation of human and computer-based training tools for cataract surgical performance

J Ophthalmol

2016

;

2016

3548039

PubMed

Frasier

Azari

Quamme

SRP

Radwin

Pugh

et al.

A marker-less technique for measuring kinematics in the operating room

Surgery

2016

;

160

1400

–

1413

Glarner

Chen

Radwin

Zhao

Craven

et al.

Quantifying technical skills during open operations using video-based motion analysis

Surgery

2014

;

156

729

–

734

Hameed

Laplante

Masino

Khalid

Zhang

Protserov

et al.

What is the educational value and clinical utility of artificial intelligence for intraoperative and postoperative video analysis? A survey of surgeons and trainees

Surg Endosc

2023

;

9453

–

9460

Humm

Peckham-Cooper

Hamade

Wood

Dawas

Stoyanov

et al.

Automated analysis of intraoperative phase in laparoscopic cholecystectomy: a comparison of one attending surgeon and their residents

J Surg Educ

2023

;

994

–

1004

Kitaguchi

Teramura

Matsuzaki

Hasegawa

Takeshita

Ito

Automatic purse-string suture skill assessment in transanal total mesorectal excision using deep learning-based video analysis

BJS Open

2023

;

zrac176

Lee

Kwon

Kong

Lee

Kim

Evaluation of surgical skills during robotic surgery by deep learning-based multiple surgical instrument tracking in training and actual operations

J Clin Med

2020

;

1964

Smith

Tang

Balntas

Young

Athanasiadis

Sullivan

et al.

”PhacoTracking”: an evolving paradigm in ophthalmic surgical training

JAMA Ophthalmol

2013

;

131

659

–

661

Wawrzynski

Smith

Tang

Hoare

Caputo

Siddiqui

et al.

Tracking camera control in endoscopic dacryocystorhinostomy surgery

Clin Otolaryngol

2015

;

646

–

650

Chen

Liu

Cao

Wei

et al.

SurgSmart: an artificial intelligent system for quality control in laparoscopic cholecystectomy: an observational study

Int J Surg

2023

;

109

1105

–

1114

Yang

Goodman

Dawes

Gahagan

Esquivel

Liebert

et al.

Using AI and computer vision to analyze technical proficiency in robotic surgery

Surg Endosc

2023

;

3010

–

3017

Demirel

Palmer

Sundberg

Karaman

Halic

Kockara

et al.

Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons

Int J Comput Assist Radiol Surg

2022

;

1823

–

1835

Jaafari

Douzi

Hssina

The impact of ensemble learning on surgical tools classification during laparoscopic cholecystectomy

J Big Data

2022

;

Crossref

Yamazaki

Kanaji

Kudo

Takiguchi

Urakawa

Hasegawa

et al.

Quantitative comparison of surgical device usage in laparoscopic gastrectomy between surgeons’ skill levels: an automated analysis using a neural network

J Gastrointest Surg

2022

;

1006

–

1014

Ríos

Molina-Rodriguez

Londoño

Guillén

Sierra

Zapata

et al.

Cholec80-CVS: an open dataset with an evaluation of Strasberg’s critical view of safety for AI

Sci Data

2023

;

194

Aggarwal

Grantcharov

Moorthy

Milland

Papasavas

Dosis

et al.

An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room

Ann Surg

2007

;

245

992

–

999

Chen

Yen

Radwin

Automated video exposure assessment of repetitive hand activity level for a load transfer task

Hum Factors

2013

;

298

–

308

Ghodoussipour

Reddy

Huang

Nguyen

Hung

An objective assessment of performance during robotic partial nephrectomy: validation and correlation of automated performance metrics with intraoperative outcomes

J Urol

2021

;

205

1294

–

1302

Oropesa

Sánchez-González

Chmarra

Lamata

Fernández

Sánchez-Margallo

et al.

EVA: laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment

Surg Endosc

2013

;

1029

–

1039

Peng

Xing

Liu

Zhang

An automatic skill evaluation framework for robotic surgery training

Int J Med Robot

2019

;

e1964

Hashimoto

Rosman

Witkowski

Stafford

Navarrete-Welton

Rattner

et al.

Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy

Ann Surg

2019

;

270

414

–

421

Hung

Chen

Ghodoussipour

Lane

Jarc

et al.

Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy

BJU Int

2019

;

123

861

–

868

Forestier

Petitjean

Senin

Riffaud

Henaux

Jannin

Finding discriminative and interpretable patterns in sequences of surgical activities

Artif Intell Med

2017

;

–

Ramaswamy

Trinh

Kiyasseh

Chu

et al.

Surgical gestures as a method to quantify surgical performance and predict patient outcomes

NPJ Digit Med

2022

;

187

Inouye

Nguyen

Laca

Kocielnik

Anandkumar

et al.

Assessing the efficacy of dissection gestures in robotic surgery

J Robot Surg

2023

;

597

–

603

Vanstrum

Nguyen

Chen

Hung

A novel dissection gesture classification to characterize robotic dissection technique for renal hilar dissection

J Urol

2021

;

205

271

–

275

Baghdadi

Hussein

Ahmed

Cavuoto

Guru

A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos

Int J Comput Assist Radiol Surg

2019

;

697

–

707

Lin

Qin

Bly

Moe

Hannaford

Automatic sinus surgery skill assessment based on instrument segmentation and tracking in endoscopic video. In:

Leahy

Dong

(eds),

Multiscale Multimodal Medical Imaging

Cham

Springer International Publishing

2020

–

100

(

Lecture Notes in Computer Science

)

Liu

Jiang

Wang

Miao

Shan

Surgical skill assessment on in-vivo clinical data via the clearness of operating field. In:

Shen

Liu

Peters

Staib

Essert

Zhou

et al. (eds),

Medical Image Computing and Computer Assisted Intervention—MICCAI 2019

Cham

Springer International Publishing

2019

476

–

484

(

Lecture Notes in Computer Science

)

Google Preview

(https://proceedings.mlr.press/v68/law17a.html)

Law

Ghani

Deng

Surgeon technical skill assessment using computer vision based analysis. In:

Proceedings of the Second Machine Learning for Healthcare Conference

PMLR

2017

;

–

Google Preview