Table 2. Open in new tab Comparative...

Method	Input	Total	Direct	Reverse	Symmetry
DDGemb	SEQ	0.68	1.40	0.99	0.53	1.40	0.99	0.52	1.40	0.99	−0.97	0.01
PROSTATA	SEQ	0.65	1.45	1.00	0.49	1.45	1.00	0.49	1.45	0.99	−0.99	−0.01
ACDC-NN	3D	0.61	1.5	1.05	0.46	1.49	1.05	0.45	1.5	1.06	−0.98	0.02
INPS-Seq	SEQ	0.61	1.52	1.1	0.43	1.52	1.09	0.43	1.53	1.1	−1.00	0.00
PremPS	3D	0.62	1.49	1.07	0.41	1.5	1.08	0.42	1.49	1.05	−0.85	0.09
ACDC-NN-Seq	SEQ	0.59	1.53	1.08	0.42	1.53	1.08	0.42	1.53	1.08	−1.00	0.00
DDGun3D	3D	0.57	1.61	1.13	0.43	1.6	1.11	0.41	1.62	1.14	−0.97	0.05
INPS3D	3D	0.55	1.64	1.19	0.43	1.5	1.07	0.33	1.77	1.31	−0.5	0.38
THPLM	SEQ	0.53	1.63		0.39	1.60		0.35	1.66		−0.96	−0.01
ThermoNet	3D	0.51	1.64	1.2	0.39	1.62	1.17	0.38	1.66	1.23	−0.85	0.05
DDGun	SEQ	0.57	1.74	1.25	0.41	1.72	1.25	0.38	1.75	1.25	−0.96	0.05
MAESTRO	3D	0.44	1.8	1.3	0.5	1.44	1.06	0.2	2.1	1.66	−0.22	0.57
ThermoMPNN	SEQ	0.43	1.52
Dynamut	3D	0.5	1.65	1.21	0.41	1.6	1.19	0.34	1.69	1.24	−0.58	0.06
PoPMuSiC	3D	0.46	1.82	1.37	0.41	1.51	1.09	0.24	2.09	1.64	−0.32	0.69
DUET	3D	0.41	1.86	1.39	0.41	1.52	1.1	0.23	2.14	1.68	−0.12	0.67
I-Mutant3.0-Seq	SEQ	0.37	1.91	1.47	0.34	1.54	1.15	0.22	2.22	1.79	−0.48	0.76
SDM	3D	0.32	1.93	1.45	0.41	1.67	1.26	0.13	2.16	1.64	−0.4	0.4
mCSM	3D	0.37	1.96	1.49	0.36	1.54	1.13	0.22	2.3	1.86	−0.05	0.85
Dynamut2	3D	0.36	1.9	1.42	0.34	1.58	1.15	0.17	2.16	1.69	0.03	0.64
I-Mutant3.0	3D	0.32	1.96	1.49	0.36	1.52	1.12	0.15	2.32	1.87	−0.06	0.81
Rosetta	3D	0.47	2.69	2.05	0.39	2.7	2.08	0.4	2.68	2.02	−0.72	0.61
FoldX	3D	0.31	2.39	1.53	0.22	2.3	1.56	0.22	2.48	1.5	−0.2	0.34
SAAFEC-SEQ	SEQ	0.26	2.02	1.54	0.36	1.54	1.13	−0.01	2.4	1.94	−0.03	0.83
MUpro	SEQ	0.32	2.03	1.58	0.25	1.61	1.21	0.20	2.38	1.96	−0.32	0.95

Method

Input

Total

Direct

Reverse

Symmetry

PCC

RMSE

MAE

PCC

RMSE

MAE

PCC

RMSE

MAE

r_d-r

⟨δ⟩

DDGemb

SEQ

0.68

1.40

0.99

0.53

1.40

0.99

0.52

1.40

0.99

−0.97

0.01

PROSTATA

SEQ

0.65

1.45

1.00

0.49

1.45

1.00

0.49

1.45

0.99

−0.99

−0.01

ACDC-NN

0.61

1.5

1.05

0.46

1.49

1.05

0.45

1.5

1.06

−0.98

0.02

INPS-Seq

SEQ

0.61

1.52

1.1

0.43

1.52

1.09

0.43

1.53

1.1

−1.00

0.00

PremPS

0.62

1.49

1.07

0.41

1.5

1.08

0.42

1.49

1.05

−0.85

0.09

ACDC-NN-Seq

SEQ

0.59

1.53

1.08

0.42

1.53

1.08

0.42

1.53

1.08

−1.00

0.00

DDGun3D

0.57

1.61

1.13

0.43

1.6

1.11

0.41

1.62

1.14

−0.97

0.05

INPS3D

0.55

1.64

1.19

0.43

1.5

1.07

0.33

1.77

1.31

−0.5

0.38

THPLM

SEQ

0.53

1.63

0.39

1.60

0.35

1.66

−0.96

−0.01

ThermoNet

0.51

1.64

1.2

0.39

1.62

1.17

0.38

1.66

1.23

−0.85

0.05

DDGun

SEQ

0.57

1.74

1.25

0.41

1.72

1.25

0.38

1.75

1.25

−0.96

0.05

MAESTRO

0.44

1.8

1.3

0.5

1.44

1.06

0.2

2.1

1.66

−0.22

0.57

ThermoMPNN

SEQ

0.43

1.52

Dynamut

0.5

1.65

1.21

0.41

1.6

1.19

0.34

1.69

1.24

−0.58

0.06

PoPMuSiC

0.46

1.82

1.37

0.41

1.51

1.09

0.24

2.09

1.64

−0.32

0.69

DUET

0.41

1.86

1.39

0.41

1.52

1.1

0.23

2.14

1.68

−0.12

0.67

I-Mutant3.0-Seq

SEQ

0.37

1.91

1.47

0.34

1.54

1.15

0.22

2.22

1.79

−0.48

0.76

SDM

0.32

1.93

1.45

0.41

1.67

1.26

0.13

2.16

1.64

−0.4

0.4

mCSM

0.37

1.96

1.49

0.36

1.54

1.13

0.22

2.3

1.86

−0.05

0.85

Dynamut2

0.36

1.9

1.42

0.34

1.58

1.15

0.17

2.16

1.69

0.03

0.64

I-Mutant3.0

0.32

1.96

1.49

0.36

1.52

1.12

0.15

2.32

1.87

−0.06

0.81

Rosetta

0.47

2.69

2.05

0.39

2.7

2.08

0.4

2.68

2.02

−0.72

0.61

FoldX

0.31

2.39

1.53

0.22

2.3

1.56

0.22

2.48

1.5

−0.2

0.34

SAAFEC-SEQ

SEQ

0.26

2.02

1.54

0.36

1.54

1.13

−0.01

2.4

1.94

−0.03

0.83

MUpro

SEQ

0.32

2.03

1.58

0.25

1.61

1.21

0.20

2.38

1.96

−0.32

0.95

Method	Input	Total	Direct	Reverse	Symmetry
DDGemb	SEQ	0.68	1.40	0.99	0.53	1.40	0.99	0.52	1.40	0.99	−0.97	0.01
PROSTATA	SEQ	0.65	1.45	1.00	0.49	1.45	1.00	0.49	1.45	0.99	−0.99	−0.01
ACDC-NN	3D	0.61	1.5	1.05	0.46	1.49	1.05	0.45	1.5	1.06	−0.98	0.02
INPS-Seq	SEQ	0.61	1.52	1.1	0.43	1.52	1.09	0.43	1.53	1.1	−1.00	0.00
PremPS	3D	0.62	1.49	1.07	0.41	1.5	1.08	0.42	1.49	1.05	−0.85	0.09
ACDC-NN-Seq	SEQ	0.59	1.53	1.08	0.42	1.53	1.08	0.42	1.53	1.08	−1.00	0.00
DDGun3D	3D	0.57	1.61	1.13	0.43	1.6	1.11	0.41	1.62	1.14	−0.97	0.05
INPS3D	3D	0.55	1.64	1.19	0.43	1.5	1.07	0.33	1.77	1.31	−0.5	0.38
THPLM	SEQ	0.53	1.63		0.39	1.60		0.35	1.66		−0.96	−0.01
ThermoNet	3D	0.51	1.64	1.2	0.39	1.62	1.17	0.38	1.66	1.23	−0.85	0.05
DDGun	SEQ	0.57	1.74	1.25	0.41	1.72	1.25	0.38	1.75	1.25	−0.96	0.05
MAESTRO	3D	0.44	1.8	1.3	0.5	1.44	1.06	0.2	2.1	1.66	−0.22	0.57
ThermoMPNN	SEQ	0.43	1.52
Dynamut	3D	0.5	1.65	1.21	0.41	1.6	1.19	0.34	1.69	1.24	−0.58	0.06
PoPMuSiC	3D	0.46	1.82	1.37	0.41	1.51	1.09	0.24	2.09	1.64	−0.32	0.69
DUET	3D	0.41	1.86	1.39	0.41	1.52	1.1	0.23	2.14	1.68	−0.12	0.67
I-Mutant3.0-Seq	SEQ	0.37	1.91	1.47	0.34	1.54	1.15	0.22	2.22	1.79	−0.48	0.76
SDM	3D	0.32	1.93	1.45	0.41	1.67	1.26	0.13	2.16	1.64	−0.4	0.4
mCSM	3D	0.37	1.96	1.49	0.36	1.54	1.13	0.22	2.3	1.86	−0.05	0.85
Dynamut2	3D	0.36	1.9	1.42	0.34	1.58	1.15	0.17	2.16	1.69	0.03	0.64
I-Mutant3.0	3D	0.32	1.96	1.49	0.36	1.52	1.12	0.15	2.32	1.87	−0.06	0.81
Rosetta	3D	0.47	2.69	2.05	0.39	2.7	2.08	0.4	2.68	2.02	−0.72	0.61
FoldX	3D	0.31	2.39	1.53	0.22	2.3	1.56	0.22	2.48	1.5	−0.2	0.34
SAAFEC-SEQ	SEQ	0.26	2.02	1.54	0.36	1.54	1.13	−0.01	2.4	1.94	−0.03	0.83
MUpro	SEQ	0.32	2.03	1.58	0.25	1.61	1.21	0.20	2.38	1.96	−0.32	0.95

Table 2.

Open in new tab

Comparative benchmark of different sequence- and structure-based methods on the S669 independent test set of single-point variations.

Method	Input	Total			Direct			Reverse			Symmetry
Method	Input	PCC	RMSE	MAE	PCC	RMSE	MAE	PCC	RMSE	MAE	r_d-r	⟨δ⟩
DDGemb	SEQ	0.68	1.40	0.99	0.53	1.40	0.99	0.52	1.40	0.99	−0.97	0.01
PROSTATA	SEQ	0.65	1.45	1.00	0.49	1.45	1.00	0.49	1.45	0.99	−0.99	−0.01
ACDC-NN	3D	0.61	1.5	1.05	0.46	1.49	1.05	0.45	1.5	1.06	−0.98	0.02
INPS-Seq	SEQ	0.61	1.52	1.1	0.43	1.52	1.09	0.43	1.53	1.1	−1.00	0.00
PremPS	3D	0.62	1.49	1.07	0.41	1.5	1.08	0.42	1.49	1.05	−0.85	0.09
ACDC-NN-Seq	SEQ	0.59	1.53	1.08	0.42	1.53	1.08	0.42	1.53	1.08	−1.00	0.00
DDGun3D	3D	0.57	1.61	1.13	0.43	1.6	1.11	0.41	1.62	1.14	−0.97	0.05
INPS3D	3D	0.55	1.64	1.19	0.43	1.5	1.07	0.33	1.77	1.31	−0.5	0.38
THPLM	SEQ	0.53	1.63		0.39	1.60		0.35	1.66		−0.96	−0.01
ThermoNet	3D	0.51	1.64	1.2	0.39	1.62	1.17	0.38	1.66	1.23	−0.85	0.05
DDGun	SEQ	0.57	1.74	1.25	0.41	1.72	1.25	0.38	1.75	1.25	−0.96	0.05
MAESTRO	3D	0.44	1.8	1.3	0.5	1.44	1.06	0.2	2.1	1.66	−0.22	0.57
ThermoMPNN	SEQ	0.43	1.52
Dynamut	3D	0.5	1.65	1.21	0.41	1.6	1.19	0.34	1.69	1.24	−0.58	0.06
PoPMuSiC	3D	0.46	1.82	1.37	0.41	1.51	1.09	0.24	2.09	1.64	−0.32	0.69
DUET	3D	0.41	1.86	1.39	0.41	1.52	1.1	0.23	2.14	1.68	−0.12	0.67
I-Mutant3.0-Seq	SEQ	0.37	1.91	1.47	0.34	1.54	1.15	0.22	2.22	1.79	−0.48	0.76
SDM	3D	0.32	1.93	1.45	0.41	1.67	1.26	0.13	2.16	1.64	−0.4	0.4
mCSM	3D	0.37	1.96	1.49	0.36	1.54	1.13	0.22	2.3	1.86	−0.05	0.85
Dynamut2	3D	0.36	1.9	1.42	0.34	1.58	1.15	0.17	2.16	1.69	0.03	0.64
I-Mutant3.0	3D	0.32	1.96	1.49	0.36	1.52	1.12	0.15	2.32	1.87	−0.06	0.81
Rosetta	3D	0.47	2.69	2.05	0.39	2.7	2.08	0.4	2.68	2.02	−0.72	0.61
FoldX	3D	0.31	2.39	1.53	0.22	2.3	1.56	0.22	2.48	1.5	−0.2	0.34
SAAFEC-SEQ	SEQ	0.26	2.02	1.54	0.36	1.54	1.13	−0.01	2.4	1.94	−0.03	0.83
MUpro	SEQ	0.32	2.03	1.58	0.25	1.61	1.21	0.20	2.38	1.96	−0.32	0.95

Method	Input	Total			Direct			Reverse			Symmetry
Method	Input	PCC	RMSE	MAE	PCC	RMSE	MAE	PCC	RMSE	MAE	r_d-r	⟨δ⟩
DDGemb	SEQ	0.68	1.40	0.99	0.53	1.40	0.99	0.52	1.40	0.99	−0.97	0.01
PROSTATA	SEQ	0.65	1.45	1.00	0.49	1.45	1.00	0.49	1.45	0.99	−0.99	−0.01
ACDC-NN	3D	0.61	1.5	1.05	0.46	1.49	1.05	0.45	1.5	1.06	−0.98	0.02
INPS-Seq	SEQ	0.61	1.52	1.1	0.43	1.52	1.09	0.43	1.53	1.1	−1.00	0.00
PremPS	3D	0.62	1.49	1.07	0.41	1.5	1.08	0.42	1.49	1.05	−0.85	0.09
ACDC-NN-Seq	SEQ	0.59	1.53	1.08	0.42	1.53	1.08	0.42	1.53	1.08	−1.00	0.00
DDGun3D	3D	0.57	1.61	1.13	0.43	1.6	1.11	0.41	1.62	1.14	−0.97	0.05
INPS3D	3D	0.55	1.64	1.19	0.43	1.5	1.07	0.33	1.77	1.31	−0.5	0.38
THPLM	SEQ	0.53	1.63		0.39	1.60		0.35	1.66		−0.96	−0.01
ThermoNet	3D	0.51	1.64	1.2	0.39	1.62	1.17	0.38	1.66	1.23	−0.85	0.05
DDGun	SEQ	0.57	1.74	1.25	0.41	1.72	1.25	0.38	1.75	1.25	−0.96	0.05
MAESTRO	3D	0.44	1.8	1.3	0.5	1.44	1.06	0.2	2.1	1.66	−0.22	0.57
ThermoMPNN	SEQ	0.43	1.52
Dynamut	3D	0.5	1.65	1.21	0.41	1.6	1.19	0.34	1.69	1.24	−0.58	0.06
PoPMuSiC	3D	0.46	1.82	1.37	0.41	1.51	1.09	0.24	2.09	1.64	−0.32	0.69
DUET	3D	0.41	1.86	1.39	0.41	1.52	1.1	0.23	2.14	1.68	−0.12	0.67
I-Mutant3.0-Seq	SEQ	0.37	1.91	1.47	0.34	1.54	1.15	0.22	2.22	1.79	−0.48	0.76
SDM	3D	0.32	1.93	1.45	0.41	1.67	1.26	0.13	2.16	1.64	−0.4	0.4
mCSM	3D	0.37	1.96	1.49	0.36	1.54	1.13	0.22	2.3	1.86	−0.05	0.85
Dynamut2	3D	0.36	1.9	1.42	0.34	1.58	1.15	0.17	2.16	1.69	0.03	0.64
I-Mutant3.0	3D	0.32	1.96	1.49	0.36	1.52	1.12	0.15	2.32	1.87	−0.06	0.81
Rosetta	3D	0.47	2.69	2.05	0.39	2.7	2.08	0.4	2.68	2.02	−0.72	0.61
FoldX	3D	0.31	2.39	1.53	0.22	2.3	1.56	0.22	2.48	1.5	−0.2	0.34
SAAFEC-SEQ	SEQ	0.26	2.02	1.54	0.36	1.54	1.13	−0.01	2.4	1.94	−0.03	0.83
MUpro	SEQ	0.32	2.03	1.58	0.25	1.61	1.21	0.20	2.38	1.96	−0.32	0.95

Results for all methods except DDGemb, THPLM, ThermoMPNN, and PROSTATA were taken from (Pancotti et al. 2022). For PROSTATA and THPLM direct and reverse PCC, RMSE, and MAE were taken from the reference papers (Gong et al. 2023, Umerenkov et al. 2023). ThermoMPNN results were taken from Dieckhaus et al. (2024) PROSTATA total PCC, Total RMSE, Total MAE, ${PCC}_{d - r}$ ⁠, and $〈δ〉$ were computed using the predictions available at the method GitHub repository. Bold values highlight the top-performing method(s) on the respective metric.

This Feature Is Available To Subscribers Only