Table 4 Open in new tab System...

System performance measure (%)	System output versus gold standard annotation	System-assisted annotations	Manual annotation
Textpresso
Sentence level
Category 4^a	System alone
Recall	37.9
Precision	77.5					Curator 1^b	Curator 2^b
F-measure	50.9					55.1	26.9
Category 5^a	System alone					41.7	63.3
Recall	39.7					47.5	37.8
Precision	81.5
F-measure	53.4
GO annotation level
Category 4^a		Curator 1	Curator 2
Recall		37.1	14.5			Curator 1^b	Curator 2^b
Precision		78.3	77.8			86.8	39.5
F-measure		50.3	24.4			42.8	41.2
Category 5^a		Curator 1	Curator 2			57.3	40.3
Recall		32.2	11.3
Precision		75.0	71.4
F-measure		45.1	19.5
PCS
Term-based EQs^c	System alone					Curator 1	Curator 2^d	Curator 3
Recall	65.0					47.0	38.0	50.0
Precision	60.0					57.0	65.0	67.0
F-measure	62.4					51.5	48.0	57.3
Label-based EQs^c	System alone					Curator 1	curator 2^d	Curator 3
Recall	24.0					44.0	51.0	51.0
Precision	23.0					54.0	81.0	74.0
F-measure	23.5					48.5	62.6	60.4
		Phenex + Charaparser	Phenex
Label-based EQs^c		Curator 1	Curator 2^d	Curator 3		Curator 1	Curator 2^d	Curator 3
Recall		51.0	38.0	66.0		37.0	63.0	36.0
Precision		58.0	70.0	84.0		49.0	88.0	60.0
F-measure		54.3	49.3	73.9		42.2	73.4	45.0
PubTator
NLM indexing mention-level	System alone	Curator 1				Curator 1
Recall	80.1	98.6				91.0
Precision	83.4	98.3				93.0
F-measure	81.7	98.0				92.0
TAIR indexing document level	System alone	Curator 2				Curator 2
Recall	76.0	90.0				91.0
Precision	73.9	77.1				75.0
F-measure	74.9	83.0				82.0
TAIR triage	System alone	Curator 2
Recall	68.6	84.6
Precision	80.5	100.0
F-measure	74.1	92.0
PPInterFinder
PPI algorithm alone	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.8	63.8			72.7	79.7
Precision	85.7	85.7			87.0	90.4
F-measure	76.9	73.2			79.2	84.7
PPI algorithm (gene mention/ gene normalization)	System alone	Curator 1	Curator 2
Recall	NR	46.9	46.9
Precision	85.7	85.7
F-measure	60.6	60.6
eFIP
PMID-centric (sentence level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.2	88.2			89.5	77.8
Precision	94.7	79.0			85.0	70.0
F-measure	80.0	83.3			87.2	73.7
Gene-centric (document level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	78.6	85.7			100.0	77.8
Precision	91.7	85.7			83.3	77.8
F-measure	84.6	85.7			90.9	77.8
Document-ranking
nDCG	93–100
T-HOD
PMID-centric (sentence level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	70.0	56.0	22.0	24.0	42.0
Precision	79.5	32.0	26.0	40.0	42.0
F-measure	74.5	40.0	24.0	30.0	42.0
Gene-centric (document level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	54.3	56.0	30.0	26.0	42.0
Precision	72.1	63.0	41.0	52.0	71.0
F-measure	62.0	59.0	35.0	35.0	53.0

System performance measure (%)

System output versus gold standard annotation

System-assisted annotations

Manual annotation

Textpresso

Sentence level

Category 4^a

System alone

Recall

37.9

Precision

77.5

Curator 1^b

Curator 2^b

F-measure

50.9

55.1

26.9

Category 5^a

System alone

41.7

63.3

Recall

39.7

47.5

37.8

Precision

81.5

F-measure

53.4

GO annotation level

Category 4^a

Curator 1

Curator 2

Recall

37.1

14.5

Curator 1^b

Curator 2^b

Precision

78.3

77.8

86.8

39.5

F-measure

50.3

24.4

42.8

41.2

Category 5^a

Curator 1

Curator 2

57.3

40.3

Recall

32.2

11.3

Precision

75.0

71.4

F-measure

45.1

19.5

PCS

Term-based EQs^c

System alone

Curator 1

Curator 2^d

Curator 3

Recall

65.0

47.0

38.0

50.0

Precision

60.0

57.0

65.0

67.0

F-measure

62.4

51.5

48.0

57.3

Label-based EQs^c

System alone

Curator 1

curator 2^d

Curator 3

Recall

24.0

44.0

51.0

Precision

23.0

54.0

81.0

74.0

F-measure

23.5

48.5

62.6

60.4

Phenex + Charaparser

Phenex

Label-based EQs^c

Curator 1

Curator 2^d

Curator 3

Curator 1

Curator 2^d

Curator 3

Recall

51.0

38.0

66.0

37.0

63.0

36.0

Precision

58.0

70.0

84.0

49.0

88.0

60.0

F-measure

54.3

49.3

73.9

42.2

73.4

45.0

PubTator

NLM indexing mention-level

System alone

Curator 1

Recall

80.1

98.6

91.0

Precision

83.4

98.3

93.0

F-measure

81.7

98.0

92.0

TAIR indexing document level

System alone

Curator 2

Recall

76.0

90.0

91.0

Precision

73.9

77.1

75.0

F-measure

74.9

83.0

82.0

TAIR triage

System alone

Curator 2

Recall

68.6

84.6

Precision

80.5

100.0

F-measure

74.1

92.0

PPInterFinder

PPI algorithm alone

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

69.8

63.8

72.7

79.7

Precision

85.7

87.0

90.4

F-measure

76.9

73.2

79.2

84.7

PPI algorithm (gene mention/ gene normalization)

System alone

Curator 1

Curator 2

Recall

46.9

Precision

85.7

F-measure

60.6

eFIP

PMID-centric (sentence level)

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

69.2

88.2

89.5

77.8

Precision

94.7

79.0

85.0

70.0

F-measure

80.0

83.3

87.2

73.7

Gene-centric (document level)

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

78.6

85.7

100.0

77.8

Precision

91.7

85.7

83.3

77.8

F-measure

84.6

85.7

90.9

77.8

Document-ranking

nDCG

93–100

T-HOD

PMID-centric (sentence level)

System alone

Curator 1

Curator 2

Curator 3

Curator 4

Recall

70.0

56.0

22.0

24.0

42.0

Precision

79.5

32.0

26.0

40.0

42.0

F-measure

74.5

40.0

24.0

30.0

42.0

Gene-centric (document level)

System alone

Curator 1

Curator 2

Curator 3

Curator 4

Recall

54.3

56.0

30.0

26.0

42.0

Precision

72.1

63.0

41.0

52.0

71.0

F-measure

62.0

59.0

35.0

53.0

System performance measure (%)	System output versus gold standard annotation	System-assisted annotations	Manual annotation
Textpresso
Sentence level
Category 4^a	System alone
Recall	37.9
Precision	77.5					Curator 1^b	Curator 2^b
F-measure	50.9					55.1	26.9
Category 5^a	System alone					41.7	63.3
Recall	39.7					47.5	37.8
Precision	81.5
F-measure	53.4
GO annotation level
Category 4^a		Curator 1	Curator 2
Recall		37.1	14.5			Curator 1^b	Curator 2^b
Precision		78.3	77.8			86.8	39.5
F-measure		50.3	24.4			42.8	41.2
Category 5^a		Curator 1	Curator 2			57.3	40.3
Recall		32.2	11.3
Precision		75.0	71.4
F-measure		45.1	19.5
PCS
Term-based EQs^c	System alone					Curator 1	Curator 2^d	Curator 3
Recall	65.0					47.0	38.0	50.0
Precision	60.0					57.0	65.0	67.0
F-measure	62.4					51.5	48.0	57.3
Label-based EQs^c	System alone					Curator 1	curator 2^d	Curator 3
Recall	24.0					44.0	51.0	51.0
Precision	23.0					54.0	81.0	74.0
F-measure	23.5					48.5	62.6	60.4
		Phenex + Charaparser	Phenex
Label-based EQs^c		Curator 1	Curator 2^d	Curator 3		Curator 1	Curator 2^d	Curator 3
Recall		51.0	38.0	66.0		37.0	63.0	36.0
Precision		58.0	70.0	84.0		49.0	88.0	60.0
F-measure		54.3	49.3	73.9		42.2	73.4	45.0
PubTator
NLM indexing mention-level	System alone	Curator 1				Curator 1
Recall	80.1	98.6				91.0
Precision	83.4	98.3				93.0
F-measure	81.7	98.0				92.0
TAIR indexing document level	System alone	Curator 2				Curator 2
Recall	76.0	90.0				91.0
Precision	73.9	77.1				75.0
F-measure	74.9	83.0				82.0
TAIR triage	System alone	Curator 2
Recall	68.6	84.6
Precision	80.5	100.0
F-measure	74.1	92.0
PPInterFinder
PPI algorithm alone	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.8	63.8			72.7	79.7
Precision	85.7	85.7			87.0	90.4
F-measure	76.9	73.2			79.2	84.7
PPI algorithm (gene mention/ gene normalization)	System alone	Curator 1	Curator 2
Recall	NR	46.9	46.9
Precision	85.7	85.7
F-measure	60.6	60.6
eFIP
PMID-centric (sentence level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.2	88.2			89.5	77.8
Precision	94.7	79.0			85.0	70.0
F-measure	80.0	83.3			87.2	73.7
Gene-centric (document level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	78.6	85.7			100.0	77.8
Precision	91.7	85.7			83.3	77.8
F-measure	84.6	85.7			90.9	77.8
Document-ranking
nDCG	93–100
T-HOD
PMID-centric (sentence level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	70.0	56.0	22.0	24.0	42.0
Precision	79.5	32.0	26.0	40.0	42.0
F-measure	74.5	40.0	24.0	30.0	42.0
Gene-centric (document level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	54.3	56.0	30.0	26.0	42.0
Precision	72.1	63.0	41.0	52.0	71.0
F-measure	62.0	59.0	35.0	35.0	53.0

System performance measure (%)

System output versus gold standard annotation

System-assisted annotations

Manual annotation

Textpresso

Sentence level

Category 4^a

System alone

Recall

37.9

Precision

77.5

Curator 1^b

Curator 2^b

F-measure

50.9

55.1

26.9

Category 5^a

System alone

41.7

63.3

Recall

39.7

47.5

37.8

Precision

81.5

F-measure

53.4

GO annotation level

Category 4^a

Curator 1

Curator 2

Recall

37.1

14.5

Curator 1^b

Curator 2^b

Precision

78.3

77.8

86.8

39.5

F-measure

50.3

24.4

42.8

41.2

Category 5^a

Curator 1

Curator 2

57.3

40.3

Recall

32.2

11.3

Precision

75.0

71.4

F-measure

45.1

19.5

PCS

Term-based EQs^c

System alone

Curator 1

Curator 2^d

Curator 3

Recall

65.0

47.0

38.0

50.0

Precision

60.0

57.0

65.0

67.0

F-measure

62.4

51.5

48.0

57.3

Label-based EQs^c

System alone

Curator 1

curator 2^d

Curator 3

Recall

24.0

44.0

51.0

Precision

23.0

54.0

81.0

74.0

F-measure

23.5

48.5

62.6

60.4

Phenex + Charaparser

Phenex

Label-based EQs^c

Curator 1

Curator 2^d

Curator 3

Curator 1

Curator 2^d

Curator 3

Recall

51.0

38.0

66.0

37.0

63.0

36.0

Precision

58.0

70.0

84.0

49.0

88.0

60.0

F-measure

54.3

49.3

73.9

42.2

73.4

45.0

PubTator

NLM indexing mention-level

System alone

Curator 1

Recall

80.1

98.6

91.0

Precision

83.4

98.3

93.0

F-measure

81.7

98.0

92.0

TAIR indexing document level

System alone

Curator 2

Recall

76.0

90.0

91.0

Precision

73.9

77.1

75.0

F-measure

74.9

83.0

82.0

TAIR triage

System alone

Curator 2

Recall

68.6

84.6

Precision

80.5

100.0

F-measure

74.1

92.0

PPInterFinder

PPI algorithm alone

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

69.8

63.8

72.7

79.7

Precision

85.7

87.0

90.4

F-measure

76.9

73.2

79.2

84.7

PPI algorithm (gene mention/ gene normalization)

System alone

Curator 1

Curator 2

Recall

46.9

Precision

85.7

F-measure

60.6

eFIP

PMID-centric (sentence level)

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

69.2

88.2

89.5

77.8

Precision

94.7

79.0

85.0

70.0

F-measure

80.0

83.3

87.2

73.7

Gene-centric (document level)

System alone

Curator 1

Curator 2

Curator 1

Curator 2

Recall

78.6

85.7

100.0

77.8

Precision

91.7

85.7

83.3

77.8

F-measure

84.6

85.7

90.9

77.8

Document-ranking

nDCG

93–100

T-HOD

PMID-centric (sentence level)

System alone

Curator 1

Curator 2

Curator 3

Curator 4

Recall

70.0

56.0

22.0

24.0

42.0

Precision

79.5

32.0

26.0

40.0

42.0

F-measure

74.5

40.0

24.0

30.0

42.0

Gene-centric (document level)

System alone

Curator 1

Curator 2

Curator 3

Curator 4

Recall

54.3

56.0

30.0

26.0

42.0

Precision

72.1

63.0

41.0

52.0

71.0

F-measure

62.0

59.0

35.0

53.0

Table 4

Open in new tab

System performance metrics in pre-workshop evaluation

System performance measure (%)	System output versus gold standard annotation	System-assisted annotations				Manual annotation
Textpresso
Sentence level
Category 4^a	System alone
Recall	37.9
Precision	77.5					Curator 1^b	Curator 2^b
F-measure	50.9					55.1	26.9
Category 5^a	System alone					41.7	63.3
Recall	39.7					47.5	37.8
Precision	81.5
F-measure	53.4
GO annotation level
Category 4^a		Curator 1	Curator 2
Recall		37.1	14.5			Curator 1^b	Curator 2^b
Precision		78.3	77.8			86.8	39.5
F-measure		50.3	24.4			42.8	41.2
Category 5^a		Curator 1	Curator 2			57.3	40.3
Recall		32.2	11.3
Precision		75.0	71.4
F-measure		45.1	19.5
PCS
Term-based EQs^c	System alone					Curator 1	Curator 2^d	Curator 3
Recall	65.0					47.0	38.0	50.0
Precision	60.0					57.0	65.0	67.0
F-measure	62.4					51.5	48.0	57.3
Label-based EQs^c	System alone					Curator 1	curator 2^d	Curator 3
Recall	24.0					44.0	51.0	51.0
Precision	23.0					54.0	81.0	74.0
F-measure	23.5					48.5	62.6	60.4
		Phenex + Charaparser				Phenex
Label-based EQs^c		Curator 1	Curator 2^d	Curator 3		Curator 1	Curator 2^d	Curator 3
Recall		51.0	38.0	66.0		37.0	63.0	36.0
Precision		58.0	70.0	84.0		49.0	88.0	60.0
F-measure		54.3	49.3	73.9		42.2	73.4	45.0
PubTator
NLM indexing mention-level	System alone	Curator 1				Curator 1
Recall	80.1	98.6				91.0
Precision	83.4	98.3				93.0
F-measure	81.7	98.0				92.0
TAIR indexing document level	System alone	Curator 2				Curator 2
Recall	76.0	90.0				91.0
Precision	73.9	77.1				75.0
F-measure	74.9	83.0				82.0
TAIR triage	System alone	Curator 2
Recall	68.6	84.6
Precision	80.5	100.0
F-measure	74.1	92.0
PPInterFinder
PPI algorithm alone	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.8	63.8			72.7	79.7
Precision		85.7	85.7			87.0	90.4
F-measure		76.9	73.2			79.2	84.7
PPI algorithm (gene mention/ gene normalization)	System alone	Curator 1	Curator 2
Recall	NR	46.9	46.9
Precision		85.7	85.7
F-measure		60.6	60.6
eFIP
PMID-centric (sentence level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.2	88.2			89.5	77.8
Precision		94.7	79.0			85.0	70.0
F-measure		80.0	83.3			87.2	73.7
Gene-centric (document level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	78.6	85.7			100.0	77.8
Precision		91.7	85.7			83.3	77.8
F-measure		84.6	85.7			90.9	77.8
Document-ranking
nDCG	93–100
T-HOD
PMID-centric (sentence level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	70.0	56.0	22.0	24.0	42.0
Precision	79.5	32.0	26.0	40.0	42.0
F-measure	74.5	40.0	24.0	30.0	42.0
Gene-centric (document level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	54.3	56.0	30.0	26.0	42.0
Precision	72.1	63.0	41.0	52.0	71.0
F-measure	62.0	59.0	35.0	35.0	53.0

System performance measure (%)	System output versus gold standard annotation	System-assisted annotations				Manual annotation
Textpresso
Sentence level
Category 4^a	System alone
Recall	37.9
Precision	77.5					Curator 1^b	Curator 2^b
F-measure	50.9					55.1	26.9
Category 5^a	System alone					41.7	63.3
Recall	39.7					47.5	37.8
Precision	81.5
F-measure	53.4
GO annotation level
Category 4^a		Curator 1	Curator 2
Recall		37.1	14.5			Curator 1^b	Curator 2^b
Precision		78.3	77.8			86.8	39.5
F-measure		50.3	24.4			42.8	41.2
Category 5^a		Curator 1	Curator 2			57.3	40.3
Recall		32.2	11.3
Precision		75.0	71.4
F-measure		45.1	19.5
PCS
Term-based EQs^c	System alone					Curator 1	Curator 2^d	Curator 3
Recall	65.0					47.0	38.0	50.0
Precision	60.0					57.0	65.0	67.0
F-measure	62.4					51.5	48.0	57.3
Label-based EQs^c	System alone					Curator 1	curator 2^d	Curator 3
Recall	24.0					44.0	51.0	51.0
Precision	23.0					54.0	81.0	74.0
F-measure	23.5					48.5	62.6	60.4
		Phenex + Charaparser				Phenex
Label-based EQs^c		Curator 1	Curator 2^d	Curator 3		Curator 1	Curator 2^d	Curator 3
Recall		51.0	38.0	66.0		37.0	63.0	36.0
Precision		58.0	70.0	84.0		49.0	88.0	60.0
F-measure		54.3	49.3	73.9		42.2	73.4	45.0
PubTator
NLM indexing mention-level	System alone	Curator 1				Curator 1
Recall	80.1	98.6				91.0
Precision	83.4	98.3				93.0
F-measure	81.7	98.0				92.0
TAIR indexing document level	System alone	Curator 2				Curator 2
Recall	76.0	90.0				91.0
Precision	73.9	77.1				75.0
F-measure	74.9	83.0				82.0
TAIR triage	System alone	Curator 2
Recall	68.6	84.6
Precision	80.5	100.0
F-measure	74.1	92.0
PPInterFinder
PPI algorithm alone	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.8	63.8			72.7	79.7
Precision		85.7	85.7			87.0	90.4
F-measure		76.9	73.2			79.2	84.7
PPI algorithm (gene mention/ gene normalization)	System alone	Curator 1	Curator 2
Recall	NR	46.9	46.9
Precision		85.7	85.7
F-measure		60.6	60.6
eFIP
PMID-centric (sentence level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	69.2	88.2			89.5	77.8
Precision		94.7	79.0			85.0	70.0
F-measure		80.0	83.3			87.2	73.7
Gene-centric (document level)	System alone	Curator 1	Curator 2			Curator 1	Curator 2
Recall	NR	78.6	85.7			100.0	77.8
Precision		91.7	85.7			83.3	77.8
F-measure		84.6	85.7			90.9	77.8
Document-ranking
nDCG	93–100
T-HOD
PMID-centric (sentence level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	70.0	56.0	22.0	24.0	42.0
Precision	79.5	32.0	26.0	40.0	42.0
F-measure	74.5	40.0	24.0	30.0	42.0
Gene-centric (document level)	System alone	Curator 1	Curator 2	Curator 3	Curator 4
Recall	54.3	56.0	30.0	26.0	42.0
Precision	72.1	63.0	41.0	52.0	71.0
F-measure	62.0	59.0	35.0	35.0	53.0

^a4-Category search use ‘bag of words’ for (1) assay terms, (2) verbs, (3) cellular component terms, and (4) gene product names, whereas 5-Category search also include words for Table and Figures. ^bManual annotations don't necessarily correspond to either the 4- or 5-category search as curators do annotations for sentences that fit both criteria. ^cTerm-label EQs are entity-quality statements created strictly based on the original descriptions, independent of any ontologies, whereas the label-based EQs are the corresponding formal statements (using ontology terms). ^dCurator ignore an unspecified number of CharaParser proposals to save time.

This Feature Is Available To Subscribers Only