Performance of humans versus automated systems (per-tag and overall precision (P), recall (R) and F value (F))
| . | Performance of humans versus automated systems . | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Annotator 3 . | Annotator 4 . | MIST2 . | MCRF2 . | |||||||||
| P . | R . | F . | P . | R . | F . | P . | R . | F . | P . | R . | F . | |
| AGE | 99.51 | 91.93 | 95.57 | 94.59 | 94.17 | 94.38 | 95.50 | 95.07 | 95.28 | 97.21 | 93.72 | 95.43 |
| DATE | 98.17 | 97.78 | 97.97 | 98.56 | 97.62 | 98.09 | 96.73 | 98.57 | 97.65 | 97.86 | 97.78 | 97.82 |
| ID | 93.67 | 85.06 | 89.16 | 88.37 | 87.36 | 87.86 | 88.89 | 88.89 | 88.89 | 95.45 | 93.33 | 94.38 |
| INST | 78.33 | 85.98 | 81.98 | 84.52 | 86.59 | 85.54 | 90.12 | 89.02 | 89.57 | 93.33 | 85.37 | 89.17 |
| LOC | 97.78 | 93.62 | 95.65 | 86.67 | 82.98 | 84.78 | 66.67 | 55.32 | 60.47 | 82.86 | 61.70 | 70.73 |
| NAME | 98.92 | 94.93 | 96.88 | 99.08 | 97.92 | 98.50 | 93.52 | 95.71 | 94.60 | 95.49 | 96.36 | 95.92 |
| OTH | 68.77 | 65.55 | 67.12 | 47.28 | 81.27 | 59.78 | 83.27 | 76.59 | 79.79 | 87.70 | 71.57 | 78.82 |
| PH | 96.92 | 98.44 | 97.67 | 96.92 | 98.44 | 97.67 | 88.41 | 95.31 | 91.73 | 95.31 | 95.31 | 95.31 |
| All | 93.95 | 92.15 | 93.04 | 88.45 | 94.55 | 91.40 | 93.31 | 93.66 | 93.49 | 95.73 | 92.91 | 94.30 |
| . | Performance of humans versus automated systems . | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Annotator 3 . | Annotator 4 . | MIST2 . | MCRF2 . | |||||||||
| P . | R . | F . | P . | R . | F . | P . | R . | F . | P . | R . | F . | |
| AGE | 99.51 | 91.93 | 95.57 | 94.59 | 94.17 | 94.38 | 95.50 | 95.07 | 95.28 | 97.21 | 93.72 | 95.43 |
| DATE | 98.17 | 97.78 | 97.97 | 98.56 | 97.62 | 98.09 | 96.73 | 98.57 | 97.65 | 97.86 | 97.78 | 97.82 |
| ID | 93.67 | 85.06 | 89.16 | 88.37 | 87.36 | 87.86 | 88.89 | 88.89 | 88.89 | 95.45 | 93.33 | 94.38 |
| INST | 78.33 | 85.98 | 81.98 | 84.52 | 86.59 | 85.54 | 90.12 | 89.02 | 89.57 | 93.33 | 85.37 | 89.17 |
| LOC | 97.78 | 93.62 | 95.65 | 86.67 | 82.98 | 84.78 | 66.67 | 55.32 | 60.47 | 82.86 | 61.70 | 70.73 |
| NAME | 98.92 | 94.93 | 96.88 | 99.08 | 97.92 | 98.50 | 93.52 | 95.71 | 94.60 | 95.49 | 96.36 | 95.92 |
| OTH | 68.77 | 65.55 | 67.12 | 47.28 | 81.27 | 59.78 | 83.27 | 76.59 | 79.79 | 87.70 | 71.57 | 78.82 |
| PH | 96.92 | 98.44 | 97.67 | 96.92 | 98.44 | 97.67 | 88.41 | 95.31 | 91.73 | 95.31 | 95.31 | 95.31 |
| All | 93.95 | 92.15 | 93.04 | 88.45 | 94.55 | 91.40 | 93.31 | 93.66 | 93.49 | 95.73 | 92.91 | 94.30 |
| Anno3 vs Anno4 | Anno3 vs MCRF2 | Anno4 vs MCRF2 | Anno3 vs MIST2 | Anno4 vs MIST2 | Anno3 vs gold standard | Anno4 vs gold standard | MCRF2 vs gold standard | MIST2 vs gold standard | |
| p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | |
| AGE | 0.5226 | 0.9628 | 0.6493 | 0.9368 | 0.6516 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| DATE | 0.881 | 0.7514 | 0.7 | 0.4288 | 0.555 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| ID | 0.7526 | 0.1885 | 0.1428 | 0.9377 | 0.8021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| INST | 0.2785 | *0.0107 | 0.3347 | *0.0078 | 0.2949 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| LOC | 0.1579 | *0.0078 | 0.1891 | *0.0001 | *0.021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| NAME | *0.0221 | 0.2246 | *0.0026 | *0.0093 | *0.0002 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| OTH | 0.0506 | *0.0029 | *0.0001 | *0.0006 | *0.0001 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| PH | 1 | 0.1299 | 0.1231 | *0.0414 | *0.0492 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| All | *0.0169 | 0.054 | *0.0002 | 0.5088 | *0.0105 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| Anno3 vs Anno4 | Anno3 vs MCRF2 | Anno4 vs MCRF2 | Anno3 vs MIST2 | Anno4 vs MIST2 | Anno3 vs gold standard | Anno4 vs gold standard | MCRF2 vs gold standard | MIST2 vs gold standard | |
| p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | |
| AGE | 0.5226 | 0.9628 | 0.6493 | 0.9368 | 0.6516 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| DATE | 0.881 | 0.7514 | 0.7 | 0.4288 | 0.555 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| ID | 0.7526 | 0.1885 | 0.1428 | 0.9377 | 0.8021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| INST | 0.2785 | *0.0107 | 0.3347 | *0.0078 | 0.2949 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| LOC | 0.1579 | *0.0078 | 0.1891 | *0.0001 | *0.021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| NAME | *0.0221 | 0.2246 | *0.0026 | *0.0093 | *0.0002 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| OTH | 0.0506 | *0.0029 | *0.0001 | *0.0006 | *0.0001 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| PH | 1 | 0.1299 | 0.1231 | *0.0414 | *0.0492 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| All | *0.0169 | 0.054 | *0.0002 | 0.5088 | *0.0105 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
Statistical significance tests between F values obtained by humans versus systems (*indicates statistical significance (p<0.05), Anno3=annotator 3, Anno4=annotator 4).
INST, institution; LOC, location; OTH, other; PH, phone.
Performance of humans versus automated systems (per-tag and overall precision (P), recall (R) and F value (F))
| . | Performance of humans versus automated systems . | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Annotator 3 . | Annotator 4 . | MIST2 . | MCRF2 . | |||||||||
| P . | R . | F . | P . | R . | F . | P . | R . | F . | P . | R . | F . | |
| AGE | 99.51 | 91.93 | 95.57 | 94.59 | 94.17 | 94.38 | 95.50 | 95.07 | 95.28 | 97.21 | 93.72 | 95.43 |
| DATE | 98.17 | 97.78 | 97.97 | 98.56 | 97.62 | 98.09 | 96.73 | 98.57 | 97.65 | 97.86 | 97.78 | 97.82 |
| ID | 93.67 | 85.06 | 89.16 | 88.37 | 87.36 | 87.86 | 88.89 | 88.89 | 88.89 | 95.45 | 93.33 | 94.38 |
| INST | 78.33 | 85.98 | 81.98 | 84.52 | 86.59 | 85.54 | 90.12 | 89.02 | 89.57 | 93.33 | 85.37 | 89.17 |
| LOC | 97.78 | 93.62 | 95.65 | 86.67 | 82.98 | 84.78 | 66.67 | 55.32 | 60.47 | 82.86 | 61.70 | 70.73 |
| NAME | 98.92 | 94.93 | 96.88 | 99.08 | 97.92 | 98.50 | 93.52 | 95.71 | 94.60 | 95.49 | 96.36 | 95.92 |
| OTH | 68.77 | 65.55 | 67.12 | 47.28 | 81.27 | 59.78 | 83.27 | 76.59 | 79.79 | 87.70 | 71.57 | 78.82 |
| PH | 96.92 | 98.44 | 97.67 | 96.92 | 98.44 | 97.67 | 88.41 | 95.31 | 91.73 | 95.31 | 95.31 | 95.31 |
| All | 93.95 | 92.15 | 93.04 | 88.45 | 94.55 | 91.40 | 93.31 | 93.66 | 93.49 | 95.73 | 92.91 | 94.30 |
| . | Performance of humans versus automated systems . | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Annotator 3 . | Annotator 4 . | MIST2 . | MCRF2 . | |||||||||
| P . | R . | F . | P . | R . | F . | P . | R . | F . | P . | R . | F . | |
| AGE | 99.51 | 91.93 | 95.57 | 94.59 | 94.17 | 94.38 | 95.50 | 95.07 | 95.28 | 97.21 | 93.72 | 95.43 |
| DATE | 98.17 | 97.78 | 97.97 | 98.56 | 97.62 | 98.09 | 96.73 | 98.57 | 97.65 | 97.86 | 97.78 | 97.82 |
| ID | 93.67 | 85.06 | 89.16 | 88.37 | 87.36 | 87.86 | 88.89 | 88.89 | 88.89 | 95.45 | 93.33 | 94.38 |
| INST | 78.33 | 85.98 | 81.98 | 84.52 | 86.59 | 85.54 | 90.12 | 89.02 | 89.57 | 93.33 | 85.37 | 89.17 |
| LOC | 97.78 | 93.62 | 95.65 | 86.67 | 82.98 | 84.78 | 66.67 | 55.32 | 60.47 | 82.86 | 61.70 | 70.73 |
| NAME | 98.92 | 94.93 | 96.88 | 99.08 | 97.92 | 98.50 | 93.52 | 95.71 | 94.60 | 95.49 | 96.36 | 95.92 |
| OTH | 68.77 | 65.55 | 67.12 | 47.28 | 81.27 | 59.78 | 83.27 | 76.59 | 79.79 | 87.70 | 71.57 | 78.82 |
| PH | 96.92 | 98.44 | 97.67 | 96.92 | 98.44 | 97.67 | 88.41 | 95.31 | 91.73 | 95.31 | 95.31 | 95.31 |
| All | 93.95 | 92.15 | 93.04 | 88.45 | 94.55 | 91.40 | 93.31 | 93.66 | 93.49 | 95.73 | 92.91 | 94.30 |
| Anno3 vs Anno4 | Anno3 vs MCRF2 | Anno4 vs MCRF2 | Anno3 vs MIST2 | Anno4 vs MIST2 | Anno3 vs gold standard | Anno4 vs gold standard | MCRF2 vs gold standard | MIST2 vs gold standard | |
| p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | |
| AGE | 0.5226 | 0.9628 | 0.6493 | 0.9368 | 0.6516 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| DATE | 0.881 | 0.7514 | 0.7 | 0.4288 | 0.555 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| ID | 0.7526 | 0.1885 | 0.1428 | 0.9377 | 0.8021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| INST | 0.2785 | *0.0107 | 0.3347 | *0.0078 | 0.2949 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| LOC | 0.1579 | *0.0078 | 0.1891 | *0.0001 | *0.021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| NAME | *0.0221 | 0.2246 | *0.0026 | *0.0093 | *0.0002 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| OTH | 0.0506 | *0.0029 | *0.0001 | *0.0006 | *0.0001 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| PH | 1 | 0.1299 | 0.1231 | *0.0414 | *0.0492 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| All | *0.0169 | 0.054 | *0.0002 | 0.5088 | *0.0105 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| Anno3 vs Anno4 | Anno3 vs MCRF2 | Anno4 vs MCRF2 | Anno3 vs MIST2 | Anno4 vs MIST2 | Anno3 vs gold standard | Anno4 vs gold standard | MCRF2 vs gold standard | MIST2 vs gold standard | |
| p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | p Value | |
| AGE | 0.5226 | 0.9628 | 0.6493 | 0.9368 | 0.6516 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| DATE | 0.881 | 0.7514 | 0.7 | 0.4288 | 0.555 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| ID | 0.7526 | 0.1885 | 0.1428 | 0.9377 | 0.8021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| INST | 0.2785 | *0.0107 | 0.3347 | *0.0078 | 0.2949 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| LOC | 0.1579 | *0.0078 | 0.1891 | *0.0001 | *0.021 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| NAME | *0.0221 | 0.2246 | *0.0026 | *0.0093 | *0.0002 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| OTH | 0.0506 | *0.0029 | *0.0001 | *0.0006 | *0.0001 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| PH | 1 | 0.1299 | 0.1231 | *0.0414 | *0.0492 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
| All | *0.0169 | 0.054 | *0.0002 | 0.5088 | *0.0105 | *0.0001 | *0.0001 | *0.0001 | *0.0001 |
Statistical significance tests between F values obtained by humans versus systems (*indicates statistical significance (p<0.05), Anno3=annotator 3, Anno4=annotator 4).
INST, institution; LOC, location; OTH, other; PH, phone.
This PDF is available to Subscribers Only
View Article Abstract & Purchase OptionsFor full access to this pdf, sign in to an existing account, or purchase an annual subscription.
