| System . | Dataset selection for pre-workshop evaluation . | Information captured . | Biocurators involved in gold standard annotation . | Biocurators involved in annotation in evaluation . |
|---|---|---|---|---|
| Textpresso | 30 full-length articles about Dictyostelium discoideum from 2011 to 2012 not yet annotated in dictyBase. This set contains 61 GO cellular component annotations in 124 sentences as annotated by senior dictyBase biocurator | Paper Identifier, annotation entity, paper section, curatable sentence, component term in sentence, GO term, GO ID and evidence code. | dictyBase senior curator | dictyBase and Plant Ontologya |
| PCS | 50 textual descriptions of phenotypic characters in NeXML format randomly selected from 50 articles about fish or other vertebrates. Gold standard 50 character descriptions annotated by a senior Phenoscape biocurator | Entity term, entity ID, quality term, quality ID, quality negated, quality modifier, entity locator, count and more | Phenoscape senior curator | ZFIN and Phenoscape |
| PubTator | TAIR set: 50 abstracts (24 relevant) sampled from November 2011 for Arabidopsis already curated by TAIR | Gene indexing: gene names and Entrez gene ID | Existing annotated corpus | TAIR and National Library of Medicine (NLM) |
| NLM set: 50 abstracts sampled from Gene Indexing Assistant Test Collection (human) | Document triage information: list of relevant PMIDs | |||
| PPInterFinder | 50 abstracts describing human kinases obtained by using a combination of tool/resources (such as UniProt, PubMeMiner, FABLE, and PIE). | PMID, protein interactant name 1, protein interactant name 2 | NR | BioGrid and MINT |
| eFIP | PMID-centric: 50 abstracts randomly selected based on proteins involved in two pathways of interest to Reactome autophagy and HIV infection | PMID, phosphorylated protein, phosphorylated site, interactant name, effect, evidence sentence | NR | Merck Serono, Reactome, and SGDb |
| gene-centric: 10 first-ranked abstracts for 4 proteins involved in the adaptive immune system (Reactome: REACT_75774) | ||||
| T-HOD | PMID-centric: 50 abstracts from 2011 journals about obesity, diabetes or hypertension | PMID, EntrezGene ID, gene name, disease, gene–disease relation, evidence sentence | Protein Ontology senior curator | Pfizer, Reactome, GAD, and MGI |
| gene-centric: review relevancy of documents for four genes |
| System . | Dataset selection for pre-workshop evaluation . | Information captured . | Biocurators involved in gold standard annotation . | Biocurators involved in annotation in evaluation . |
|---|---|---|---|---|
| Textpresso | 30 full-length articles about Dictyostelium discoideum from 2011 to 2012 not yet annotated in dictyBase. This set contains 61 GO cellular component annotations in 124 sentences as annotated by senior dictyBase biocurator | Paper Identifier, annotation entity, paper section, curatable sentence, component term in sentence, GO term, GO ID and evidence code. | dictyBase senior curator | dictyBase and Plant Ontologya |
| PCS | 50 textual descriptions of phenotypic characters in NeXML format randomly selected from 50 articles about fish or other vertebrates. Gold standard 50 character descriptions annotated by a senior Phenoscape biocurator | Entity term, entity ID, quality term, quality ID, quality negated, quality modifier, entity locator, count and more | Phenoscape senior curator | ZFIN and Phenoscape |
| PubTator | TAIR set: 50 abstracts (24 relevant) sampled from November 2011 for Arabidopsis already curated by TAIR | Gene indexing: gene names and Entrez gene ID | Existing annotated corpus | TAIR and National Library of Medicine (NLM) |
| NLM set: 50 abstracts sampled from Gene Indexing Assistant Test Collection (human) | Document triage information: list of relevant PMIDs | |||
| PPInterFinder | 50 abstracts describing human kinases obtained by using a combination of tool/resources (such as UniProt, PubMeMiner, FABLE, and PIE). | PMID, protein interactant name 1, protein interactant name 2 | NR | BioGrid and MINT |
| eFIP | PMID-centric: 50 abstracts randomly selected based on proteins involved in two pathways of interest to Reactome autophagy and HIV infection | PMID, phosphorylated protein, phosphorylated site, interactant name, effect, evidence sentence | NR | Merck Serono, Reactome, and SGDb |
| gene-centric: 10 first-ranked abstracts for 4 proteins involved in the adaptive immune system (Reactome: REACT_75774) | ||||
| T-HOD | PMID-centric: 50 abstracts from 2011 journals about obesity, diabetes or hypertension | PMID, EntrezGene ID, gene name, disease, gene–disease relation, evidence sentence | Protein Ontology senior curator | Pfizer, Reactome, GAD, and MGI |
| gene-centric: review relevancy of documents for four genes |
NR:non-recorded. aCurator novice to GO annotation. bSGD curator participated in first evaluation which is not reported in performance results here.
| System . | Dataset selection for pre-workshop evaluation . | Information captured . | Biocurators involved in gold standard annotation . | Biocurators involved in annotation in evaluation . |
|---|---|---|---|---|
| Textpresso | 30 full-length articles about Dictyostelium discoideum from 2011 to 2012 not yet annotated in dictyBase. This set contains 61 GO cellular component annotations in 124 sentences as annotated by senior dictyBase biocurator | Paper Identifier, annotation entity, paper section, curatable sentence, component term in sentence, GO term, GO ID and evidence code. | dictyBase senior curator | dictyBase and Plant Ontologya |
| PCS | 50 textual descriptions of phenotypic characters in NeXML format randomly selected from 50 articles about fish or other vertebrates. Gold standard 50 character descriptions annotated by a senior Phenoscape biocurator | Entity term, entity ID, quality term, quality ID, quality negated, quality modifier, entity locator, count and more | Phenoscape senior curator | ZFIN and Phenoscape |
| PubTator | TAIR set: 50 abstracts (24 relevant) sampled from November 2011 for Arabidopsis already curated by TAIR | Gene indexing: gene names and Entrez gene ID | Existing annotated corpus | TAIR and National Library of Medicine (NLM) |
| NLM set: 50 abstracts sampled from Gene Indexing Assistant Test Collection (human) | Document triage information: list of relevant PMIDs | |||
| PPInterFinder | 50 abstracts describing human kinases obtained by using a combination of tool/resources (such as UniProt, PubMeMiner, FABLE, and PIE). | PMID, protein interactant name 1, protein interactant name 2 | NR | BioGrid and MINT |
| eFIP | PMID-centric: 50 abstracts randomly selected based on proteins involved in two pathways of interest to Reactome autophagy and HIV infection | PMID, phosphorylated protein, phosphorylated site, interactant name, effect, evidence sentence | NR | Merck Serono, Reactome, and SGDb |
| gene-centric: 10 first-ranked abstracts for 4 proteins involved in the adaptive immune system (Reactome: REACT_75774) | ||||
| T-HOD | PMID-centric: 50 abstracts from 2011 journals about obesity, diabetes or hypertension | PMID, EntrezGene ID, gene name, disease, gene–disease relation, evidence sentence | Protein Ontology senior curator | Pfizer, Reactome, GAD, and MGI |
| gene-centric: review relevancy of documents for four genes |
| System . | Dataset selection for pre-workshop evaluation . | Information captured . | Biocurators involved in gold standard annotation . | Biocurators involved in annotation in evaluation . |
|---|---|---|---|---|
| Textpresso | 30 full-length articles about Dictyostelium discoideum from 2011 to 2012 not yet annotated in dictyBase. This set contains 61 GO cellular component annotations in 124 sentences as annotated by senior dictyBase biocurator | Paper Identifier, annotation entity, paper section, curatable sentence, component term in sentence, GO term, GO ID and evidence code. | dictyBase senior curator | dictyBase and Plant Ontologya |
| PCS | 50 textual descriptions of phenotypic characters in NeXML format randomly selected from 50 articles about fish or other vertebrates. Gold standard 50 character descriptions annotated by a senior Phenoscape biocurator | Entity term, entity ID, quality term, quality ID, quality negated, quality modifier, entity locator, count and more | Phenoscape senior curator | ZFIN and Phenoscape |
| PubTator | TAIR set: 50 abstracts (24 relevant) sampled from November 2011 for Arabidopsis already curated by TAIR | Gene indexing: gene names and Entrez gene ID | Existing annotated corpus | TAIR and National Library of Medicine (NLM) |
| NLM set: 50 abstracts sampled from Gene Indexing Assistant Test Collection (human) | Document triage information: list of relevant PMIDs | |||
| PPInterFinder | 50 abstracts describing human kinases obtained by using a combination of tool/resources (such as UniProt, PubMeMiner, FABLE, and PIE). | PMID, protein interactant name 1, protein interactant name 2 | NR | BioGrid and MINT |
| eFIP | PMID-centric: 50 abstracts randomly selected based on proteins involved in two pathways of interest to Reactome autophagy and HIV infection | PMID, phosphorylated protein, phosphorylated site, interactant name, effect, evidence sentence | NR | Merck Serono, Reactome, and SGDb |
| gene-centric: 10 first-ranked abstracts for 4 proteins involved in the adaptive immune system (Reactome: REACT_75774) | ||||
| T-HOD | PMID-centric: 50 abstracts from 2011 journals about obesity, diabetes or hypertension | PMID, EntrezGene ID, gene name, disease, gene–disease relation, evidence sentence | Protein Ontology senior curator | Pfizer, Reactome, GAD, and MGI |
| gene-centric: review relevancy of documents for four genes |
NR:non-recorded. aCurator novice to GO annotation. bSGD curator participated in first evaluation which is not reported in performance results here.
This PDF is available to Subscribers Only
View Article Abstract & Purchase OptionsFor full access to this pdf, sign in to an existing account, or purchase an annual subscription.
