Ution Gold typical annotation Pre-workshop Evaluation Workshop
Ution Gold standard annotation Pre-workshop Evaluation Workshop evaluation IndustryAstraZeneca Merck Serono 
Pfizer NLM AgBase dictyBase FlyBase MaizeDB MGI SGD TAIR WormBase XenBase ZFIN Plant ontology Protein ontology Reactome GAD Phenoscape BioGrid MINT Other folks (approx.) Literature Model Organism (MOD)Gene Ontology Consortium (GOC) Ontology Pathway Phenotype Protein rotein interactionNumbers in parentheses will be the quantity of biocurators from every institution. Biocurators aided in dataset annotations and technique evaluationsDatasetsThe selection of appropriate information collections for the evaluation was inspired by genuine curation tasks at the same time as keeping in thoughts the biocuration workflows. Each technique had its personal dataset that was chosen by its coordinators and the domain authorities that had been inved inside the annotation from the gold normal. In most instances, the dataset consisted of a collection of PubMed abstracts randomly chosen from a pool of attainable relevant articles. A summary of your dataset selection and details captured is presented in TableNote that the format of an annotated corpus varied based around the system’s output. This table also shows groups inved within the annotation of such corpora, and those who ultimately evaluated the systems.Functionality and usability of systems were calculated primarily based on the following metrics: As `performance measures’ we included comparison of time on process for system-assisted versus manual curation; and a precisionrecallF-measure of the automatic system versus the gold standard annotations (dataset independently manually curated by domain specialist) andor manual versus system-assisted annotations once again rated by the gold typical. We define these measures as follows: Precision TP TP recision Recall Recall FTP FP TP FN Precision RecallEvaluationWe planned two evaluations, a pre-workshop formal evaluation in the systems primarily based around the chosen corpus that included PubMed ID:http://www.ncbi.nlm.nih.gov/pubmed/21911092?dopt=Abstract both systems’ functionality and subjective measures (explained later) and an informal evaluation consisting in the systems’ testing in the workshop throughout the demonstration (demo) session. The latter included only the subjective measure representing largely the user’s initially impression of a method.exactly where TP, FP and FN are true positives, false positives and false negatives, respectively. For the `subjective measure’ we prepared a survey meant to record the subjective expertise of your user with the technique. The survey consisted of 5 main categories, namely, general reaction, system’s capacity to help full tasks, design of application, mastering to work with the application and usability, additionally to `recommendation of your system’ that was evaluated separately; these categories were primarily based on those developed for the Questionnaire for User Interface Satisfaction (QUIS) created by Chin et al. and shown to be a reputable guide to understanding user reactionsEach category contained concerns to be rated.Web page ofPage ofInformation captured Biocurators inved in gold typical annotation dictyBase senior curator Biocurators inved in annotation in evaluation dictyBase and Plant Ontologya Paper Identifier, annotation entity, paper section, curatable sentence, component term in sentence, GO term, GO ID and proof code. Entity term, entity ID, top NSC348884 chemical information quality term, good quality ID, quality negated, high-quality modifier, entity locator, count and much more Gene indexing: gene names and Entrez gene ID Document triage data: list of relevant PMIDs PMID, protein i.
