Skip navigation
Skip navigation

Evaluating the state of the art in disorder recognition and normalization of the clinical narrative

Pradhan, Sameer; Elhadad, Noemie; South, Brett R.; Martinez, David; Christensen, Lee; Vogel, Amy; Suominen, Hanna; Chapman, Wendy W; Savova, Guergana

Description

Objective The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of...[Show more]

dc.contributor.authorPradhan, Sameer
dc.contributor.authorElhadad, Noemie
dc.contributor.authorSouth, Brett R.
dc.contributor.authorMartinez, David
dc.contributor.authorChristensen, Lee
dc.contributor.authorVogel, Amy
dc.contributor.authorSuominen, Hanna
dc.contributor.authorChapman, Wendy W
dc.contributor.authorSavova, Guergana
dc.date.accessioned2015-12-10T23:32:38Z
dc.identifier.issn1067-5027
dc.identifier.urihttp://hdl.handle.net/1885/68914
dc.description.abstractObjective The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text-199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.
dc.publisherHanley and Belfus, Inc.
dc.sourceJournal of the American Medical Informatics Association : JAMIA
dc.titleEvaluating the state of the art in disorder recognition and normalization of the clinical narrative
dc.typeJournal article
local.description.notesImported from ARIES
local.identifier.citationvolume22
dc.date.issued2015
local.identifier.absfor170201 - Computer Perception, Memory and Attention
local.identifier.ariespublicationa383154xPUB1866
local.type.statusPublished Version
local.contributor.affiliationPradhan, Sameer, Harvard University
local.contributor.affiliationElhadad, Noemie, Columbia University
local.contributor.affiliationSouth, Brett R., University of Utah
local.contributor.affiliationMartinez, David, University of Melbourne
local.contributor.affiliationChristensen, Lee, University of Utah
local.contributor.affiliationVogel, Amy, Columbia University
local.contributor.affiliationSuominen, Hanna, College of Engineering and Computer Science, ANU
local.contributor.affiliationChapman, Wendy W, University of Utah
local.contributor.affiliationSavova, Guergana, Harvard University
local.description.embargo2037-12-31
local.bibliographicCitation.startpage143
local.bibliographicCitation.lastpage154
local.identifier.doi10.1136/amiajnl-2013-002544
dc.date.updated2015-12-10T11:20:59Z
local.identifier.scopusID2-s2.0-84906071994
CollectionsANU Research Publications

Download

File Description SizeFormat Image
01_Pradhan_Evaluating_the_state_of_the_2015.pdf370.48 kBAdobe PDF    Request a copy


Items in Open Research are protected by copyright, with all rights reserved, unless otherwise indicated.

Updated:  19 May 2020/ Responsible Officer:  University Librarian/ Page Contact:  Library Systems & Web Coordinator