ctakes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Finan, Sean" <Sean.Fi...@childrens.harvard.edu>
Subject RE: cTakes Annotation Comparison
Date Fri, 19 Dec 2014 18:40:19 GMT
I’m bringing it up in case the Human Annotations were done using a different version.

From: Kim Ebert [mailto:kim.ebert@perfectsearchcorp.com]
Sent: Friday, December 19, 2014 1:40 PM
To: dev@ctakes.apache.org
Subject: Re: cTakes Annotation Comparison

Sean,

I don't think that would be an issue since both the rare word lookup and the first word lookup
are using UMLS 2011AB. Or is the rare word lookup using a different dictionary?

I would expect roughly similar results between the two when it comes to differences between
UMLS versions.

[IMAT Solutions]<http://imatsolutions.com>
Kim Ebert
Software Engineer
[Office:]801.669.7342
kim.ebert@imatsolutions.com<mailto:greg.hubert@imatsolutions.com>
On 12/19/2014 11:31 AM, Finan, Sean wrote:

One quick mention:



The cTakes dictionaries are built with UMLS 2011AB.  If the Human annotations were not done
using the same UMLS version then there WILL be differences in CUI and Semantic group.  I don't
have time to go into it with details, examples, etc. just be aware that every 6 months cuis
are added, removed, deprecated, and moved from one TUI to another.



Sean



-----Original Message-----

From: Savova, Guergana [mailto:Guergana.Savova@childrens.harvard.edu]

Sent: Friday, December 19, 2014 1:28 PM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org>

Subject: RE: cTakes Annotation Comparison



Several thoughts:

1. The ShARE corpus annotates only mentions of type Diseases/Disorders and only Anatomical
Sites associated with a Disease/Disorder. This is by design. cTAKES annotates all mentions
of types Diseases/Disorders, Signs/Symptoms, Procedures, Medications and Anatomical Sites.
Therefore you will get MANY more annotations with cTAKES. Eventually the ShARe corpus will
be expanded to the other types.



2. Keeping (1) in mind, you can approximately estimate the precision/recall/f1 of cTAKES on
the ShARe corpus if you output only mentions of type Disease/Disorder.



3. Could you send us the list of files you use from ShARe to test? We have the corpus and
would like to run against as well.



Hope this makes sense...

--Guergana



-----Original Message-----

From: Bruce Tietjen [mailto:bruce.tietjen@perfectsearchcorp.com]

Sent: Friday, December 19, 2014 1:16 PM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org>

Subject: Re: cTakes Annotation Comparison



Our analysis against the human adjudicated gold standard from this SHARE corpus is using a
simple check to see if the cTakes output included the annotation specified by the gold standard.
The initial results I reported were for exact matches of CUI and text span.  Only exact matches
were counted.



It looks like if we also count as matches cTakes annotations with a matching CUI and a text
span that overlaps the gold standard text span then the matches increase to 224 matching annotations
for the FastUMLS pipeline and 2319 for the the old pipeline.



The question was also asked about annotations in the cTakes output that were not in the human
adjudicated gold standard. The answer is yes, there were a lot of additional annotations made
by cTakes that don't appear to be in the gold standard. We haven't analyzed that yet, but
it looks like the gold standard we are using may only have Disease_Disorder annotations.







 [image: IMAT Solutions] <http://imatsolutions.com><http://imatsolutions.com>
 Bruce Tietjen Senior Software Engineer

[image: Mobile:] 801.634.1547

bruce.tietjen@imatsolutions.com<mailto:bruce.tietjen@imatsolutions.com>



On Fri, Dec 19, 2014 at 9:54 AM, Miller, Timothy < Timothy.Miller@childrens.harvard.edu<mailto:Timothy.Miller@childrens.harvard.edu>>
wrote:



Thanks Kim,

This sounds interesting though I don't totally understand it. Are you

saying that extraction performance for a given note depends on which

order the note was in the processing queue? If so that's pretty bad!

If you (or anyone else who understands this issue) has a concrete

example I think that might help me understand what the problem is/was.



Even though, as Pei mentioned, we are going to try moving the

community to the faster dictionary, I would like to understand better

just to help myself avoid issues of this type going forward (and

verify the new dictionary doesn't use similar logic).



Also, when we finish annotating the sample notes, might we use that as

a point of comparison for the two dictionaries? That would get around

the issue that not everyone has access to the datasets we used for

validation and others are likely not able to share theirs either. And

maybe we can replicate the notes if we want to simulate the scenario

Kim is talking about with thousands or more notes.



Tim





On 12/19/2014 10:24 AM, Kim Ebert wrote:

Guergana,



I'm curious to the number of records that are in your gold standard

sets, or if your gold standard set was run through a long running cTAKES process.

I know at some point we fixed a bug in the old dictionary lookup that

caused the permutations to become corrupted over time. Typically this

isn't seen in the first few records, but over time as patterns are

used the permutations would become corrupted. This caused documents

that were fed through cTAKES more than once to have less codes

returned than the first time.



For example, if a permutation of 4,2,3,1 was found, the permutation

would be corrupted to be 1,2,3,4. It would no longer be possible to

detect permutations of 4,2,3,1 until cTAKES was restarted. We got the

fix in after the cTAKES 3.2.0 release.

https://issues.apache.org/jira/browse/CTAKES-310

Depending upon the corpus size, I could see the permutation engine

eventually only have a single permutation of 1,2,3,4.



Typically though, this isn't very easily detected in the first 100 or

so documents.



We discovered this issue when we made cTAKES have consistent output of

codes in our system.



[IMAT Solutions]<http://imatsolutions.com><http://imatsolutions.com>

Kim Ebert

Software Engineer

[Office:] 801.669.7342

kim.ebert@imatsolutions.com<mailto:kim.ebert@imatsolutions.com><mailto:greg.hubert@imatsolutions.com><mailto:greg.hubert@imatsolutions.com>

On 12/19/2014 07:05 AM, Savova, Guergana wrote:



We are doing a similar kind of evaluation and will report the results.



Before we released the Fast lookup, we did a systematic evaluation

across three gold standard sets. We did not see the trend that Bruce

reported below. The P, R and F1 results from the old dictionary look

up and the fast one were similar.



Thank you everyone!

--Guergana



-----Original Message-----

From: David Kincaid [mailto:kincaid.dave@gmail.com]

Sent: Friday, December 19, 2014 9:02 AM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org><mailto:dev@ctakes.apache.org><mailto:dev@ctakes.apache.org>

Subject: Re: cTakes Annotation Comparison



Thanks for this, Bruce! Very interesting work. It confirms what I've

seen in my small tests that I've done in a non-systematic way. Did you

happen to capture the number of false positives yet (annotations made

by cTAKES that are not in the human adjudicated standard)? I've seen a

lot of dictionary hits that are not actually entity mentions, but I

haven't had a chance to do a systematic analysis (we're working on our

annotated gold standard now). One great example is the antibiotic

"Today". Every time the word today appears in any text it is annotated

as a medication mention when it almost never is being used in that sense.



These results by themselves are quite disappointing to me. Both the

UMLSProcessor and especially the FastUMLSProcessor seem to have pretty

poor recall. It seems like the trade off for more speed is a ten-fold

(or more) decrease in entity recognition.



Thanks again for sharing your results with us. I think they are very

useful to the project.



- Dave



On Thu, Dec 18, 2014 at 5:06 PM, Bruce Tietjen <

bruce.tietjen@perfectsearchcorp.com<mailto:bruce.tietjen@perfectsearchcorp.com><mailto:<mailto:bruce.tietjen@perfectsearchcorp.com>

bruce.tietjen@perfectsearchcorp.com><mailto:bruce.tietjen@perfectsearchcorp.com>>
wrote:





Actually, we are working on a similar tool to compare it to the human

adjudicated standard for the set we tested against.  I didn't mention

it before because the tool isn't complete yet, but initial results for

the set (excluding those marked as "CUI-less") was as follows:



Human adjudicated annotations: 4591 (excluding CUI-less)



Annotations found matching the human adjudicated standard

UMLSProcessor                  2245

FastUMLSProcessor           215













 [image: IMAT Solutions] <http://imatsolutions.com><http://imatsolutions.com><
<http://imatsolutions.com>

http://imatsolutions.com><http://imatsolutions.com>  Bruce Tietjen Senior Software
Engineer

[image: Mobile:] 801.634.1547

bruce.tietjen@imatsolutions.com<mailto:bruce.tietjen@imatsolutions.com><mailto:bruce.tietjen@imatsolutions.com





On Thu, Dec 18, 2014 at 3:37 PM, Chen, Pei

<Pei.Chen@childrens.harvard.edu<mailto:Pei.Chen@childrens.harvard.edu><mailto:Pei.Chen@childrens.harvard.edu><mailto:Pei.Chen@childrens.harvard.edu>





wrote:





Bruce,

Thanks for this-- very useful.

Perhaps Sean Finan comment more-

but it's also probably worth it to compare to an adjudicated human

annotated gold standard.



--Pei



-----Original Message-----

From: Bruce Tietjen [mailto:bruce.tietjen@perfectsearchcorp.com]

Sent: Thursday, December 18, 2014 1:45 PM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org><mailto:dev@ctakes.apache.org><mailto:dev@ctakes.apache.org>

Subject: cTakes Annotation Comparison



With the recent release of cTakes 3.2.1, we were very interested in

checking for any differences in annotations between using the

AggregatePlaintextUMLSProcessor pipeline and the

AggregatePlanetextFastUMLSProcessor pipeline within this release of





cTakes





with its associated set of UMLS resources.



We chose to use the SHARE 14-a-b Training data that consists of 199

documents (Discharge  61, ECG 54, Echo 42 and Radiology 42) as the

basis for the comparison.



We decided to share a summary of the results with the development

community.



Documents Processed: 199



Processing Time:

UMLSProcessor           2,439 seconds

FastUMLSProcessor    1,837 seconds



Total Annotations Reported:

UMLSProcessor                  20,365 annotations

FastUMLSProcessor             8,284 annotations





Annotation Comparisons:

Annotations common to both sets:                                  3,940

Annotations reported only by the UMLSProcessor:         16,425

Annotations reported only by the FastUMLSProcessor:    4,344





If anyone is interested, following was our test procedure:



We used the UIMA CPE to process the document set twice, once using the

AggregatePlaintextUMLSProcessor pipeline and once using the

AggregatePlaintextFastUMLSProcessor pipeline. We used the

WriteCAStoFile CAS consumer to write the results to output files.



We used a tool we recently developed to analyze and compare the

annotations generated by the two pipelines. The tool compares the two

outputs for each file and reports any differences in the annotations

(MedicationMention, SignSymptomMention, ProcedureMention,

AnatomicalSiteMention, and

DiseaseDisorderMention) between the two output sets. The tool reports

the number of 'matches' and 'misses' between each annotation set.

A 'match'





is





defined as the presence of an identified source text interval with its

associated CUI appearing in both annotation sets. A 'miss' is defined

as the presence of an identified source text interval and its

associated CUI in one annotation set, but no matching identified

source text interval





and





CUI in the other. The tool also reports the total number of

annotations (source text intervals with associated CUIs) reported in

each annotation set. The compare tool is in our GitHub repository at

https://github.com/perfectsearch/cTAKES-compare











Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message