ctakes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Finan, Sean" <Sean.Fi...@childrens.harvard.edu>
Subject RE: dictionary lookup config for best F1 measure [was RE: cTakes Annotation Comparison : Span Overlap addendum
Date Sat, 10 Jan 2015 00:15:44 GMT
Hi James,

I've checked in a descriptor for the UmlsOverlapLookupAnnotator in fast/desc/ .  I also checked
in a modification for the CuisOnlyPlaintextUMLSProcessor.xml with the Overlap annotator commented
out as an option:

      <delegateAnalysisEngine key="DictionaryLookupAnnotatorDB">
         <!-- UmlsLookupAnnotator only finds exact span matches -->
         <import location="../../../ctakes-dictionary-lookup-fast/desc/analysis_engine/UmlsLookupAnnotator.xml"/>
         <!-- UmlsOverlapLookupAnnotator finds exact span matches and overlapping span
matches -->
         <!--<import location="../../../ctakes-dictionary-lookup-fast/desc/analysis_engine/UmlsOverlapLookupAnnotator.xml"/>-->
      </delegateAnalysisEngine>

As an example of its difference from the Default, I ran the example colon cancer document
from thyme and it finds the following:
"blood with stool" > C1321898: blood in stool
"polyps, all adenomatous" > C0206677: adenomatous polyps
"lesions in his liver" > C0577053: lesion of liver
"PAST MEDICAL/SURGICAL HISTORY" > C0262926: medical history , C0455458: past medical history
"MEDICAL/SURGICAL HISTORY" > C0262926: medical history
"tonsils and adenoids" > C0580788: tonsil and adenoid structure ; this is also found without
overlap, but overlap finds it a second time
"torn left Achilles tendon" > C0263970: rupture of Achilles tendon
"ankle scar on left" > C0230448: structure of left ankle *
"prostate, no masses palpable" > C0577252: prostate palpable
"cancer of the cecum" > C0153437: malignant neoplasm of cecum  ; this is also found without
overlap, but overlap finds it a second time
"complications of anesthesia" > C0392008: complication of anesthesia  ; this is also found
without overlap, but overlap finds it a second time

* One important item is that the overlap annotator understands discontiguous spans.  There
is, in fact, a ...lookup2.textspan.MultiTextSpan class.  So, for items such as "ankle scar
on left" the annotator is actually annotating only "ankle ... left" but it has to be stored
in the cas as one big happy albeit underspecified span.

I think that I mentioned in the previous email that the Overlap annotator has a couple of
extra parameters.  They are called "totalTokenSkips" and "consecutiveTokenSkips".  The names
are pretty self-explanatory; the algorithm will allow a maximum number of tokens to be skipped,
consecutive or not, as long as the total number of consecutive tokens to be skipped is not
above a certain number.  For instance, total=4 and consecutive=2 (the defaults) will match
"this kinda sorta should maybe hopefully match" with "this should match".  This is pretty
lenient, but seems to work in my tests.  "this kinda-sorta should ..." will not match ...
though maybe '-' should be a special case.  Let me know what you think.

Enjoy,
Sean


-----Original Message-----
From: Masanz, James J. [mailto:Masanz.James@mayo.edu] 
Sent: Friday, January 09, 2015 3:57 PM
To: 'dev@ctakes.apache.org'
Subject: dictionary lookup config for best F1 measure [was RE: cTakes Annotation Comparison

Sean (or others), 

Of the various configuration options described below, which values/choices would you recommend
for best F1 measure for something like the shared clef 2013 task?
https://sites.google.com/site/shareclefehealth/

I'm looking for something that doesn't have to be the best speed-wise, but that is the recommended
for optimizing F1 measure.

Regards,
James 

-----Original Message-----
From: Finan, Sean [mailto:Sean.Finan@childrens.harvard.edu] 
Sent: Friday, December 19, 2014 11:55 AM
To: dev@ctakes.apache.org; kim.ebert@imatsolutions.com
Subject: RE: cTakes Annotation Comparison

Well, I guess that it is time for me to speak up …

I must say that I’m happy that people are showing interest in the fast lookup.  I am also
happy (sort of) that some concerns are being raised – and that there is now community participation
in my little toy.  I  have some concerns about what people are reporting.  This does not coincide
with what I have seen at all.  Yesterday I started (without knowing this thread existed) testing
a bare-minimum pipeline for CUI extraction.  It is just the stripped-down Aggregate with only:
segment, tokens, sentences, POS, and the fast lookup.  The people at Children’s wanted to
know how fast we could get.  1,196 notes in under 90 seconds on my laptop with over 210,000
annotations, which is 175/note.  After reading the thread I decided to run the fast lookup
with several configurations.  I also ran the default for 10.5 hours.  I am comparing the annotations
from each system against the human annotations that we have, and I will let everybody know
what I find – for better or worse.

The fast lookup does not (out-of-box) do the exact same thing as the default.  Some things
can be configured to make it more closely approximate the default dictionary.

1.        Set the minimum annotation span length to 2 (default is 3).  This is in desc/[ae]/UmlsLookupAnnotator.xml
: line #78.  The annotator should then pick up text like “CT” and improve recall, but
it will hurt precision.

2.       Set the Lookup Window to LookupWindowAnnotation.  This is in desc/[ae]/UmlsLookupAnnotator.xml:
lines #65 & #93.   The LookupWindowAnnotator will need to be added to the aggregate pipeline
AggregatePlaintextFastUMLSProcesor.xml  lines #50 & #172.  This will narrow the lookup
window and may increase precision, but (in my experience) reduces recall.

3.       Allow the –rough- identification of Overlapping spans.  The default dictionary
will often identify text like “metastatic colorectal carcinoma” when that text actually
does not exist anywhere in umls.  It basically ignores “colorectal” and gives the whole
span the CUI for “metastatic carcinoma”.  In this case it is arguably a good thing.  In
many others it is arguably not so much.  There is a Class ... lookup2.ae.OverlapJCasTermAnnotator.java
that will do the same thing.  You can create a new desc/[ae]/*Annotator.xml or just change
the <annotatorImplementationName> in desc/[ae]/UmlsLookupAnnotator.xml line #25.  I
will check in a new desc xml (sorry; thought I had) because there are 2 parameters unique
to OverlapJCasTermAnnotator

4.       You can play with the OverlapJCasTermAnnotator parameters “consecutiveSkips”
and “totalTokenSkips”.  These control just how lenient you want the overlap tagging to
be.

5.       Create a new dictionary database.  There is a (bit messy) DictionaryTool in sandbox
that will let you dump whatever you do or do not want from UMLS into a database.  It will
also help you clean up or –select- stored entries as well.  There is a lot of garbage in
the default dictionary database: repeated terms with caps/no caps (“Cancer”,”cancer”),
text with metadata (“cancer [finding]”) and text that just clutters (“PhenX: entry for
cancer”, “1”, “2”).  The fast lookup database should have most of the Snomed and
RxNorm terms (and synonyms) of interest, but you could always make a new database that is
much more inclusive.

The main key to the speed of the fast dictionary lookup is actually … the key.  It is the
way that the database is indexed and the lookup by “rare” word instead of “first”
word.  Everything else can be changed around it and it should still be a faster version.

As for the false positives like “Today”, that will always be a problem until we have disambiguation.
 The lookup is basically a glorified grep.

Sean

From: Chen, Pei [mailto:Pei.Chen@childrens.harvard.edu]
Sent: Friday, December 19, 2014 10:43 AM
To: dev@ctakes.apache.org; kim.ebert@imatsolutions.com
Subject: RE: cTakes Annotation Comparison

Also check out stats that Sean ran before releasing the new component on:
http://svn.apache.org/repos/asf/ctakes/trunk/ctakes-dictionary-lookup-fast/doc/DictionaryLookupStats.docx
From the evaluation and experience, the new lookup algorithm should be a huge improvement
in terms of both speed and accuracy.
This is very different than what Bruce mentioned…  I’m sure Sean will chime here.
(The old dictionary lookup is essentially obsolete now- plagued with bugs/issues as you mentioned.)
--Pei

From: Kim Ebert [mailto:kim.ebert@perfectsearchcorp.com]
Sent: Friday, December 19, 2014 10:25 AM
To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org>
Subject: Re: cTakes Annotation Comparison

Guergana,

I'm curious to the number of records that are in your gold standard sets, or if your gold
standard set was run through a long running cTAKES process. I know at some point we fixed
a bug in the old dictionary lookup that caused the permutations to become corrupted over time.
Typically this isn't seen in the first few records, but over time as patterns are used the
permutations would become corrupted. This caused documents that were fed through cTAKES more
than once to have less codes returned than the first time.

For example, if a permutation of 4,2,3,1 was found, the permutation would be corrupted to
be 1,2,3,4. It would no longer be possible to detect permutations of 4,2,3,1 until cTAKES
was restarted. We got the fix in after the cTAKES 3.2.0 release. https://issues.apache.org/jira/browse/CTAKES-310
Depending upon the corpus size, I could see the permutation engine eventually only have a
single permutation of 1,2,3,4.

Typically though, this isn't very easily detected in the first 100 or so documents.

We discovered this issue when we made cTAKES have consistent output of codes in our system.

[IMAT Solutions]<http://imatsolutions.com>
Kim Ebert
Software Engineer
[Office:]801.669.7342
kim.ebert@imatsolutions.com<mailto:greg.hubert@imatsolutions.com>
On 12/19/2014 07:05 AM, Savova, Guergana wrote:

We are doing a similar kind of evaluation and will report the results.



Before we released the Fast lookup, we did a systematic evaluation across three gold standard
sets. We did not see the trend that Bruce reported below. The P, R and F1 results from the
old dictionary look up and the fast one were similar.



Thank you everyone!

--Guergana



-----Original Message-----

From: David Kincaid [mailto:kincaid.dave@gmail.com]

Sent: Friday, December 19, 2014 9:02 AM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org>

Subject: Re: cTakes Annotation Comparison



Thanks for this, Bruce! Very interesting work. It confirms what I've seen in my small tests
that I've done in a non-systematic way. Did you happen to capture the number of false positives
yet (annotations made by cTAKES that are not in the human adjudicated standard)? I've seen
a lot of dictionary hits that are not actually entity mentions, but I haven't had a chance
to do a systematic analysis (we're working on our annotated gold standard now). One great
example is the antibiotic "Today". Every time the word today appears in any text it is annotated
as a medication mention when it almost never is being used in that sense.



These results by themselves are quite disappointing to me. Both the UMLSProcessor and especially
the FastUMLSProcessor seem to have pretty poor recall. It seems like the trade off for more
speed is a ten-fold (or more) decrease in entity recognition.



Thanks again for sharing your results with us. I think they are very useful to the project.



- Dave



On Thu, Dec 18, 2014 at 5:06 PM, Bruce Tietjen < bruce.tietjen@perfectsearchcorp.com<mailto:bruce.tietjen@perfectsearchcorp.com>>
wrote:



Actually, we are working on a similar tool to compare it to the human

adjudicated standard for the set we tested against.  I didn't mention

it before because the tool isn't complete yet, but initial results for

the set (excluding those marked as "CUI-less") was as follows:



Human adjudicated annotations: 4591 (excluding CUI-less)



Annotations found matching the human adjudicated standard

UMLSProcessor                  2245

FastUMLSProcessor           215













 [image: IMAT Solutions] <http://imatsolutions.com><http://imatsolutions.com>
 Bruce Tietjen

Senior Software Engineer

[image: Mobile:] 801.634.1547

bruce.tietjen@imatsolutions.com<mailto:bruce.tietjen@imatsolutions.com>



On Thu, Dec 18, 2014 at 3:37 PM, Chen, Pei

<Pei.Chen@childrens.harvard.edu<mailto:Pei.Chen@childrens.harvard.edu>



wrote:



Bruce,

Thanks for this-- very useful.

Perhaps Sean Finan comment more-

but it's also probably worth it to compare to an adjudicated human

annotated gold standard.



--Pei



-----Original Message-----

From: Bruce Tietjen [mailto:bruce.tietjen@perfectsearchcorp.com]

Sent: Thursday, December 18, 2014 1:45 PM

To: dev@ctakes.apache.org<mailto:dev@ctakes.apache.org>

Subject: cTakes Annotation Comparison



With the recent release of cTakes 3.2.1, we were very interested in

checking for any differences in annotations between using the

AggregatePlaintextUMLSProcessor pipeline and the

AggregatePlanetextFastUMLSProcessor pipeline within this release of

cTakes

with its associated set of UMLS resources.



We chose to use the SHARE 14-a-b Training data that consists of 199

documents (Discharge  61, ECG 54, Echo 42 and Radiology 42) as the

basis for the comparison.



We decided to share a summary of the results with the development

community.



Documents Processed: 199



Processing Time:

UMLSProcessor           2,439 seconds

FastUMLSProcessor    1,837 seconds



Total Annotations Reported:

UMLSProcessor                  20,365 annotations

FastUMLSProcessor             8,284 annotations





Annotation Comparisons:

Annotations common to both sets:                                  3,940

Annotations reported only by the UMLSProcessor:         16,425

Annotations reported only by the FastUMLSProcessor:    4,344





If anyone is interested, following was our test procedure:



We used the UIMA CPE to process the document set twice, once using

the AggregatePlaintextUMLSProcessor pipeline and once using the

AggregatePlaintextFastUMLSProcessor pipeline. We used the

WriteCAStoFile CAS consumer to write the results to output files.



We used a tool we recently developed to analyze and compare the

annotations generated by the two pipelines. The tool compares the

two outputs for each file and reports any differences in the

annotations (MedicationMention, SignSymptomMention,

ProcedureMention, AnatomicalSiteMention, and

DiseaseDisorderMention) between the two output sets. The tool

reports the number of 'matches' and 'misses' between each annotation set. A 'match'

is

defined as the presence of an identified source text interval with

its associated CUI appearing in both annotation sets. A 'miss' is

defined as the presence of an identified source text interval and

its associated CUI in one annotation set, but no matching identified

source text interval

and

CUI in the other. The tool also reports the total number of

annotations (source text intervals with associated CUIs) reported in

each annotation set. The compare tool is in our GitHub repository at

https://github.com/perfectsearch/cTAKES-compare





Mime
View raw message