ctakes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Finan, Sean" <Sean.Fi...@childrens.harvard.edu>
Subject RE: v_snomed_fword_lookup view
Date Wed, 13 Aug 2014 17:41:05 GMT
Hi Clayton,

I'm glad that you got it working.  Though I stated that I would, I haven't yet checked the
fidelity of trunk.  Urgent data request one day, "must have" writing the next ... and I still
live with the delusion that I left academia to have free time ...

I have never used ytex or weka, so I'm unfamiliar with all things .arff .  Could it be that
the ytex .arff exporter needs to change consumed cTakes annotation classes (>3.1)?

I have a custom CasConsumer that saves text spans and Cuis to file in a simple list, and that
is what I used for the performance analysis of the lookup module.  For our other projects
here in Beantown we have other various outputs that fit the job at hand: text flat files,
xml files, sql database tables, knot-encoded lace doilies, etc.

I'm sure that none of the above helps you, but I felt obliged to provide some kind of answer
to your question.

Sean

> -----Original Message-----
> From: clayclay911@gmail.com [mailto:clayclay911@gmail.com] On Behalf Of
> Clayton Turner
> Sent: Wednesday, August 13, 2014 12:25 PM
> To: dev@ctakes.apache.org
> Subject: Re: v_snomed_fword_lookup view
> 
> Okay, I believe I have ctakes dictionary fast working now. Something I'm curious
> about, though, is how you extract the data in order to conduct analysis.
> 
> I've, in the past, been using the SparseDataExporterImpl from ytex in order to
> create a .arff file for use in weka, but the ctakes pipeline I'm using doesn't seem
> to be compatible with this ytex exporting as I'm not getting any cuis in my arff
> file.
> 
> I'm using the aggregate plain text umls processor analysis engine from ctakes
> and then using the dbconsumer analysis engine from ytex (for storing into the
> database with regard to analysis batch).
> 
> Any tips for exporting or some simple issue I'm missing?
> 
> Thanks,
> Clayton
> 
> 
> On Mon, Aug 11, 2014 at 2:09 PM, Harpreet Khanduja <hsk5004@rit.edu>
> wrote:
> 
> > Yes, absolutely and
> > no problem at all.
> >
> > Regards,
> > Harpreet
> >
> >
> > On Mon, Aug 11, 2014 at 1:16 PM, Finan, Sean <
> > Sean.Finan@childrens.harvard.edu> wrote:
> >
> > > Thanks Harpreet,
> > > That is definitely necessary to build!
> > >
> > > Those lines should already be in the pom, but commented out.  I
> > > think
> > that
> > > some version/branching issues may have arisen at some point wrt this
> > module
> > > ...
> > >
> > > If somebody beats me to it then cheers, otherwise I will try to
> > > check out tonight and get all the bits in place.
> > >
> > > Sean
> > >
> > > > -----Original Message-----
> > > > From: Harpreet Khanduja [mailto:hsk5004@rit.edu]
> > > > Sent: Monday, August 11, 2014 1:12 PM
> > > > To: dev@ctakes.apache.org
> > > > Subject: Re: v_snomed_fword_lookup view
> > > >
> > > > Hello Clayton,
> > > >   I do not know about ytex, but I did switch from
> > > > dictionary-lookup to
> > > dictionary-
> > > > lookup-fast.
> > > >   I update my ctakes-dictionary-lookup-fast project using maven.
> > > >   I think I used Team- Update and switched to the latest revision
> > > available and
> > > > then
> > > >   I downloaded new 3.2 resources from the for umls. and then I
> > > > added
> > > these
> > > > resources to my
> > > >   ctakes-dictionary-lookup-fast resources folder and also the
> > > > classpath
> > > in ctakes-
> > > > clinical-pipeline.
> > > >
> > > >  Then I changed the pom.xml file which belongs to the whole ctakes
> > > project and
> > > > added <dependency> <groupId>org.apache.ctakes</groupId>
> > > > <artifactId>ctakes-dictionary-lookup-res</artifactId>
> > > > <version>${ctakes.version}</version>
> > > > </dependency>
> > > > <dependency>
> > > > <groupId>org.apache.ctakes</groupId>
> > > > <artifactId>ctakes-dictionary-lookup-fast</artifactId>
> > > > <version>${ctakes.version}</version>
> > > > </dependency>
> > > >
> > > >
> > > >  these two dependencies to the file.
> > > >
> > > >
> > > > After this, I also added the dependency
> > > >     <dependency>
> > > > <groupId>org.apache.ctakes</groupId>
> > > > <artifactId>ctakes-dictionary-lookup-fast</artifactId>
> > > > </dependency>
> > > >
> > > > to the pom.xml of ctakes-clinical-pipeline.
> > > >
> > > > And then add the resources folder in ctakes-clinical-pipeline
> > > > using
> > > build path
> > > > configuration under "add class" option.
> > > >
> > > > After this it should work.
> > > >
> > > >
> > > > Regards,
> > > > Harpreet
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Aug 11, 2014 at 12:44 PM, Clayton Turner
> > > > <caturner3@g.cofc.edu
> > >
> > > > wrote:
> > > >
> > > > > I still get the same error with the ctakes3.2 branch. Any
> > suggestions?
> > > > >
> > > > >
> > > > > On Mon, Aug 11, 2014 at 12:06 PM, Clayton Turner
> > > > > <caturner3@g.cofc.edu>
> > > > > wrote:
> > > > >
> > > > > > I'm going to do a clean install through the repo rather than
> > > > > > the binaries and see if that fixes my issue because I think
I
> > > > > > just read a past post saying the lookup2 folders exist there.
> > > > > >
> > > > > >
> > > > > > On Mon, Aug 11, 2014 at 11:52 AM, Clayton Turner
> > > > > > <caturner3@g.cofc.edu>
> > > > > > wrote:
> > > > > >
> > > > > >> When navigating to
> > > > > >> ctakes-dictionary-lookup-fast\desc\analysis_engine
> > > > > >> there are 2 files, assumedly analysis engines.
> > > > > >>
> > > > > >> SnomedLookupAnnotator.xml and SnomedOvLookupAnnotator.xml
> > > > > >>
> > > > > >> If I pick either, I put in my UMLS information but receive
an
> > error
> > > > > >> when trying to run the CPE:
> > > > > >>
> > > > > >> Initialization of CAS Processor with name
> > "SnomedOvLookupAnnotator"
> > > > > >> failed.
> > > > > >> CausedBy:
> org.apache.uima.resource.ResourceConfigurationException:
> > > > > >> Initialization of CAS processor with name
> > "SnomedOvLookupAnnotator"
> > > > > >> failed.
> > > > > >> CausedBy:
> > org.apache.uima.resource.ResourceInitializationException:
> > > > > Error
> > > > > >> initializing "org.apache.uima.resource.impl.DataResource_impl"
> > from
> > > > > >> descriptor file:..............SnomedLookupAnnotator.xml
> > > > > >> CausedBy:
> > org.apache.uima.resource.ResourceInitializationException:
> > > > > Could
> > > > > >> not
> > > > > >> access the resource data at
> > > > > >>
> > > > > >>
> > > > >
> > file:org\apache\ctakes\dictionary\lookup2\Snomed2011ab_ctakesTui\cTake
> > > > > sSnomed.xml
> > > > > >>
> > > > > >> Now, I don't even have a "lookup2" folder and, subsequently
> > > > > >> the
> > Tui
> > > > > >> folder and cTakesSnomed.xml file. This seems to be the
> > > > > >> problem,
> > but
> > > > > >> I'm
> > > > > not
> > > > > >> sure where these files are supposed to be grabbed from.
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Aug 11, 2014 at 11:47 AM, Clayton Turner
> > > > > >> <caturner3@g.cofc.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi again:
> > > > > >>>
> > > > > >>> How exactly do you switch to using the cTakes
> > > dictionary-lookup-fast.
> > > > > Do
> > > > > >>> I need to go in and alter xml files or is it as simple
as
> > > > > >>> adding
> > a
> > > > > certain
> > > > > >>> item to the list of analysis engines?
> > > > > >>>
> > > > > >>>
> > > > > >>> On Fri, Aug 8, 2014 at 3:48 PM, Finan, Sean <
> > > > > >>> Sean.Finan@childrens.harvard.edu> wrote:
> > > > > >>>
> > > > > >>>> Hi Clayton,
> > > > > >>>>
> > > > > >>>> I don't know how the ytex dictionary lookup works,
so I'm
> > > > > >>>> afraid that
> > > > > I
> > > > > >>>> can't help you with an answer.  Maybe Vijay is the
best
> > > > > >>>> person
> > to
> > > > > >>>> do
> > > > > this.
> > > > > >>>>  If you aren't tied to ytex you could try the new
cTakes
> > > > > >>>> dictionary-lookup-fast.  I tested "Patient came
in with a
> > > > > >>>> malar
> > > rash"
> > > > > and
> > > > > >>>> it found "malar" and "malar rash".
> > > > > >>>>
> > > > > >>>> Vijay,
> > > > > >>>>
> > > > > >>>> At some point the lookup-fast module will be the
default
> > > > > >>>> for the
> > > > > cTakes
> > > > > >>>> clinical pipeline.  In order to synchronize the
ytex lookup
> > > > > >>>> with
> > > > > cTakes,
> > > > > >>>> would you like to eventually work together on reusing
the
> > > > > >>>> same code
> > > > > for
> > > > > >>>> ytex?  I have no idea what ytex does, but I know
the ins
> > > > > >>>> and
> > outs
> > > > > >>>> of
> > > > > the
> > > > > >>>> cdl-fast module.
> > > > > >>>>
> > > > > >>>> Sean
> > > > > >>>>
> > > > > >>>> > -----Original Message-----
> > > > > >>>> > From: clayclay911@gmail.com
> > > > > >>>> > [mailto:clayclay911@gmail.com] On
> > > > > Behalf
> > > > > >>>> Of
> > > > > >>>> > Clayton Turner
> > > > > >>>> > Sent: Friday, August 08, 2014 2:08 PM
> > > > > >>>> > To: dev@ctakes.apache.org
> > > > > >>>> > Subject: v_snomed_fword_lookup view
> > > > > >>>> >
> > > > > >>>> > Hi Everyone:
> > > > > >>>> >
> > > > > >>>> > I have a question about how the v_snomed_fword_lookup
> > > > > >>>> > view works
> > > > > when
> > > > > >>>> > running the CPE.
> > > > > >>>> >
> > > > > >>>> > So my understanding of the view is that it
is a view
> > > > > >>>> > comprised of
> > > > > the
> > > > > >>>> > ytex.umls_aui_fword table, the umls.mrconso
table and
> > > > > >>>> > bits/pieces
> > > > > from
> > > > > >>>> > other umls tables.
> > > > > >>>> >
> > > > > >>>> > I feel like this is not completely correct
or my idea of
> > > > > >>>> > how the
> > > > > join
> > > > > >>>> to
> > > > > >>>> > create the view works is off. For example,
let's say I
> > > > > >>>> > want
> > the
> > > > > >>>> > CPE
> > > > > >>>> to find
> > > > > >>>> > "malar ____" (e.g. malar rash) as a concept
in the
> > annotations.
> > > > > >>>> > It
> > > > > >>>> never
> > > > > >>>> > happens after running my CPE descriptor and
I cannot find
> > > > > >>>> > it
> > in
> > > > > >>>> > my v_snomed_fword_lookup view.
> > > > > >>>> >
> > > > > >>>> > select count(*) from umls_aui_fword where fword='malar';
> > yields
> > > > > >>>> > 34
> > > > > >>>> results
> > > > > >>>> >
> > > > > >>>> > select count(*) from umls.mrconso where str='malar';
> > > > > >>>> > yields 3
> > > > > results.
> > > > > >>>> >
> > > > > >>>> > So clearly these two tables know what the cui
and
> > > > > >>>> > context(s) are for
> > > > > >>>> malar
> > > > > >>>> > ____. Yet, whenever I run a gold standard set
of notes
> > > > > >>>> > through the
> > > > > >>>> CPE,
> > > > > >>>> > malar is constantly flagged as just a word
token and the
> > > > > >>>> > concept is
> > > > > >>>> never
> > > > > >>>> > grabbed. This is recurrent for lots of other
concepts, as
> > well,
> > > > > >>>> > I
> > > > > just
> > > > > >>>> > wanted to use an example to illustrate my issue.
> > > > > >>>> >
> > > > > >>>> > Some troubleshooting I already went through:
> > > > > >>>> > 1) Reinstalled ytex and umls database objects
> > > > > >>>> > 2) Reinstalled a second time after redownloading
umls
> > > > > >>>> > through metamorphosys, ensuring that snomed
vocabularies
> > > > > >>>> > were included (also checked file sizes and
noticed a big
> > > > > >>>> > difference so I
> > know
> > > > > >>>> > those vocabularies ARE included
> > > > > >>>> >
> > > > > >>>> > Anyone got any ideas as to what the issue could
be?
> > > > > >>>> >
> > > > > >>>> > Thank you,
> > > > > >>>> > Clayton Turner
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> --
> > > > > >>> Clayton Turner
> > > > > >>> email: caturner3@g.cofc.edu
> > > > > >>> phone: (843)-424-3784
> > > > > >>> web: claytonturner.blogspot.com
> > > > > >>>
> > > > > >>>
> > > > >
> > ----------------------------------------------------------------------
> > > > > ---------------------------
> > > > > >>> “When scientifically investigating the natural world,
the
> > > > > >>> only thing worse than a blind believer is a seeing denier.”
> > > > > >>> - Neil deGrasse Tyson
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> --
> > > > > >> Clayton Turner
> > > > > >> email: caturner3@g.cofc.edu
> > > > > >> phone: (843)-424-3784
> > > > > >> web: claytonturner.blogspot.com
> > > > > >>
> > > > > >>
> > > > >
> > ----------------------------------------------------------------------
> > > > > ---------------------------
> > > > > >> “When scientifically investigating the natural world,
the
> > > > > >> only thing worse than a blind believer is a seeing denier.”
> > > > > >> - Neil deGrasse Tyson
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > --
> > > > > > Clayton Turner
> > > > > > email: caturner3@g.cofc.edu
> > > > > > phone: (843)-424-3784
> > > > > > web: claytonturner.blogspot.com
> > > > > >
> > > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > > ---------------------------
> > > > > > “When scientifically investigating the natural world, the
only
> > thing
> > > > > worse
> > > > > > than a blind believer is a seeing denier.”
> > > > > > - Neil deGrasse Tyson
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --
> > > > > Clayton Turner
> > > > > email: caturner3@g.cofc.edu
> > > > > phone: (843)-424-3784
> > > > > web: claytonturner.blogspot.com
> > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > > --------------------------- “When scientifically investigating
> > > > > the natural world, the only thing worse than a blind believer is
> > > > > a seeing denier.”
> > > > > - Neil deGrasse Tyson
> > > > >
> > >
> >
> 
> 
> 
> --
> --
> Clayton Turner
> email: caturner3@g.cofc.edu
> phone: (843)-424-3784
> web: claytonturner.blogspot.com
> -------------------------------------------------------------------------------------------------
> “When scientifically investigating the natural world, the only thing worse than a
> blind believer is a seeing denier.”
> - Neil deGrasse Tyson
Mime
View raw message