Mailing-List: contact dev-help@ctakes.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ctakes.apache.org
Subject: Re: Combining Knowledge- and Data-driven Methods for
 De-identification of Clinical Narratives
To: dev@ctakes.apache.org
References: 
 <CAPqz87osK7XfwK-fn_tox9hhCzfNG_Ew3mJ-SF0jPF-6rR0K+A@mail.gmail.com>
 <56165C26.5010908@averbis.com>
 <3c05baec8f9749d4b1e61703aeb0178b@CHEXMAIL5A.CHBOSTON.ORG>
 <CAPMhGj6OeTpQiihvuUHf4p6u9vtvtz3WM=-QQPw=h-TqXkTfcw@mail.gmail.com>
 <CAMgZBShSmGdu5twB9xDcDHC0tD1Vcmv6fiJrCdwnJtSGvJQoEg@mail.gmail.com>
 <56372554.6050000@averbis.com>
 <CAPqz87os=O6ZJuasifEOJVV4o-OfcSbcPEp729osWENrsJDPfw@mail.gmail.com>
 <5638CE7B.5070000@averbis.com>
 <CAPqz87q8E-wyj43AEKU9B+h7KbGqgjM0boE88mbDwqTa+w8LuA@mail.gmail.com>
 <CAPMhGj4HAL_88VefgBw_hretFGijp81YWrc-YF5qv4DKmi5P7g@mail.gmail.com>
 <563C6702.6010909@averbis.com>
 <CAPqz87pV_BGNKVqTzCtcQfrt28jSBD6P8Z2WOJcCfUc72Ey=tg@mail.gmail.com>
 <5654346D.7050503@averbis.com>
 <CAPMhGj6a8Nj6VZvAv-rLsnS=8v4tUJitWp74WCWJ7h4qy32RfQ@mail.gmail.com>
 <5673D981.4040309@averbis.com> <5693E8E8.7060008@averbis.com>
 <68DE0D2C-1BB6-44F4-806C-1BE7CF230029@wiredinformatics.com>
From: =?UTF-8?Q?Peter_Kl=c3=bcgl?= <peter.kluegl@averbis.com>
Message-ID: <5694C3B1.9010409@averbis.com>
Date: Tue, 12 Jan 2016 10:13:21 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <68DE0D2C-1BB6-44F4-806C-1BE7CF230029@wiredinformatics.com>
Content-Type: multipart/alternative;
 boundary="------------060909050606030500000804"

--------------060909050606030500000804
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Hi,

Ruta requires a newer uima version since Marshall fixed some bugs. In
previous ruta versions there were workarounds but I removed them in the
latest version(s). Thus, using ruta with such an old uima version could
lead to flawed rule inference.

I would recommend ugrading to the latest uima version but this is not a
trivial task and I do not know how good the test coverage of cTAKES is.
I managed the uima version to 2.8.1 for now in the deid project.

There's a ClassNotFoundException of a class that is not part of xml-apis
1.0 but 1.4. I added the stacktrace below.

There was also a problem when merging the patch. I will fix that with
the next patch.

Best,

Peter


log4j:WARN Error during default initialization
java.lang.NoClassDefFoundError: org/w3c/dom/ElementTraversal
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.xerces.parsers.AbstractDOMParser.startDocument(Unknown
Source)
    at org.apache.xerces.impl.dtd.XMLDTDValidator.startDocument(Unknown
Source)
    at org.apache.xerces.impl.XMLDocumentScannerImpl.startEntity(Unknown
Source)
    at
org.apache.xerces.impl.XMLVersionDetector.startDocumentParsing(Unknown
Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at
org.apache.log4j.xml.DOMConfigurator$2.parse(DOMConfigurator.java:767)
    at
org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:866)
    at
org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:773)
    at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.apache.log4j.Logger.getLogger(Logger.java:104)
    at
org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:289)
    at
org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:109)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at
org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1116)
    at
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:914)
    at
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:604)
    at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:336)
    at
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:310)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
    at
org.springframework.core.io.support.PathMatchingResourcePatternResolver.<clinit>(PathMatchingResourcePatternResolver.java:169)
    at
org.apache.uima.fit.internal.MetaDataUtil.resolve(MetaDataUtil.java:92)
    at
org.apache.uima.fit.internal.MetaDataUtil.scanImportsAndManifests(MetaDataUtil.java:65)
    at
org.apache.uima.fit.internal.MetaDataUtil.scanDescriptors(MetaDataUtil.java:170)
    at
org.apache.uima.fit.factory.TypeSystemDescriptionFactory.scanTypeDescriptors(TypeSystemDescriptionFactory.java:131)
    at
org.apache.uima.fit.factory.TypeSystemDescriptionFactory.createTypeSystemDescription(TypeSystemDescriptionFactory.java:102)
    at
org.apache.uima.fit.factory.JCasFactory.createJCas(JCasFactory.java:50)
    at
org.apache.ctakes.deid.DeidPipelineTest.test(DeidPipelineTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.ClassNotFoundException: org.w3c.dom.ElementTraversal
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 70 more

Am 11.01.2016 um 23:52 schrieb Pei Chen:
> Patch applied:
> http://svn.apache.org/repos/asf/ctakes/sandbox/ctakes-clinical-deid/
> Thanks Peter.
>
> What error did you get with xml-api’s?  Do you mean upgrade ctakes to
> the latest version of uima instead of 2.4.0?
>
> —Pei
>
>> On Jan 11, 2016, at 12:39 PM, Peter Klügl <peter.kluegl@averbis.com
>> <mailto:peter.kluegl@averbis.com>> wrote:
>>
>> Hi,
>>
>> I just added a small patch which adds a maven build process and a dummy
>> unit test.
>>
>> I had some problems with the version of xml-apis. Is this known or
>> rather a local problem on my build machine?
>> Is there a reason why cTAKES requires uima 2.4.0?
>>
>> Next step would be translating the rules. Azad mentioned that he already
>> started with that :-)
>>
>> Best,
>>
>> Peter
>>
>>
>> Am 18.12.2015 um 11:01 schrieb Peter Klügl:
>>> Hi,
>>>
>>> sorry, there was no free time left in December for this issue, but I
>>> will be able to provide the patches in January (for real).
>>>
>>> Best,
>>>
>>> Peter
>>>
>>> Am 24.11.2015 um 11:37 schrieb Azad Dehghan:
>>>> This is on my todo list for Dec. as well. If there are any more
>>>> volunteers
>>>> for translating JAPE to RUTA, please get in touch.
>>>>
>>>> Cheers,
>>>> Azad
>>>>
>>>> On 24 Nov 2015 09:55, "Peter Klügl" <peter.kluegl@averbis.com
>>>> <mailto:peter.kluegl@averbis.com>> wrote:
>>>>> Hi,
>>>>>
>>>>> I just wanted to mention that I haven't forgot about it.
>>>>> Unfortunately,
>>>>> there is just no spare time right now. I hope I will be able to
>>>>> provide
>>>>> the patches in December.
>>>>>
>>>>> Best,
>>>>>
>>>>> Peter
>>>>>
>>>>> Am 06.11.2015 um 16:40 schrieb Pei Chen:
>>>>>> Hi Peter,
>>>>>> I think the ctakes-examples is probably a good starting point at
>>>>>> least
>>>>>> in terms of maven modules, etc.  I think it would be good if we use
>>>>>> uimaFIT style as primary approach to wiring components together and
>>>>>> generate desc's as secondary...
>>>>>> I think the actual components that would be required is probably best
>>>>>> left up to what is actually required for best performing c-deid.  The
>>>>>> output would be interesting, I'm not sure if we should treat this as
>>>>>> an independent preprocessing component or part of a pipeline (in
>>>>>> which
>>>>>> case, we may need to propose a change to the type system or
>>>>>> perhaps an
>>>>>> alternative JCas view.  You can probably open up that discussion to
>>>>>> the dev group as you see fit.)
>>>>>>
>>>>>> My 2 cents...
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 6, 2015 at 3:38 AM, Peter Klügl
>>>>>> <peter.kluegl@averbis.com <mailto:peter.kluegl@averbis.com>>
>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Is there a cTAKES project that may serve as an example on how the
>>>> cTAKES
>>>>>>> community develops or how a project should look like?
>>>>>>> I learned that different people set up UIMA project in a quite
>>>> different
>>>>>>> manner and I do not what to get inspired by "some sort of out-dated"
>>>>>>> approach in the cTAKES repo.
>>>>>>>
>>>>>>> Are there restriction or preferences about the preprocessing
>>>>>>> components
>>>>>>> that should be used and the kind of "output" of the project.
>>>>>>> Components: On which components may the componetns rely:
>>>>>>> tokenizer, ...
>>>>>>> parser, ... dict lookup?
>>>>>>> "output": Should the project provide a pipeline or a single AE?
>>>>>>>
>>>>>>> More comments below.
>>>>>>>
>>>>>>> Am 03.11.2015 um 16:54 schrieb Azad Dehghan:
>>>>>>>>> Who else plans to provide patches for it? Just to avoid duplicate
>>>> work
>>>>>>>>> and to coordnate the efforts ...
>>>>>>>>>
>>>>>>>> I would like to help with the translating JAPE to RUTA.
>>>>>>> You can already go ahead with the UIMA Ruta Workbench if you
>>>>>>> want, or
>>>>>>> wait until I set up the project with ruta integration.
>>>>>>>
>>>>>>> If any questions arise, just ask :-)
>>>>>>>
>>>>>>>>> Is there a development dataset which was utilized for the initial
>>>>>>>>> development, and if yes, is it possible to contribute it too?
>>>>>>>>>
>>>>>>>> The data set is unfortunately not publicly available; i2b2
>>>>>>>> <https://www.i2b2.org/NLP/DataSets/Main.php> typically releases the
>>>> data
>>>>>>>> sets 12 months after a given challenge; this is done on an
>>>>>>>> individual
>>>> basis
>>>>>>>> and involve a Data Use Agreement.
>>>>>>>>
>>>>>>>> However, I will be able to conduct and coordinate the validation.
>>>>>>>>
>>>>>>> Ok, I'll investigate if we have already access to the dataset here.
>>>>>>>
>>>>>>>
>>>>>>>>> My first step would be:
>>>>>>>>> - set up a maven project
>>>>>>>>> - set up a development pipeline in a test (with cTAKES components
>>>>>>>>> replacing the previous ANNIE preprocessing)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But one item that we need to review is the 3rd party libs jars
>>>>>>>>> that
>>>>>>>>> were included to ensure compatibility.  I’ll be sure to take a
>>>>>>>>> look
>>>> at
>>>>>>>>> that over the next few weeks.
>>>>>>>>>
>>>>>>>>> —Pei
>>>>>>>>>
>>>>>>>>>
>>>>>>>> @Pei - once ANNIE components are replaced there is should not be a
>>>> need to
>>>>>>>> worry about the 3rd party libs.
>>>>>>>>
>>>>>>>> Also, just a thought: we may want to create an independent
>>>>>>>> component
>>>> for
>>>>>>>> the Two Pass recognition (TwoPass.java) as this method have shown
>>>> useful
>>>>>>>> for general NER on longitudinal data and surely useful
>>>>>>>> independent of
>>>> the
>>>>>>>> deid component.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Azad
>>>>>>>>
>>
>


--------------060909050606030500000804--