ctakes-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Finan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CTAKES-371) update PTB tokenization logic in fast dictionary module
Date Wed, 22 Jul 2015 20:20:05 GMT

    [ https://issues.apache.org/jira/browse/CTAKES-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637554#comment-14637554
] 

Sean Finan commented on CTAKES-371:
-----------------------------------

To be clear, this isn't really a bug in dictionary lookup; there is inconsistency in ctakes
ptb tokenization.  Type the following into ctakes and check out the tokenization differences
for “.2”:

“22q11.2”
22q11   |   .   |   2

“22q11.2 deletion”
22q11   |   .2   |   deletion

“22q11.22 deletion”
22q11   |   .   |   22   |   deletion



> update PTB tokenization logic in fast dictionary module
> -------------------------------------------------------
>
>                 Key: CTAKES-371
>                 URL: https://issues.apache.org/jira/browse/CTAKES-371
>             Project: cTAKES
>          Issue Type: Bug
>          Components: ctakes-dictionary-lookup
>    Affects Versions: 3.2.2
>            Reporter: britt fitch
>            Assignee: Sean Finan
>             Fix For: 3.2.3
>
>
> PTB tokenization logic is used in places like the tokenizer & dictionary building
code. 
> For example, given “22q11.2 deletion syndrome”:
> PTB tokenizer: [22q11, .2, deletion, syndrome]
> Dictionary module:  [22q11, ., 2, deletion, syndrome]   
> (RareWordTermMapCreator.getTokens)
> Dictionary module should be updated to match PTB tokenization logic used elsewhere in
ctakes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message