Mailing-List: contact dev-help@ctakes.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ctakes.apache.org
Received-SPF: pass (nike.apache.org: domain of kottmann@gmail.com designates
 74.125.83.49 as permitted sender)
Message-ID: <52E5227F.1000506@gmail.com>
Date: Sun, 26 Jan 2014 15:58:07 +0100
From: =?ISO-8859-1?Q?J=F6rn_Kottmann?= <kottmann@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: "dev@ctakes.apache.org" <dev@ctakes.apache.org>
Subject: Re: sentence detector newline behavior
References: <E084D8EFE2B03A408B324458C5212E9421157B32@CHEXMBX3A.CHBOSTON.ORG>
 <E084D8EFE2B03A408B324458C5212E94211587FD@CHEXMBX3A.CHBOSTON.ORG>
 <519CB3F4.20404@gmail.com> <52DD23AF.3090105@gmail.com>
 <CADGOtThHQw25_KKda6aDLT-+Ruiz5a60VW9AQb3_W4scbV_K1Q@mail.gmail.com>
 <F7475864-AF69-46D6-A104-CEE1F7B2A346@childrens.harvard.edu>
 <52DE4BFD.803@gmail.com> <d2fb82$86rb39@ironport10.mayo.edu>
 <393252F14C42F946952F1ED75D316CAD3865AE43@CHEXMBX2A.CHBOSTON.ORG>
 <d2fb82$86ssqn@ironport10.mayo.edu>
 <393252F14C42F946952F1ED75D316CAD3865AE84@CHEXMBX2A.CHBOSTON.ORG>
 <393252F14C42F946952F1ED75D316CAD3865BE97@CHEXMBX2A.CHBOSTON.ORG>
 <CADGOtTiSN0XQoBmYwQ7KSbGBSTvbghSrNJm7eXMggraB6fSd_g@mail.gmail.com>
 <CAOf_dRkSOBWnnHGXXr4h_=r+Jsbsb9J4PsOQUuoDq7eAAGpn_A@mail.gmail.com>
 <52E1844D.3010507@childrens.harvard.edu> <52E2D79C.60101@gmail.com>
 <E084D8EFE2B03A408B324458C5212E94212EF4D9@CHEXMBX3A.CHBOSTON.ORG>
 <52E3F32B.8090604@gmail.com>
 <E084D8EFE2B03A408B324458C5212E94212F033E@CHEXMBX3A.CHBOSTON.ORG>
In-Reply-To: <E084D8EFE2B03A408B324458C5212E94212F033E@CHEXMBX3A.CHBOSTON.ORG>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 01/25/2014 10:03 PM, Miller, Timothy wrote:
> On 01/25/2014 12:24 PM, J�rn Kottmann wrote:
>> The code which computes the spans tries to remove white space from it.
>> Removing the white space from a whitespace only sentence is causing
>> the exception your are seeing. Which response would you expect from
>> the sentence detector? Should a white space only sentence be returned?
> I would say no.
>
>> In case a sentence is terminated by a new line. Should the new line
>> char be included in the sentence span or not?
> I would also say no.
>
>
> I made a quick patch for this issue -- now it runs but scores really
> poorly compared to my model file (30 vs 75 or so). I suspect something
> is wrong with the evaluation, the spans being slightly off somehow.

The evaluation should ignore white spaces. I committed now my fix, it 
would be nice if you can
test it.

There might be still something wrong. In my test data I replaced all 
question marks with white spaces, and the result
is slightly worse than with the original data.

J�rn