Mailing-List: contact dev-help@ctakes.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ctakes.apache.org
Received-SPF: pass (athena.apache.org: domain of kottmann@gmail.com designates
 209.85.214.51 as permitted sender)
Message-ID: <519CB3F4.20404@gmail.com>
Date: Wed, 22 May 2013 14:03:00 +0200
From: =?ISO-8859-1?Q?J=F6rn_Kottmann?= <kottmann@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130404 Thunderbird/17.0.5
MIME-Version: 1.0
To: dev@ctakes.apache.org
Subject: Re: sentence detector newline behavior
References: <E084D8EFE2B03A408B324458C5212E9421157B32@CHEXMBX3A.CHBOSTON.ORG>
 <996FC801C05DF64A84246A106FACACD010AAD8@MSGPEXCHA08A.mfad.mfroot.org>
 <519B8C79.7060607@childrens.harvard.edu>
 <82291210-B468-49DF-BDC0-BAB09C84CAAE@colorado.edu>
 <E5A9FA5ABBF1CA4085D4F0794852A51E2102B090@CHEXMBX3A.CHBOSTON.ORG>
 <01F1B83B-C2EE-45B5-A47B-8BCE096CD419@colorado.edu>
 <E5A9FA5ABBF1CA4085D4F0794852A51E2102B14D@CHEXMBX3A.CHBOSTON.ORG>
 <AF4BFC93-4C26-43AF-9D8C-57D670D90F94@colorado.edu>
 <519C8D92.7080407@gmail.com>
 <E084D8EFE2B03A408B324458C5212E94211587FD@CHEXMBX3A.CHBOSTON.ORG>
In-Reply-To: <E084D8EFE2B03A408B324458C5212E94211587FD@CHEXMBX3A.CHBOSTON.ORG>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 05/22/2013 01:17 PM, Miller, Timothy wrote:
> That's awesome! It might be worth trying at least. How does the training
> process change? Previously the training data would be one sentence per
> line, but with newlines as possible mid-sentence characters that could
> be trouble, is there a new representation for training data? Or would we
> have to use the training api?

Good point, yes that will be a problem with the default training format, 
but it shouldn't be hard
to solve. In the format itself we could define a new line tag e.g. 
<NEWLINE> to mark new lines.
as a hack to make it work with 1.5.3 you could instead use a special 
char as a replacement
for the new line char.
When you pass the text down to the sentence detector a simple string 
replace could be used to
convert all new line chars to the special new line marker char.

If things work out for you performance wise as well we will just 
integrate it properly into OpenNLP
for the next release.

Could you produce a sentence detector training file with a new line 
marker char?

You should try to pick a char you can also pass in on a terminal 
otherwise you have to use the
API to train the model. The build in cross validation could be used to 
evaluate the performance.

J�rn