Mailing-List: contact uima-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: uima-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of dj_mccloskey@ie.ibm.com
 designates 195.212.29.137 as permitted sender)
Subject: RE: Language recognition
To: uima-user@incubator.apache.org
Cc: Marie Wallace <mwallace@ie.ibm.com>
Message-ID: 
 <OFDF6FE49B.47BC0E0D-ON80257519.00695195-80257519.006AA63B@ie.ibm.com>
From: "D.J. McCloskey" <dj_mccloskey@ie.ibm.com>
Date: Mon, 8 Dec 2008 19:24:51 +0000
MIME-Version: 1.0
Content-type: multipart/mixed;
	Boundary="0__=0FBBFF8ADFFAD7058f9e8a93df938690918c0FBBFF8ADFFAD705"
Content-Disposition: inline

--0__=0FBBFF8ADFFAD7058f9e8a93df938690918c0FBBFF8ADFFAD705
Content-type: text/plain; charset=US-ASCII


Hi Tommaso,

I saw the mail below on MarkMail and thought you might find what you need
at http://www.alphaworks.ibm.com/tech/lrw.
There's a new improved version coming soon but as it stands you will find
automatic language identification annotator there which is fast and easy to
improve. It also classifies languages when a sufficient confidence is not
reached into complex text or simple text, essentially indicating whether
ngramming or whitespace tokenization would be appropriate for further
interrogation. Which languages are you interested in?

The technology is available for evaluation and if you have further interest
and would like to know more I'd be happy to help you.

                                           
 Subject: Language recognition(Embedded    
          image moved to file:             
          pic21701.gif)Link to this        
          message                          
                                           
  From:   Tommaso Teofili                  
          (tomm...@gmail.com)              
                                           
  Date:   12/08/2008 01:22:52 AM           
                                           
  List:   org.apache.incubator.uima-user   
                                           

Hello,


I am writing an AE pipeline and i need to recognize in which language the
starting document is written. My idea is to use the Whitespace Tokenizer
and the HMM Tagger together in order to analyze the extracted tokens,
calculate the percentage of well known tokens for each language (against a
dictionary) and then select the highest percentage value language... Do you
know other (better) language recognition methods? Thanks. Tommaso


Regards,
-DJ
-------------------
D.J McCloskey
IBM LanguageWare Architect
Email: dj_mccloskey@ie.ibm.com

... our external website:
http://www-306.ibm.com/software/globalization/topics/languageware/index.jsp
... our Alphaworks: http://www.alphaworks.ibm.com/tech/lrw
... our Wikipedia: http://en.wikipedia.org/wiki/Languageware

IBM Ireland Product Distribution Limited registered in Ireland with number
92815.  Registered office: Oldbrook House, 24-32 Pembroke Road,
Ballsbridge, Dublin 4
--0__=0FBBFF8ADFFAD7058f9e8a93df938690918c0FBBFF8ADFFAD705--