Return-Path: Delivered-To: apmail-lucene-solr-dev-archive@minotaur.apache.org Received: (qmail 29862 invoked from network); 8 Jan 2010 21:36:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 8 Jan 2010 21:36:20 -0000 Received: (qmail 4955 invoked by uid 500); 8 Jan 2010 21:36:18 -0000 Delivered-To: apmail-lucene-solr-dev-archive@lucene.apache.org Received: (qmail 4811 invoked by uid 500); 8 Jan 2010 21:36:18 -0000 Mailing-List: contact solr-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-dev@lucene.apache.org Delivered-To: mailing list solr-dev@lucene.apache.org Received: (qmail 4793 invoked by uid 99); 8 Jan 2010 21:36:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 21:36:18 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 21:36:16 +0000 Received: from brutus.apache.org (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 3CF39234C1E9 for ; Fri, 8 Jan 2010 13:35:55 -0800 (PST) Message-ID: <1613141994.129601262986555248.JavaMail.jira@brutus.apache.org> Date: Fri, 8 Jan 2010 21:35:55 +0000 (UTC) From: "Yonik Seeley (JIRA)" To: solr-dev@lucene.apache.org Subject: [jira] Commented: (SOLR-1657) convert the rest of solr to use the new tokenstream API In-Reply-To: <1476370607.1260898218157.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/SOLR-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798188#action_12798188 ] Yonik Seeley commented on SOLR-1657: ------------------------------------ What about preserving the attributes for just the first token? That makes a lot of sense in many cases (say when WDF is just removing punctuation). So if preserveOriginal==true, the first token would always be the original. This should also be the most performant since it's just a modification to the first token (offset and termText)? > convert the rest of solr to use the new tokenstream API > ------------------------------------------------------- > > Key: SOLR-1657 > URL: https://issues.apache.org/jira/browse/SOLR-1657 > Project: Solr > Issue Type: Task > Reporter: Robert Muir > Attachments: SOLR-1657.patch, SOLR-1657.patch > > > org.apache.solr.analysis: > BufferedTokenStream > -> -CommonGramsFilter- > -> -CommonGramsQueryFilter- > -> -RemoveDuplicatesTokenFilter- > -CapitalizationFilterFactory- > -HyphenatedWordsFilter- > -LengthFilter (deprecated, remove)- > SynonymFilter > SynonymFilterFactory > WordDelimiterFilter > org.apache.solr.handler: > AnalysisRequestHandler > AnalysisRequestHandlerBase > org.apache.solr.handler.component: > QueryElevationComponent > SpellCheckComponent > org.apache.solr.highlight: > DefaultSolrHighlighter > org.apache.solr.search: > FieldQParserPlugin > org.apache.solr.spelling: > SpellingQueryConverter -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.