Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3D8DD45A2 for ; Wed, 11 May 2011 13:16:54 +0000 (UTC) Received: (qmail 66389 invoked by uid 500); 11 May 2011 13:16:52 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 66352 invoked by uid 500); 11 May 2011 13:16:52 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 66344 invoked by uid 99); 11 May 2011 13:16:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:16:51 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [128.230.18.82] (HELO smtp1.syr.edu) (128.230.18.82) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:16:43 +0000 Received: from suex07-hub-02.ad.syr.edu (suex07-hub-02.ad.syr.edu [128.230.108.196]) by smtp1.syr.edu (8.14.3/8.14.3) with ESMTP id p4BDGL9i005708 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 11 May 2011 09:16:22 -0400 Received: from suex07-mbx-03.ad.syr.edu ([128.230.108.133]) by suex07-hub-02.ad.syr.edu ([128.230.108.196]) with mapi; Wed, 11 May 2011 09:16:22 -0400 From: Steven A Rowe To: "java-user@lucene.apache.org" Date: Wed, 11 May 2011 09:16:20 -0400 Subject: RE: Can I omit ShingleFilter's filler tokens Thread-Topic: Can I omit ShingleFilter's filler tokens Thread-Index: AcwP266BMb0te5zNT8i7FTGsJVJ7kwAAT2+Q Message-ID: <2D127F11DC79714E9B6A43AC9458147FB3CE739E@suex07-mbx-03.ad.syr.edu> References: <2D127F11DC79714E9B6A43AC9458147FB3CE7396@suex07-mbx-03.ad.syr.edu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.2.15,1.0.148,0.0.0000 definitions=2011-05-11_06:2011-05-11,2011-05-11,1970-01-01 signatures=0 X-Proofpoint-Spam-Reason: safe X-Virus-Checked: Checked by ClamAV on apache.org WWVzLCBTdG9wRmlsdGVyLnNldEVuYWJsZVBvc2l0aW9uSW5jcmVtZW50cyhmYWxzZSkgd2lsbCBh bG1vc3QgY2VydGFpbmx5IGdldCBoaWdoZXIgdGhyb3VnaHB1dCB0aGFuIGluc2VydGluZyBQb3Np dGlvbkZpbHRlci4gIExpa2UgUG9zaXRpb25GaWx0ZXIsIHRoaXMgd2lsbCBidXkgeW91ICMyIChj cmVhdGUgc2hpbmdsZXMgYXMgaWYgc3RvcHdvcmRzIHdlcmUgbmV2ZXIgdGhlcmUpLCBidXQgbm90 ICMxIChkb24ndCBjcmVhdGUgc2hpbmdsZXMgYWNyb3NzIHN0b3B3b3JkcykuDQoNCj4gLS0tLS1P cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogUm9iZXJ0IE11aXIgW21haWx0bzpyY211aXJA Z21haWwuY29tXQ0KPiBTZW50OiBXZWRuZXNkYXksIE1heSAxMSwgMjAxMSA5OjAyIEFNDQo+IFRv OiBqYXZhLXVzZXJAbHVjZW5lLmFwYWNoZS5vcmcNCj4gU3ViamVjdDogUmU6IENhbiBJIG9taXQg U2hpbmdsZUZpbHRlcidzIGZpbGxlciB0b2tlbnMNCj4gDQo+IGFub3RoZXIgaWRlYSBpcyB0byAu c2V0RW5hYmxlUG9zaXRpb25JbmNyZW1lbnRzKGZhbHNlKSBvbiB5b3VyDQo+IHN0b3BmaWx0ZXIu DQo+IA0KPiBPbiBXZWQsIE1heSAxMSwgMjAxMSBhdCA4OjI3IEFNLCBTdGV2ZW4gQSBSb3dlIDxz YXJvd2VAc3lyLmVkdT4gd3JvdGU6DQo+ID4gSGkgQmlsbCwNCj4gPg0KPiA+IEkgY2FuIHRoaW5r IG9mIHR3byBwb3NzaWJsZSBpbnRlcnByZXRhdGlvbnMgb2YgInJlbW92aW5nIGZpbGxlcg0KPiB0 b2tlbnMiOg0KPiA+DQo+ID4gMS4gRG9uJ3QgY3JlYXRlIHNoaW5nbGVzIGFjcm9zcyBzdG9wd29y ZHMsIGUuZy4gZm9yIHRleHQgIm9uZSB0d28gdGhyZWUNCj4gZm91ciBmaXZlIiBhbmQgc3RvcHdv cmQgInRocmVlIiwgYmlncmFtcyBvbmx5LCB5b3UnZCBnZXQgKCJvbmUgdHdvIiwNCj4gImZvdXIg Zml2ZSIpLCBpbnN0ZWFkIG9mIHRoZSBjdXJyZW50ICgib25lIHR3byIsICJ0d28gXyIsICJfIGZv dXIiLCAiZm91cg0KPiBmaXZlIikuDQo+ID4NCj4gPiAyLiBDcmVhdGUgc2hpbmdsZXMgYXMgaWYg dGhlIHN0b3B3b3JkcyB3ZXJlIG5ldmVyIHRoZXJlLCBlLmcuIGZvciB0aGUNCj4gc2FtZSB0ZXh0 IGFuZCBzdG9wd29yZCwgYmlncmFtcyBvbmx5LCB5b3UnZCBnZXQgKCJvbmUgdHdvIiwgInR3byBm b3VyIiwNCj4gImZvdXIgZml2ZSIpLg0KPiA+DQo+ID4gV2hpY2ggb25lIGRpZCB5b3UgaGF2ZSBp biBtaW5kPyDCoCMyIGNhbiBiZSBhY2hpZXZlZCBieSBhZGRpbmcNCj4gUG9zaXRpb25GaWx0ZXIg YWZ0ZXIgU3RvcEZpbHRlciBhbmQgYmVmb3JlIFNoaW5nbGVGaWx0ZXIuIMKgSSB0aGluayAjMQ0K PiByZXF1aXJlcyBTaGluZ2xlRmlsdGVyIG1vZGlmaWNhdGlvbnMuDQo+ID4NCj4gPiBTdGV2ZQ0K PiA+DQo+ID4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4+IEZyb206IFdpbGxpYW0g S29zY2hvIFttYWlsdG86d2tvc2Nob0BnbWFpbC5jb21dDQo+ID4+IFNlbnQ6IFdlZG5lc2RheSwg TWF5IDExLCAyMDExIDEyOjA1IEFNDQo+ID4+IFRvOiBqYXZhLXVzZXJAbHVjZW5lLmFwYWNoZS5v cmcNCj4gPj4gU3ViamVjdDogQ2FuIEkgb21pdCBTaGluZ2xlRmlsdGVyJ3MgZmlsbGVyIHRva2Vu cw0KPiA+Pg0KPiA+PiBIaSwNCj4gPj4NCj4gPj4gQ2FuIEkgcmVtb3ZlIHRoZSBmaWxsZXIgdG9r ZW4gXyBmcm9tIHRoZSBuLWdyYW0tdG9rZW5zIHRoYXQgYXJlDQo+IGdlbmVyYXRlZA0KPiA+PiBi eQ0KPiA+PiBhIFNoaW5nbGVGaWx0ZXI/DQo+ID4+DQo+ID4+IEknbSB1c2luZyBhIGNoYWluIG9m IGZpbHRlcnM6IENsYXNzaWNGaWx0ZXIsIFN0b3BGaWx0ZXIsDQo+IExvd2VyQ2FzZUZpbHRlciwN Cj4gPj4gYW5kIFNoaW5nbGVGaWx0ZXIgdG8gY3JlYXRlIHBocmFzZSBuLWdyYW1zLiDCoFRoZSBT aGluZ2xlRmlsdGVyIGluc2VydHMNCj4gPj4gRklMTEVSX1RPS0VOcyBpbiBwbGFjZSBvZiB0aGUg c3RvcHdvcmRzLCBidXQgSSBkb24ndCB3YW50IHRoZW0uDQo+ID4+DQo+ID4+IEhvdyBjYW4gSSBv bWl0IHRoZSBmaWxsZXIgdG9rZW5zPw0KPiA+Pg0KPiA+PiB0aGFua3MNCj4gPj4gQmlsbA0KPiA+ DQo+IA0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gVG8gdW5zdWJzY3JpYmUsIGUtbWFpbDogamF2YS11c2Vy LXVuc3Vic2NyaWJlQGx1Y2VuZS5hcGFjaGUub3JnDQo+IEZvciBhZGRpdGlvbmFsIGNvbW1hbmRz LCBlLW1haWw6IGphdmEtdXNlci1oZWxwQGx1Y2VuZS5hcGFjaGUub3JnDQoNCg==