Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C400F9D7D for ; Wed, 19 Oct 2011 14:18:10 +0000 (UTC) Received: (qmail 96225 invoked by uid 500); 19 Oct 2011 14:18:08 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 96181 invoked by uid 500); 19 Oct 2011 14:18:08 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 96173 invoked by uid 99); 19 Oct 2011 14:18:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Oct 2011 14:18:08 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [128.230.18.92] (HELO smtp2.syr.edu) (128.230.18.92) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Oct 2011 14:17:59 +0000 Received: from suex10-hub-02.ad.syr.edu (suex10-hub-02.ad.syr.edu [128.230.108.125]) by smtp2.syr.edu (8.14.3/8.14.3) with ESMTP id p9JEHa7w020336 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 19 Oct 2011 10:17:38 -0400 Received: from SUEX10-MBX-03.ad.syr.edu ([128.230.108.186]) by suex10-hub-02.ad.syr.edu ([128.230.108.125]) with mapi id 14.01.0218.012; Wed, 19 Oct 2011 10:17:37 -0400 From: Steven A Rowe To: "java-user@lucene.apache.org" Subject: RE: How do you see if a tokenstream has tokens without consuming the tokens ? Thread-Topic: How do you see if a tokenstream has tokens without consuming the tokens ? Thread-Index: AQHMjMYzxZSGPtTNCkquHQrzKR+klJWBfo6QgAJElwD///a9EA== Date: Wed, 19 Oct 2011 14:17:35 +0000 Message-ID: <6C78E97C707B5B4C8CC61D44F8754586033202@SUEX10-mbx-03.ad.syr.edu> References: <4E9C1BCB.7080900@fastmail.fm> <6C78E97C707B5B4C8CC61D44F8754586032615@SUEX10-mbx-03.ad.syr.edu> <4E9EAB4C.2070507@fastmail.fm> In-Reply-To: <4E9EAB4C.2070507@fastmail.fm> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.230.84.107] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.4.6813,1.0.211,0.0.0000 definitions=2011-10-19_04:2011-10-19,2011-10-19,1970-01-01 signatures=0 X-Proofpoint-Spam-Reason: safe SGkgUGF1bCwNCg0KV2hhdCB2ZXJzaW9uIG9mIEx1Y2VuZSBhcmUgeW91IHVzaW5nPyAgVGhlIEpG bGV4IHNwZWMgeW91IHF1b3RlIGJlbG93IGxvb2tzIHByZS12My4xPw0KDQpTdGV2ZQ0KDQo+IC0t LS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IFBhdWwgVGF5bG9yIFttYWlsdG86cGF1 bF90MTAwQGZhc3RtYWlsLmZtXQ0KPiBTZW50OiBXZWRuZXNkYXksIE9jdG9iZXIgMTksIDIwMTEg Njo1MCBBTQ0KPiBUbzogU3RldmVuIEEgUm93ZTsgamF2YS11c2VyQGx1Y2VuZS5hcGFjaGUub3Jn ID4+ICInamF2YS0NCj4gdXNlckBsdWNlbmUuYXBhY2hlLm9yZyciDQo+IFN1YmplY3Q6IFJlOiBI b3cgZG8geW91IHNlZSBpZiBhIHRva2Vuc3RyZWFtIGhhcyB0b2tlbnMgd2l0aG91dCBjb25zdW1p bmcNCj4gdGhlIHRva2VucyA/DQo+IA0KPiBPbiAxOC8xMC8yMDExIDA1OjE5LCBTdGV2ZW4gQSBS b3dlIHdyb3RlOg0KPiA+IEhpIFBhdWwsDQo+ID4NCj4gPiBZb3UgY291bGQgYWRkIGEgcnVsZSB0 byB0aGUgU3RhbmRhcmRUb2tlbml6ZXIgSkZsZXggZ3JhbW1hciB0byBoYW5kbGUNCj4gdGhpcyBj YXNlLCBieXBhc3NpbmcgaXRzIG90aGVyIHJ1bGVzLg0KPiBUSGlzIHNlZW1lZCB0byBiZSB3b3Jr aW5nLCBqdXN0IHRvIHRlc3QgaXQgb3V0IEkgY2hhbmdlZCB0aGUgRU1BSUwgb25lDQo+IHRvIHRo aXMNCj4gDQo+IEVNQUlMICAgICA9ICAoIiEifCIqInwiXiJ8IiEifCIuInwiQCJ8IiUifCLimaAi fCJcIiIpKw0KPiANCj4gQW5kIGNoYW5nZWQgdGhlIG9yZGVyIHRoZSB0b2tlbnMgd2VyZSBjaGVj a2VkDQo+IA0KPiAlJQ0KPiANCj4ge0FMUEhBTlVNfSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgeyByZXR1cm4NCj4gQUxQSEFOVU07IH0NCj4ge0FQ T1NUUk9QSEV9ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgeyByZXR1cm4NCj4gQVBPU1RST1BIRTsgfQ0KPiB7QUNST05ZTX0gICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB7IHJldHVybg0KPiBBQ1JPTllN OyB9DQo+IHtDT01QQU5ZfSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHsgcmV0dXJuDQo+IENPTVBBTlk7IH0NCj4ge0hPU1R9ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeyByZXR1cm4NCj4g SE9TVDsgfQ0KPiB7TlVNfSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB7IHJldHVybg0KPiBOVU07IH0NCj4ge0NKfSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeyByZXR1cm4NCj4g Q0o7IH0NCj4ge0FDUk9OWU1fREVQfSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgeyByZXR1cm4NCj4gQUNST05ZTV9ERVA7IH0NCj4ge0VNQUlMfSAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeyByZXR1 cm4NCj4gRU1BSUw7IH0NCj4gDQo+IC8qKiBJZ25vcmUgdGhlIHJlc3QgKi8NCj4gLiB8IHtXSElU RVNQQUNFfSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeyAv Kg0KPiBpZ25vcmUgKi8gfQ0KPiANCj4gDQo+IFNvIHRoZW4gaWYgSSBwYXNzZWQgIiEhIScgdG8g dGhlIHRva2VuaXplciwgaXQga2VwdCBpdCB3aGljaCB3YXMgZXhhY3RseQ0KPiB3aGF0IEkgd2Fu dGVkDQo+IA0KPiBIb3dldmVyIGlmIEkgcGFzc2VkIGl0ICdmcmVkISEhJyBpdCAgc3BsaXQgaXQg aW50byB0d28gdG9rZW5zDQo+IA0KPiAnZnJlZCcgYW5kICchISEnDQo+IA0KPiB3aGljaCBpcyBu b3Qgd2hhdCBJIHdhbnRlZCwgSSBqdXN0IHdhbnRlZCB0byBnZXQgYmFjaw0KPiANCj4gZnJlZA0K PiANCj4gDQo+IEkgdHJpZWQgY2huYWdpbmcgRU1BSUwgdG8NCj4gDQo+IEVNQUlMICAgICA9ICBe KCIhInwiKiJ8Il4ifCIhInwiLiJ8IkAifCIlInwi4pmgInwiXCIiKSsNCj4gDQo+IGJ1dCB1c2Ug b2YgXiBhbmQgJCBzZWVtIHRvIGJlIGRpc2FsbG93ZWQsIHNvIEkgY2FudCBzZWUgaWYgdGhlcmUg aXMNCj4gYW55d2F5IHRvIGRvIHdoYXQgSSB3YW50IGluIHRoZSBqZmxleCwgaWYgdGhhdHMgdGhl IGNhc2UgY2FuIEkgZHJvcCB0aGUNCj4gMm5kIGZpbHRlciBzb21laG93IGluIGEgc3Vic2VxdWVu dCBmaWx0ZXIgPw0KPiANCj4gDQo+IFBhdWwNCj4gDQo+IA0KPiANCj4gDQo+IA0KPiANCj4gPg0K PiA+IEFub3RoZXIgb3B0aW9uIGlzIHRvIGNyZWF0ZSBhIGNoYXIgZmlsdGVyIHRoYXQgc3Vic3Rp dHV0ZXMgUFVOQ1QtDQo+IEVYQ0xBTUFUSU9OIGZvciBleGNsYW1hdGlvbiBwb2ludHMsIFBVTkNU LVBFUklPRCBmb3IgcGVyaW9kcywgZXRjLiwgYnV0DQo+IG9ubHkgd2hlbiB0aGUgZW50aXJlIGlu cHV0IGNvbnNpc3RzIGV4Y2x1c2l2ZWx5IG9mIHdoaXRlc3BhY2UgYW5kDQo+IHB1bmN0dWF0aW9u LiAgVGhlc2Ugc3ltYm9scyB3b3VsZCB0aGVuIGJlIGxlZnQgaW50YWN0IGJ5DQo+IFN0YW5kYXJk VG9rZW5pemVyLg0KPiA+DQo+ID4gU3RldmUNCj4gPg0KPiA+PiAtLS0tLU9yaWdpbmFsIE1lc3Nh Z2UtLS0tLQ0KPiA+PiBGcm9tOiBQYXVsIFRheWxvciBbbWFpbHRvOnBhdWxfdDEwMEBmYXN0bWFp bC5mbV0NCj4gPj4gU2VudDogTW9uZGF5LCBPY3RvYmVyIDE3LCAyMDExIDg6MTMgQU0NCj4gPj4g VG86ICdqYXZhLXVzZXJAbHVjZW5lLmFwYWNoZS5vcmcnDQo+ID4+IFN1YmplY3Q6IEhvdyBkbyB5 b3Ugc2VlIGlmIGEgdG9rZW5zdHJlYW0gaGFzIHRva2VucyB3aXRob3V0IGNvbnN1bWluZw0KPiB0 aGUNCj4gPj4gdG9rZW5zID8NCj4gPj4NCj4gPj4NCj4gPj4gV2UgaGF2ZSBhIG1vZGlmaWVkIHZl cnNpb24gb2YgYSBMdWNlbmUgU3RhbmRhcmRBbmFseXplciAsIHdlIHVzZSBpdA0KPiBmb3INCj4g Pj4gdG9rZW5pemluZyBtdXNpYyBtZXRhZGF0YSBzdWNoIGFzIGFzIGFydGlzdCBuYW1lcyYgIHNv bmcgdGl0bGVzLCBzbw0KPiA+PiB0eXBpY2FsbHkgb25seSBhIGZldyB3b3Jkcy4gT24gdG9rZW5p emluZyBpdCB1c3VhbGx5IGl0IHN0cmlwcyBvdXQNCj4gPj4gcHVuY3R1YXRpb25zIHdoaWNoIGlz IGNvcnJlY3QsIGhvd2V2ZXIgaWYgdGhlIGlucHV0IHRleHQgY29uc2lzdHMgb2YNCj4gPj4gb25s eSBwdW5jdHVhdGlvbiBjaGFyYWN0ZXJzIHRoZW4gd2UgZW5kIHVwIHdpdGggbm90aGluZywgZm9y IHRoZXNlDQo+ID4+IHBhcnRpY3VsYXIgUkFSRSBjYXNlcyBJIHdhbnQgdG8gdXNlIGEgbWFwcGlu ZyBmaWx0ZXIuDQo+ID4+DQo+ID4+IFNvIHdoYXQgSSB0cnkgdG8gZG8gaXMgaGF2ZSBteSBhbmFs eXplciB0b2tlbml6ZSBhcyBub3JtYWwsIHRoZW4gaWYNCj4gdGhlDQo+ID4+IHJlc3VsdHMgaXMg bm8gdG9rZW5zIHJldG9rZW5pemUgd2l0aCB0aGUgbWFwcGluZyBmaWx0ZXIgLCBJIGNoZWNrIGl0 DQo+IGhhcw0KPiA+PiBubyB0b2tlbiB1c2luZyBpbmNyZW1lbnRUb2tlbigpIGJ1dCB0aGVuIGNh bnQgc2VlIGhvdyBJDQo+ID4+IGRlY3JlbWVudFRva2VuKCkuIEhvdyBjYW4gSSBkbyB0aGlzLCBv ciBpcyB0aGVyZSBhIG1vcmUgZWZmaWNpZW50IHdheQ0KPiBvZg0KPiA+PiBkb2luZyB0aGlzLiBO b3RlIG9mIG1heWJlIDEwLDAwMCwwMDAgcmVjb3JkcyBvbmx5IGEgZmV3IDEwMCByZWNvcmRzDQo+ IHdpbGwNCj4gPj4gaGF2ZSB0aGlzIHByb2JsZW0gc28gSSBuZWVkIGEgc29sdXRpb24gd2hpY2gg ZG9lc24ndCBpbXBhY3QNCj4gcGVyZm9ybWFuY2UNCj4gPj4gdW5yZWFzb25hYmx5Lg0KPiA+Pg0K PiA+PiAgICAgICBOb3JtYWxpemVDaGFyTWFwIHNwZWNpYWxjaGFyQ29udmVydE1hcCA9IG5ldyBO b3JtYWxpemVDaGFyTWFwKCk7DQo+ID4+ICAgICAgIHNwZWNpYWxjaGFyQ29udmVydE1hcC5hZGQo IiEiLCAiRXhjbGFtYXRpb24iKTsNCj4gPj4gICAgICAgc3BlY2lhbGNoYXJDb252ZXJ0TWFwLmFk ZCgiPyIsIlF1ZXN0aW9uTWFyayIpOw0KPiA+PiAgICAgICAuLi4uLi4uLi4uLi4uLi4NCj4gPj4N Cj4gPj4gICAgICAgcHVibGljICBUb2tlblN0cmVhbSB0b2tlblN0cmVhbShTdHJpbmcgZmllbGRO YW1lLCBSZWFkZXIgcmVhZGVyKQ0KPiB7DQo+ID4+ICAgICAgICAgICBDaGFyRmlsdGVyIHNwZWNp YWxDaGFyRmlsdGVyID0gbmV3DQo+ID4+IE1hcHBpbmdDaGFyRmlsdGVyKHNwZWNpYWxjaGFyQ29u dmVydE1hcCxyZWFkZXIpOw0KPiA+Pg0KPiA+PiAgICAgICAgICAgU3RhbmRhcmRUb2tlbml6ZXIg dG9rZW5TdHJlYW0gPSBuZXcNCj4gPj4gU3RhbmRhcmRUb2tlbml6ZXIoTHVjZW5lVmVyc2lvbi5M VUNFTkVfVkVSU0lPTik7DQo+ID4+ICAgICAgICAgICB0cnkNCj4gPj4gICAgICAgICAgIHsNCj4g Pj4gICAgICAgICAgICAgICBpZih0b2tlblN0cmVhbS5pbmNyZW1lbnRUb2tlbigpPT1mYWxzZSkN Cj4gPj4gICAgICAgICAgICAgICB7DQo+ID4+ICAgICAgICAgICAgICAgICAgIHRva2VuU3RyZWFt ID0gbmV3DQo+ID4+IFN0YW5kYXJkVG9rZW5pemVyKEx1Y2VuZVZlcnNpb24uTFVDRU5FX1ZFUlNJ T04sIHNwZWNpYWxDaGFyRmlsdGVyKTsNCj4gPj4gICAgICAgICAgICAgICB9DQo+ID4+ICAgICAg ICAgICAgICAgZWxzZQ0KPiA+PiAgICAgICAgICAgICAgIHsNCj4gPj4gICAgICAgICAgICAgICAg ICAgLy9UT0RPICoqKioqKioqKioqKioqKiogc2V0IHRva2Vuc3RyZWFtIGJhY2sgYXMgaXQNCj4g d2FzDQo+ID4+IGJlZm9yZSBpbmNyZW1lbnQgdG9rZW4NCj4gPj4gICAgICAgICAgICAgICB9DQo+ ID4+ICAgICAgICAgICB9DQo+ID4+ICAgICAgICAgICBjYXRjaChJT0V4Y2VwdGlvbiBpb2UpDQo+ ID4+ICAgICAgICAgICB7DQo+ID4+DQo+ID4+ICAgICAgICAgICB9DQo+ID4+ICAgICAgICAgICBU b2tlblN0cmVhbSByZXN1bHQgPSBuZXcgTG93ZXJjYXNlRmlsdGVyKHJlc3VsdCk7DQo+ID4+ICAg ICAgICAgICByZXR1cm4gcmVzdWx0Ow0KPiA+PiAgICAgICB9DQo+ID4+DQo+ID4+IHRoYW5rcyBm b3IgYW55IGhlbHANCj4gPj4NCj4gPj4NCj4gPj4gUGF1bA0KPiA+Pg0KPiA+PiAtLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0NCj4gPj4gVG8gdW5zdWJzY3JpYmUsIGUtbWFpbDogamF2YS11c2VyLXVuc3Vic2NyaWJlQGx1 Y2VuZS5hcGFjaGUub3JnDQo+ID4+IEZvciBhZGRpdGlvbmFsIGNvbW1hbmRzLCBlLW1haWw6IGph dmEtdXNlci1oZWxwQGx1Y2VuZS5hcGFjaGUub3JnDQoNCg==