Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of ian.lea@gmail.com designates
 209.85.216.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=cCn40SDm/3wIESJvG7A5UkMa+X+hGShS3TqRrccP9FiycHq8OPCBdlzxtd2woTRAAh
         XEzQy/R6oFqFCUX5yqkhZ6N1ksBDjxgFikoGOk4b0dNic7EAALXXQGVHU0TkEVD11Wmh
         Uqs2P1HPrl7pf3yT+oceDLxXooChvxWEz1JnM=
MIME-Version: 1.0
In-Reply-To: <COL118-W13364397C396B01377F7FAF38F0@phx.gbl>
References: <COL118-W13364397C396B01377F7FAF38F0@phx.gbl>
From: Ian Lea <ian.lea@gmail.com>
Date: Mon, 6 Sep 2010 09:38:05 +0100
Message-ID: <AANLkTinkU+R_oTqwHr_GWAQQVDkVPom5ug4LXgCcQzL=@mail.gmail.com>
Subject: Re: Line filtering
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You'd be better off reading and selecting the syslog lines outside of
lucene.  Then pass the lines you are interested in to lucene using
whatever analyzer you want.


--
Ian.


On Sun, Sep 5, 2010 at 10:09 PM, Lev Bronshtein
<lev_bronshtein@hotmail.com> wrote:
>
> Hello group,
>
> I am new to Lucene and ran into a bit of trouble while writing an app.=A0=
 I would like to selectively index lines from a syslog on a unix system, to=
 this end I first wrote tokenizer that returns an entire line as a token ex=
tending CharTokenizer
>
> =A0 protected boolean isTokenChar(char c) {
> =A0=A0=A0 return !((c =3D=3D '\n') || (c =3D=3D '\r'));
> =A0 }
>
> Perhaps that is my first mistake and I should have done things differentl=
y?
>
> I then pass this to a filter that only selects the lines with text I am i=
nterested in
>
> =A0public final boolean incrementToken() throws IOException
> =A0{
> =A0 while (input.incrementToken())
> =A0 {
> =A0=A0 Matcher lineMatcher =3D linePattern.matcher(termAtt.term());
> =A0=A0 if (lineMatcher.find()) //(we like the payload)
> =A0=A0=A0=A0 return true;
> =A0 }
> =A0 //reached EOS -- return false
> =A0 return false;
> =A0}
>
> However the issue is that, now that I have the line I want to break up th=
e individual line into tokens along white space, but the WhitespaceTokenize=
r does not take a TokenStream as a constructor parameter.=A0 Can anyone off=
er=A0 suggestion for a workaround?
>
> Regards,
>
> Lev Bronshtein
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org