Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of goodell@gmail.com designates
 64.233.170.191 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth;
        b=TABtQ4DlxBu4/vI/ThP/zhzvsjsAA6ZJxXM+m2MDJJcwHZkQvsT+SJav17Xn1GcasbCm99HO2/7NKf2x0JxE4+Lcamqasnaae1AG20sKYuWRLhTSN21d0CUpuSQy1uOXsalEEfMDPYn7Lf8Miwvw/rpV0O8s+F/kKVuVs8JS72s=
Message-ID: <55b2c6b90712120927v5d7dbb8bxc67e24c9319eba64@mail.gmail.com>
Date: Wed, 12 Dec 2007 09:27:26 -0800
From: "Andy Goodell" <agoodell@discoverymining.com>
Sender: goodell@gmail.com
To: java-user@lucene.apache.org
Subject: Re: Indexing Wikipedia dumps
In-Reply-To: <547045.76613.qm@web50306.mail.re2.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <547045.76613.qm@web50306.mail.re2.yahoo.com>

My firm uses a parser based on javax.xml.stream.XMLStreamReader to
break (english and nonenglish) wikipedia xml dumps into lucene-style
"documents and fields."  We use wikipedia to test our
language-specific code, so we've probably indexed 20 wikipedia dumps.

- andy g

On Dec 11, 2007 9:35 PM, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> Hi,
>
> I need to index a Wikipedia dump.  I know there is code in contrib/benchmark for indexing *English* Wikipedia for benchmarking purposes.  However, I'd like to index a non-English dump, and I actually don't need it for benchmarking, I just want to end up with a Lucene index.
>
> Any suggestions where I should start?  That is, can anything in contrib/benchmark already do this, or is there anything there that I should use as a starting point?  As opposed to writing my own Wikipedia XML dump parser+indexer.
>
> Thanks,
> Otis
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org