Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of yseeley@gmail.com designates
 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:reply-to:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=uCVscqtjhexFcUAchkB/ydv5aOISoU+A5XtVCr5uAtwxd5G01fZx+k/VidV0XvvejH
         r5PHqVAzJqc7NW00bBFHKJgHvPqreTiFAuITnLR5Dc/TBN9+bz+2hOxAwW0aR8RWqWoh
         J+aaMwHZe2gQgLLQovW612awWJVortXXbMuV0=
MIME-Version: 1.0
Sender: yseeley@gmail.com
Reply-To: yonik@lucidimagination.com
In-Reply-To: <BANLkTi=Y7A62XUWdzwuFDFqi5gcVR_SERQ@mail.gmail.com>
References: <BANLkTindV8DLyr=r1W5br-bSo0-nc2xO0Q@mail.gmail.com>
	<BANLkTimdWSNn7S-HAanMrA+eTu5uPuzm9w@mail.gmail.com>
	<BANLkTi=Y7A62XUWdzwuFDFqi5gcVR_SERQ@mail.gmail.com>
Date: Thu, 21 Apr 2011 19:47:51 -0400
Message-ID: <BANLkTimoL=6O4HJt0OXM8NMCMvM_okT75Q@mail.gmail.com>
Subject: Re: Stand-alone Index updating using EmbeddedSolrServer
From: Yonik Seeley <yonik@lucidimagination.com>
To: dev@lucene.apache.org, kiko@alum.mit.edu
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Apr 21, 2011 at 7:27 PM, Kiko Aumond <kiko@alum.mit.edu> wrote:
> Yes, I've seen that page, but I went a bit beyond the material there, as =
the
> code I wrote is able to set parameters such as separators, encapsulators =
and
> the index columns,=A0 whether to split parameters, auto-commit as well as=
 the
> ability to do incremental or full index reloads.

Is this a CSV loader?
If so, did you know the CSV loader (and other data loaders) have the
option to bypass HTTP also and stream directly from a local file (or
other URL)?

> Also, from what I've seen in DirectSolrConnection (version 1.4.1), you ha=
ve
> to supply the document body as a String.=A0 We want to avoid havindgto lo=
ad
> the entire document into memory, which is why we load the files into
> ContentStream objects and pass them to the embedded Solr server (I am
> assuming=A0 ContentStream actually streams the file as its name suggests
> instead of trying to load it into memory).=A0 The utility I wrote gets a =
path,
> a Regex expression for all the files to be loaded, as well as the paramet=
ers
> mentioned above and it does either a full or incremental upload of multip=
le
> files with a single command.
>
> We run a very high load application with SOLR in the back end that requir=
es
> that we use the Embedded solr server to eliminate the network round-trip.
> Even a small incremental gain in performance is important for us.

Eliminating the network round-trip is certainly important for good
bulk indexing performance.  Luckily you don't have to
embed to do that.  You can use multiple threads (say 16 for a 4 core
server) that essentially covers up
any round-trip latency (use persistent connections though!  or use
SolrJ which does by default),
or you can use the StreamingUpdateSolrServer that eliminates
round-trip network delays
by streaming documents over multiple already open connections.

-Yonik
http://www.lucenerevolution.org -- Lucene/Solr User Conference, May
25-26, San Francisco

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org