Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.rutherglen@gmail.com
 designates 209.85.212.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <8C865897-9AA9-4945-ABEB-65F0F907326E@transpac.com>
References: <8C865897-9AA9-4945-ABEB-65F0F907326E@transpac.com>
Date: Fri, 4 Nov 2011 13:32:04 -0400
Message-ID: 
 <CACn_-AwNRhHwSQECqcfbJU84qotmDQLSd1ab5u1AWW15GF18vA@mail.gmail.com>
Subject: Re: overwrite=false support with SolrJ client
From: Jason Rutherglen <jason.rutherglen@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

It should be supported in SolrJ, I'm surprised it's been lopped out.
Bulk indexing is extremely common.

On Fri, Nov 4, 2011 at 1:16 PM, Ken Krugler <kkrugler_lists@transpac.com> wrote:
> Hi list,
>
> I'm working on improving the performance of the Solr scheme for Cascading.
>
> This supports generating a Solr index as the output of a Hadoop job. We use SolrJ to write the index locally (via EmbeddedSolrServer).
>
> There are mentions of using overwrite=false with the CSV request handler, as a way of improving performance.
>
> I see that https://issues.apache.org/jira/browse/SOLR-653 removed this support from SolrJ, because it was deemed too dangerous for mere mortals.
>
> My question is whether anyone knows just how much performance boost this really provides.
>
> For Hadoop-based workflows, it's straightforward to ensure that the unique key field is really unique, thus if the performance gain is significant, I might look into figuring out some way (with a trigger lock) of re-enabling this support in SolrJ.
>
> Thanks,
>
> -- Ken
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>
>