Return-Path: 
 <solr-user-return-99379-apmail-lucene-solr-user-archive=lucene.apache.org@lucene.apache.org>
X-Original-To: apmail-lucene-solr-user-archive@minotaur.apache.org
Delivered-To: apmail-lucene-solr-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 859F411A05
	for <apmail-lucene-solr-user-archive@minotaur.apache.org>;
 Sat, 12 Apr 2014 05:23:56 +0000 (UTC)
Received: (qmail 54296 invoked by uid 500); 12 Apr 2014 05:23:51 -0000
Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org
Received: (qmail 53725 invoked by uid 500); 12 Apr 2014 05:23:46 -0000
Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:solr-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:solr-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:solr-user@lucene.apache.org>
List-Id: <solr-user.lucene.apache.org>
Reply-To: solr-user@lucene.apache.org
Delivered-To: mailing list solr-user@lucene.apache.org
Received: (qmail 53717 invoked by uid 99); 12 Apr 2014 05:23:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Apr 2014 05:23:43 +0000
X-ASF-Spam-Status: No, hits=1.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of amantandon.10@gmail.com
 designates 74.125.82.44 as permitted sender)
Received: from [74.125.82.44] (HELO mail-wg0-f44.google.com) (74.125.82.44)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Apr 2014 05:23:38 +0000
Received: by mail-wg0-f44.google.com with SMTP id m15so6146872wgh.3
        for <solr-user@lucene.apache.org>;
 Fri, 11 Apr 2014 22:23:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        bh=wJrPrfoRwlmwrnQO0vatbqicio/Pmko8gp7nsFf//lw=;
        b=p/1IUdTCp+sW969m2Kg6Yk9sYuJCVNgJ3yqi9KP381p3P8JTqreWhM4ajJY55wgG0r
         HrqOehVs4BfzBRJp8BgweS8C4oBWPQU9TEpGRUdtwI8CEz3MYRyEhvkb4AX0gJCDfU7J
         iopIjODK4SgzxEbvUYp/D1VMBS54Kq6X3Ih1y1SFjoDjKvhrkqXpsqSUGAmYnm9f33ay
         FORwfSxhjEaowh2Y56sdofi5gaaVUsXAye/IgRbgaZsh+yEgaoxa+HJ4sEBxQFIjBzpo
         XRrttzimG2+3RTePBF8JQD1fK4ezArdpCgNkFrG2wgEY0nKmxQLjnW18ZjsrlzLrxXMb
         OC6A==
X-Received: by 10.180.13.8 with SMTP id d8mr1167095wic.13.1397280196737; Fri,
 11 Apr 2014 22:23:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.217.66.129 with HTTP; Fri, 11 Apr 2014 22:22:56 -0700 (PDT)
In-Reply-To: 
 <CAARqL6fZN770554zff8obo8ODaeHrmvJX+-W86iMCeWiUfUZ1Q@mail.gmail.com>
References: 
 <CAARqL6cR6uyX17aX1HX-sUdgT7TXozD=E5dzwMkwQQjx4dtJgA@mail.gmail.com>
 <53486C9A.2030903@elyograg.org>
 <CAARqL6fZN770554zff8obo8ODaeHrmvJX+-W86iMCeWiUfUZ1Q@mail.gmail.com>
From: Aman Tandon <amantandon.10@gmail.com>
Date: Sat, 12 Apr 2014 10:52:56 +0530
Message-ID: 
 <CAJ9V1mZBUNsRCSNexD+_aH7GMBdJBtzUfSPhd1n1KdEf-W3sKg@mail.gmail.com>
Subject: Re: deleting large amount data from solr cloud
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001a11c2437cc73f8404f6d1a581
X-Virus-Checked: Checked by ClamAV on apache.org

--001a11c2437cc73f8404f6d1a581
Content-Type: text/plain; charset=ISO-8859-1

Vinay please share your experience after trying this solution.


On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <pothnis@gmail.com> wrote:

> The query is something like this:
>
>
> *curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1 OR
> val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> 1385164800000]</query></delete>'
> 'http://host:port/solr/coll-name1/update?commit=true'*
>
> Trying to restrict the number of documents deleted via the date parameter.
>
> Had not tried the "distrib=false" option. I could give that a try. Thanks
> for the link! I will check on the cache sizes and autowarm values. Will try
> and disable the caches when I am deleting and give that a try.
>
> Thanks Erick and Shawn for your inputs!
>
> -Vinay
>
>
>
> On 11 April 2014 15:28, Shawn Heisey <solr@elyograg.org> wrote:
>
> > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> >
> >> When we tried to delete the data through a query - say 1 day/month's
> worth
> >> of data. But after deleting just 1 month's worth of data, the master
> node
> >> is going out of memory - heap space.
> >>
> >> Wondering is there any way to incrementally delete the data without
> >> affecting the cluster adversely.
> >>
> >
> > I'm curious about the actual query being used here.  Can you share it, or
> > a redacted version of it?  Perhaps there might be a clue there?
> >
> > Is this a fully distributed delete request?  One thing you might try,
> > assuming Solr even supports it, is sending the same delete request
> directly
> > to each shard core with distrib=false.
> >
> > Here's a very incomplete list about how you can reduce Solr heap
> > requirements:
> >
> > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > Reducing_heap_requirements
> >
> > Thanks,
> > Shawn
> >
> >
>



-- 
With Regards
Aman Tandon

--001a11c2437cc73f8404f6d1a581--