lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Lundgren <scott.lundg...@carbonblack.com>
Subject Re: [solr cloud] solr hangs when indexing large number of documents from multiple threads
Date Tue, 25 Jun 2013 02:12:14 GMT
Jason,

Regarding your statement "push you over the edge"- what does that mean?
 Does it mean "uncharted territory with unknown ramifications" or something
more like specific, known symptoms?

I ask because our use is similar to Vinay's in some respects, and we want
to be able to push the capabilities of write perf - but not over the edge!
 In particular, I am interested in knowing the symptoms of failure, to help
us troubleshoot the underlying problems if and when they arise.

Thanks,

Scott

On Monday, June 24, 2013, Jason Hellman wrote:

> Vinay,
>
> You may wish to pay attention to how many transaction logs are being
> created along the way to your hard autoCommit, which should truncate the
> open handles for those files.  I might suggest setting a maxDocs value in
> parallel with your maxTime value (you can use both) to ensure the commit
> occurs at either breakpoint.  30 seconds is plenty of time for 5 parallel
> processes of 20 document submissions to push you over the edge.
>
> Jason
>
> On Jun 24, 2013, at 2:21 PM, Vinay Pothnis <pothnis@gmail.com> wrote:
>
> > I have 'softAutoCommit' at 1 second and 'hardAutoCommit' at 30 seconds.
> >
> > On Mon, Jun 24, 2013 at 1:54 PM, Jason Hellman <
> > jhellman@innoventsolutions.com> wrote:
> >
> >> Vinay,
> >>
> >> What autoCommit settings do you have for your indexing process?
> >>
> >> Jason
> >>
> >> On Jun 24, 2013, at 1:28 PM, Vinay Pothnis <pothnis@gmail.com> wrote:
> >>
> >>> Here is the ulimit -a output:
> >>>
> >>> core file size           (blocks, -c)  0  data seg size
> >> (kbytes,
> >>> -d)  unlimited  scheduling priority              (-e)  0  file size
> >>>               (blocks, -f)  unlimited  pending signals
> >>> (-i)  179963  max locked memory        (kbytes, -l)  64  max memory
> size
> >>>         (kbytes, -m)  unlimited  open files                       (-n)
> >>> 32769  pipe size             (512 bytes, -p)  8  POSIX message queues
> >>>   (bytes,
> >>> -q)  819200  real-time priority               (-r)  0  stack size
> >>> (kbytes, -s)  10240  cpu time                (seconds, -t)  unlimited
> >> max
> >>> user processes               (-u)  140000  virtual memory
> >> (kbytes,
> >>> -v)  unlimited  file locks                       (-x)  unlimited
> >>>
> >>> On Mon, Jun 24, 2013 at 12:47 PM, Yago Riveiro <yago.riveiro@gmail.com
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have the same issue too, and the deploy is quasi exact like than
> mine,
> >>>>
> >>
> http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html#a4067862
> >>>>
> >>>> With some concurrence and batches of 10 solr apparently have some
> >> deadlock
> >>>> distributing updates
> >>>>
> >>>> Can you dump the configuration of the ulimit on your servers?, some
> >> people
> >>>> had the same issues because they are reach the ulimit maximum defined
> >> for
> >>>> descriptor and process.
> >>>>
> >>>> --
> >>>> Yago Riveiro
> >>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >>>>
> >>>>
> >>>> On Monday, June 24, 2013 at 7:49 PM, Vinay Pothnis wrote:
> >>>>
> >>>>> Hello All,
> >>>>>
> >>>>> I have the following set up of solr cloud.
> >>>>>
> >>>>> * solr version 4.3.1
> >>>>> * 3 node solr cloud + replciation factor 2
> >>>>> * 3 zoo keepers
> >>>>> * load balancer in front of the 3 solr nodes
> >>>>>
> >>>>> I am seeing this strange behavior when I am indexing a large number
> of
> >>>>> documents (10 mil). When I have more than 3-5 threads sending
> documents
> >>>> (in
> >>>>> batch of 20) to solr, sometimes solr goes into a hung state. After
> this
> >>>> all
> >>>>> the update requests get timed out. What we see via AppDynamics (a
> >>>>> performance monitoring tool) is that there are a number of threads
> that
> >>>> are
> >>>>> stalled. The stack trace for one of the threads is shown below.
> >>>>>
> >>>>> The cluster has to be restarted to recover from this. When I reduce
> the
> >>>>>



-- 
Scott Lundgren
Director of Engineering
Carbon Black, Inc.
(210) 204-0483 | scott.lundgren@carbonblack.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message