Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of markrmiller@gmail.com
 designates 209.85.220.181 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: overseer queue clogged
From: Mark Miller <markrmiller@gmail.com>
In-Reply-To: 
 <CAOGgQSKiKs4q=H-1Wfhf2gMX+RjVFpNAhUrzNWVPGk7dLDK7LQ@mail.gmail.com>
Date: Fri, 15 Mar 2013 20:05:20 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <0FDE1E95-0E15-4830-98D4-DFACBBC2DB46@gmail.com>
References: 
 <CAOGgQS+i2-UXdFoq+cC4UzupHdSWSwpB9ahCEUc3V0zVbORufw@mail.gmail.com>
 <9F3D9860-4C7D-4E99-BFF2-1D1DE4FB75A3@gmail.com>
 <CAOGgQS+XSXrOnZc1e_UxXTjnXjBU02N0SHREFHhmy=T4=iHhmg@mail.gmail.com>
 <CAOGgQSKiKs4q=H-1Wfhf2gMX+RjVFpNAhUrzNWVPGk7dLDK7LQ@mail.gmail.com>
To: solr-user@lucene.apache.org

Strange - we hardened that loop in 4.1 - so I'm not sure what happened =
here.

Can you do a stack dump on the overseer and see if you see an Overseer =
thread running perhaps? Or just post the results?

To recover, you should be able to just restart the Overseer node and =
have someone else take over - they should pick up processing the queue.

Any logs you might be able to share could be useful too.

- Mark

On Mar 15, 2013, at 7:51 PM, Gary Yngve <gary.yngve@gmail.com> wrote:

> Also, looking at overseer_elect, everything looks fine.  node is valid =
and
> live.
>=20
>=20
> On Fri, Mar 15, 2013 at 4:47 PM, Gary Yngve <gary.yngve@gmail.com> =
wrote:
>=20
>> Sorry, should have specified.  4.1
>>=20
>>=20
>>=20
>>=20
>> On Fri, Mar 15, 2013 at 4:33 PM, Mark Miller =
<markrmiller@gmail.com>wrote:
>>=20
>>> What Solr version? 4.0, 4.1 4.2?
>>>=20
>>> - Mark
>>>=20
>>> On Mar 15, 2013, at 7:19 PM, Gary Yngve <gary.yngve@gmail.com> =
wrote:
>>>=20
>>>> my solr cloud has been running fine for weeks, but about a week =
ago, it
>>>> stopped dequeueing from the overseer queue, and now there are =
thousands
>>> of
>>>> tasks on the queue, most which look like
>>>>=20
>>>> {
>>>> "operation":"state",
>>>> "numShards":null,
>>>> "shard":"shard3",
>>>> "roles":null,
>>>> "state":"recovering",
>>>> "core":"production_things_shard3_2",
>>>> "collection":"production_things",
>>>> "node_name":"10.31.41.59:8883_solr",
>>>> "base_url":"http://10.31.41.59:8883/solr"}
>>>>=20
>>>> i'm trying to create a new collection through collection API, and
>>>> obviously, nothing is happening...
>>>>=20
>>>> any suggestion on how to fix this?  drop the queue in zk?
>>>>=20
>>>> how could did it have gotten in this state in the first place?
>>>>=20
>>>> thanks,
>>>> gary
>>>=20
>>>=20
>>=20