accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: Persistent outstanding migrations message
Date Thu, 04 Aug 2016 16:18:14 GMT
Are you currently experiencing 1 outstanding migration?  Does it go away on
it's own?  Unless servers are going down, tablets will migrate when their
split threshold is reached.  Is it possible you are constantly splitting a
table?

If all the tservers appear to be in good shape, maybe it is an issue with
the master.  What does the jstack look like for that?

On Thu, Aug 4, 2016 at 12:06 PM, Tim I <tim@timisrael.com> wrote:

> Hi Mike,
>
> Thanks for the direction.
>
> Empty result set from the scan you suggested
>
> There was a lock without an associated FATE operation.
>
> The following locks did not have an associated FATE operation
> txid: 667becf32c0fe544  locked: [R:+default]
>
>
> No recoveries stuck currently, and no long running scans.
>
> Otherwise, the system seems fine.
>
> Is it possible this is just benign?  Should we monitor for locks that
> don't have FATE operations and delete them from time to time?
>
> Thanks,
>
> Tim
>
> On Thu, Aug 4, 2016 at 11:44 AM, Michael Wall <mjwall@gmail.com> wrote:
>
>> Hi Tim,
>>
>> You can try scanning the metadata table for a future colfam.  Something
>> like
>>
>> scan -t accumulo.metadata -c fut
>>
>> If you find one, look at the tabletserver that is slated to host that
>> tablet.  There could be an issue with that server preventing assignment
>> from completing.  Get a jstack and save the logs so you can further
>> troubleshoot.  Killing that tserver will cause the assignment to go
>> elsewhere, but make sure you get as much info as you can before killing it.
>>
>> What else is going on with the system?  Do you have any recoveries that
>> are stuck?  Are there any fate transactions that have been running for a
>> while?  Any long running scans?
>>
>> HTH
>>
>> Mike
>>
>> On Thu, Aug 4, 2016 at 11:04 AM, Tim I <tim@timisrael.com> wrote:
>>
>>> Hi all,
>>>
>>> We're running accumulo 1.6.5
>>>
>>> One of the issues we're seeing on a consistent basis is this message:
>>>
>>> "Not balancing due to 1 outstanding migrations".
>>>
>>>
>>> Is there a simple way to see the number of outstanding migrations?
>>> Based on what we've read and experienced, it eventually means we have to
>>> bounce the master to get things to a better state, however the message
>>> comes back within about 1 hour.
>>>
>>> Any thoughts and suggestions would be greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>
>>
>

Mime
View raw message