accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim I <...@timisrael.com>
Subject Re: Persistent outstanding migrations message
Date Thu, 04 Aug 2016 16:06:44 GMT
Hi Mike,

Thanks for the direction.

Empty result set from the scan you suggested

There was a lock without an associated FATE operation.

The following locks did not have an associated FATE operation
txid: 667becf32c0fe544  locked: [R:+default]


No recoveries stuck currently, and no long running scans.

Otherwise, the system seems fine.

Is it possible this is just benign?  Should we monitor for locks that don't
have FATE operations and delete them from time to time?

Thanks,

Tim

On Thu, Aug 4, 2016 at 11:44 AM, Michael Wall <mjwall@gmail.com> wrote:

> Hi Tim,
>
> You can try scanning the metadata table for a future colfam.  Something
> like
>
> scan -t accumulo.metadata -c fut
>
> If you find one, look at the tabletserver that is slated to host that
> tablet.  There could be an issue with that server preventing assignment
> from completing.  Get a jstack and save the logs so you can further
> troubleshoot.  Killing that tserver will cause the assignment to go
> elsewhere, but make sure you get as much info as you can before killing it.
>
> What else is going on with the system?  Do you have any recoveries that
> are stuck?  Are there any fate transactions that have been running for a
> while?  Any long running scans?
>
> HTH
>
> Mike
>
> On Thu, Aug 4, 2016 at 11:04 AM, Tim I <tim@timisrael.com> wrote:
>
>> Hi all,
>>
>> We're running accumulo 1.6.5
>>
>> One of the issues we're seeing on a consistent basis is this message:
>>
>> "Not balancing due to 1 outstanding migrations".
>>
>>
>> Is there a simple way to see the number of outstanding migrations?  Based
>> on what we've read and experienced, it eventually means we have to bounce
>> the master to get things to a better state, however the message comes back
>> within about 1 hour.
>>
>> Any thoughts and suggestions would be greatly appreciated.
>>
>> Thanks,
>>
>> Tim
>>
>
>

Mime
View raw message