accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ACCUMULO-954) ZooLock watcher can stop watching
Date Mon, 04 Mar 2013 20:15:12 GMT

     [ https://issues.apache.org/jira/browse/ACCUMULO-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Keith Turner updated ACCUMULO-954:
----------------------------------

    Priority: Critical  (was: Minor)
    
> ZooLock watcher can stop watching
> ---------------------------------
>
>                 Key: ACCUMULO-954
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-954
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.4.2
>            Reporter: Adam Fuchs
>            Assignee: Keith Turner
>            Priority: Critical
>             Fix For: 1.5.0, 1.4.3
>
>
> Basically, this will result in tablet servers failing to recognize when they lose their
locks. I think the worst that can happen with this is a tablet server can fail to die after
it loses its lock, which could bog down clients and create a bunch of noise in the cluster.
I believe there could also be useless files generated that wouldn't get garbage collected.
!METADATA table write protections and logger write protections should prevent any permanent
damage or data loss. We have seen this result in warnings and errors that look like multiple
hosting of tablets.
> {code}
> 2013-01-09 19:59:27,742 [tabletserver.TabletServer] INFO : port = 9997
> 2013-01-09 19:59:27,926 [zookeeper.ZooLock] DEBUG: event /accumulo/655f93d8-20fc-451f-a457-458b5717a11e/tservers/172.16.2.25:9997
NodeDeleted SyncConnected
> 2013-01-09 19:59:27,931 [tabletserver.TabletServer] INFO : Waiting for tablet server
lock
> 2013-01-09 19:59:32,943 [tabletserver.TabletServer] DEBUG: Obtained tablet server lock
/accumulo/655f93d8-20fc-451f-a457-458b5717a11e/tservers/172.16.2.25:9997/zlock-0000000000
> 2013-01-09 19:59:36,703 [tabletserver.TabletServer] DEBUG: Got loadTablet message from
user: !SYSTEM
> {code}
> Here's what happened:
> 1. Tablet server fails to get lock, triggering the watcher on the parent node.
> 2. Watcher doesn't get reset, and doesn't take any action.
> 3. Loop in TabletServer:~2659 retries, but uses the same ZooLock object.
> 4. TabletServer loses its lock, but receives a connection loss message before the NodeDeleted
message.
> 5. TabletServer continues to try to do work instead of killing itself.
> We could probably patch this for 1.4 by creating the ZooLock within the announceExistence
loop, instead of reusing the one. Eventually, we ought to have an else branch in both of the
Watchers that either reset the watch (resilient against zookeeper connection hiccups) or just
kill the server to be safe.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message