hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-8577) TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry failed intermittently
Date Sun, 19 May 2013 21:31:16 GMT

     [ https://issues.apache.org/jira/browse/HBASE-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-8577:
-------------------------

       Resolution: Fixed
    Fix Version/s: 0.95.1
                   0.98.0
     Hadoop Flags: Reviewed
           Status: Resolved  (was: Patch Available)

Committed to branch and trunk.  Thanks Jeffrey.

Sounds to me like there is something badly wrong w/ our log recovery.  Committed this for
now since there is nothing you can do if a FNFE but the fact that a zero-length file is going
away needs to be figured definitively.  Will dig in on HBASE-8551 and hbase-8449.
                
> TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry failed intermittently
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-8577
>                 URL: https://issues.apache.org/jira/browse/HBASE-8577
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Jeffrey Zhong
>            Assignee: Jeffrey Zhong
>             Fix For: 0.98.0, 0.95.1
>
>         Attachments: hbase-8577.patch
>
>
> Basically we tried to recover lease of a wal file while the file doesn't exist anymore.
We keep retrying recover the file(it always fails for sure because the file is gone). Then
the test case failed with time out
> Here is a recent run failure log:
> https://builds.apache.org/job/HBase-0.95/204/testReport/junit/org.apache.hadoop.hbase/TestZooKeeper/testLogSplittingAfterMasterRecoveryDueToZKExpiry/
> Related exception:
> {code}
> 2013-05-18 05:53:03,918 ERROR [IPC Server handler 2 on 36965] security.UserGroupInformation(1152):
PriviledgedActionException as:jenkins.hfs.2 cause:java.io.FileNotFoundException: File not
found /user/jenkins/hbase/.logs/hemera.apache.org,42628,1368856131392-splitting/hemera.apache.org,42628,1368856131392
> 2013-05-18 05:53:03,918 WARN  [SplitLogWorker-hemera.apache.org,47651,1368856143179]
util.FSHDFSUtils(80): Got IOException on attempt 241 to recover lease for file hdfs://localhost:36965/user/jenkins/hbase/.logs/hemera.apache.org,42628,1368856131392-splitting/hemera.apache.org,42628,1368856131392,
retrying.
> java.io.FileNotFoundException: java.io.FileNotFoundException: File not found /user/jenkins/hbase/.logs/hemera.apache.org,42628,1368856131392-splitting/hemera.apache.org,42628,1368856131392
> 	at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> 	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> 	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> 	at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:885)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:177)
> 	at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:71)
> 	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:821)
> 	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:504)
> 	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:455)
> 	at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:132)
> 	at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:337)
> 	at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:225)
> 	at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:192)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message