hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "churro morales (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-13724) ReplicationSource dies under certain conditions reading a sequence file
Date Wed, 20 May 2015 17:38:00 GMT
churro morales created HBASE-13724:
--------------------------------------

             Summary: ReplicationSource dies under certain conditions reading a sequence file
                 Key: HBASE-13724
                 URL: https://issues.apache.org/jira/browse/HBASE-13724
             Project: HBase
          Issue Type: Bug
            Reporter: churro morales


A little background, 

We run our server in -ea mode and have seen quite a few replication sources silently die over
the past few months.

Note: the stacktrace I posted below comes from a regionserver running 0.94 but quickly looking
at this issue, I believe this will happen in 98 too.  

Should we harden replication source to deal with these types of assertion errors by catching
throwables, should we be dealing with this at the sequence file reader level?  Still looking
into the root cause of this issue but when manually shutdown our regionservers the regionserver
that recovered its queue replicated that log just fine.  So in our case a simple retry would've
worked just fine.  

{code}
2015-05-08 11:04:23,348 ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
Unexpected exception in ReplicationSource, currentPath=hdfs://hm6.xxx.flurry.com:9000/hbase/.logs/xxxxx.yy.flurry.com,60020,1426792702998/xxxxx.atl.flurry.com%2C60020%2C1426792702998.1431107922449
java.lang.AssertionError
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:121)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1489)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:734)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:69)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:583)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:373)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message