accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-716) Corrupt WAL file
Date Sun, 05 Aug 2012 22:00:02 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428916#comment-13428916
] 

Josh Elser commented on ACCUMULO-716:
-------------------------------------

Looking at my tserver's debug log, I can see that processing one of the WAL entries failed
with the below exception:

{noformat}
05 17:09:05,604 [tabletserver.TabletServer] DEBUG: MultiScanSess null 0 entries in 0.00 secs
(lookup_time:0.00 secs tablets:1 ranges:1) 
05 17:09:05,706 [tabletserver.TabletServer] DEBUG: MultiScanSess null 0 entries in 0.00 secs
(lookup_time:0.00 secs tablets:1 ranges:1) 
05 17:09:05,707 [tabletserver.TabletServer] DEBUG: MultiScanSess null 0 entries in 0.00 secs
(lookup_time:0.00 secs tablets:1 ranges:1) 
05 17:09:05,728 [fs.FSInputChecker] INFO : Found checksum error: b[0, 512]=0000012c99d6b958000000000e080010011a08313936363039323400000001000000000764657461636f6c0000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a083139363630393234000000010000000006676e69766f6d0000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a083139363630393234000000010000000004657375660000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a08313936363039323400000001000000000565676e61720000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a08313936363039323400000001000000000665726f6665620000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a083139363630393234000000010000000004656e6f740000003f000000045445585400000009313900656e77696b690000000a616c6c7c656e77696b69010000012c99d6b958000000000e080010011a083139363630he.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
        at org.apache.hadoop.hdfs.DFSClient$RemoteBlockReader.read(DFSClient.java:1385)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2121)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2173)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.data.Mutation.readFields(Mutation.java:443)
        at org.apache.accumulo.server.logger.LogFileValue.readFields(LogFileValue.java:42)
        at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.sort(LogSorter.java:122)
        at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.process(LogSorter.java:87)
        at org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(DistributedWorkQueue.java:101)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
05 17:09:05,729 [hdfs.DFSClient] WARN : Found Checksum error for blk_-7588599586768272616_79756
from 127.0.0.1:50010 at 1170472960
05 17:09:05,731 [hdfs.DFSClient] INFO : Could not obtain block blk_-7588599586768272616_79756
from any node: java.io.IOException: No live nodes contain current block. Will get new block
locations from namenode and retry...] DEBUG: MultiScanSess null 0 entries in 0.00 secs (lookup_time:0.00
secs tablets:1 ranges:1) 
05 17:09:05,810 [tabletserver.TabletServer] DEBUG: MultiScanSess null 0 entries in 0.00 secs
(lookup_time:0.00 secs tablets:1 ranges:1) 
05 17:09:05,913 [tabletserver.TabletServer] DEBUG: MultiScanSess null 0 entries in 0.00 secs
(lookup_time:0.00 secs tablets:1 ranges:1) 
...
...
05 17:09:08,805 [log.LogSorter] ERROR: org.apache.hadoop.fs.ChecksumException: Checksum error:
/blk_-7588599586768272616:of:/accumulo-1.5/wal/127.0.0.1+9997/4e9df088-8c15-4499-bd9c-4a664cfe1a66
at 1170472960umException: Checksum error: /blk_-7588599586768272616:of:/accumulo-1.5/wal/127.0.0.1+9997/4e9df088-8c15-4499-bd9c-4a664cfe1a66
at 1170472960
        at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
        at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
        at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
        at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
        at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
        at org.apache.hadoop.hdfs.DFSClient$RemoteBlockReader.read(DFSClient.java:1385)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2121)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2173)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.accumulo.core.data.Mutation.readFields(Mutation.java:443)
        at org.apache.accumulo.server.logger.LogFileValue.readFields(LogFileValue.java:42)
        at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.sort(LogSorter.java:122)
        at org.apache.accumulo.server.tabletserver.log.LogSorter$LogProcessor.process(LogSorter.java:87)
        at org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(DistributedWorkQueue.java:101)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)
{noformat}

The offending entry in HDFS is:

{noformat}
$ hadoop fs -ls /accumulo-1.5/wal/127.0.0.1+9997            (svn)-[trunk:1369523]
12/08/05 17:42:28 INFO security.UserGroupInformation: JAAS Configuration already set up for
Hadoop, not re-installing.
Found 6 items
...
-rw-r--r--   2 elserj supergroup          0 2012-08-05 16:42 /accumulo-1.5/wal/127.0.0.1+9997/4e9df088-8c15-4499-bd9c-4a664cfe1a66
...
{noformat}

The fun part is if I `hadoop fs -text` that file, there's definitely content in it. Making
this slightly more confusing too, I had moved some files from one dfs.data.dir to another
dfs.data.dir to reclaim some space. But in the spirit of full disclosure, I did something
along the lines of the following (which, to my knowledge, shouldn't have messed anything up):

{noformat}
$ mv /dfs1/blocks/subdir44/* /dfs2/blocks/subdir44/
{noformat}

I also see that the block whose checksum failed in the stack trace (from above), is still
listed in the partition's drive who filled up:

{noformat}
$ find /dfs* -name "blk_-7588599586768272616*"                                           
 -
/dfs1/blocksBeingWritten/blk_-7588599586768272616_79756.meta
/dfs1/blocksBeingWritten/blk_-7588599586768272616
{noformat}


Not sure if this is something inside Accumulo itself, or if this is an HDFS bug. Given experiences
of ensuring files in HDFS being fully written (closing the file, re-opening, seeking to the
end), I thought there may be something that Accumulo could do to prevent this from happening.
                
> Corrupt WAL file
> ----------------
>
>                 Key: ACCUMULO-716
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-716
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.5.0
>         Environment: java version "1.6.0_33", hadoop-0.20.2-cdh3u3
>            Reporter: Josh Elser
>            Assignee: Eric Newton
>
> Ran wikisearch-ingest. Ended up filling up a drive used by HDFS and things failed not-so-gracefully.
Upon restart, log recovery started, appeared to finish (failed HDFS checksum on one WAL entry),
and left Accumulo in a state where no tablets were assigned.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message