hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amir Hellershtein <am...@mellanox.com>
Subject Disable Checksum
Date Mon, 28 Apr 2014 12:22:35 GMT
Hello,

I wish to disable checksum calculation & verification in Hadoop's framework,
I run the TestDfsIO application with the parameter dfs.checksum.type=NULL,
To my best understanding, this was supposed to turnoff check summing in the framework,
But I received the following error -

...
...
...
14/04/28 14:54:26.433 INFO  hdfs.DFSOutputStream (1349) 96232.106, 96232.10464359 : Vanilla
DFSOutputStream constructed successfully.
java.lang.ArrayIndexOutOfBoundsException: 0
        at org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148)
        at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137)
        at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99)
        at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240)
        at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
        at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
        at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
        at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
14/04/28 14:54:26.581 INFO  fs.FileSystem (585) 96232.249, 0.585955522 : Vanilla distributeFileSystem
was closed.

Is seems that the framework does not support disabling the checksum by the above flag,
Does anybody knows why I get this error?
Is there another way to disable checksum?

Thanks

Amir

Mime
View raw message