hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12866) TestHFilePerformance is broken
Date Tue, 20 Jan 2015 05:59:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283447#comment-14283447
] 

Andrew Purtell commented on HBASE-12866:
----------------------------------------

TestHFilePerformance is broken generally. 

The tool also behaves really differently in 0.98 than in later branches. I think to start
we should sync the master version of this file back to 0.98.

This is master, see the exception at the bottom of the below ("Added a key not lexically larger
than previous."):
{noformat}
$ ./bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
2015-01-19 21:50:32,664 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
****************************** Sequence File *****************************
File Type: SequenceFile
Writing SequenceFile with codecName: none cipherName: none
2015-01-19 21:50:33 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/SequenceFile.Performance
2015-01-19 21:50:34 Stopped timing.
2015-01-19 21:50:34  Data written: 
2015-01-19 21:50:34    rate  = 72MB/s
2015-01-19 21:50:34    total = 52200000B
2015-01-19 21:50:34  File written: 
2015-01-19 21:50:34    rate  = 73MB/s
2015-01-19 21:50:34    total = 52560076B

+++++++

Reading file of type: SequenceFile
Input file size: 52560076
2015-01-19 21:50:34 Started timing.
2015-01-19 21:50:34  Compression codec class: none
2015-01-19 21:50:34 Stopped timing.
2015-01-19 21:50:34  Finished in 406ms
2015-01-19 21:50:34  Data read: 
2015-01-19 21:50:34    rate  = 122MB/s
2015-01-19 21:50:34    total = 52200000B
2015-01-19 21:50:34  File read: 
2015-01-19 21:50:34    rate  = 123MB/s
2015-01-19 21:50:34    total = 52560076B

----------------------

File Type: SequenceFile
Writing SequenceFile with codecName: gz cipherName: none
2015-01-19 21:50:34 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/SequenceFile.Performance
Skipping sequencefile gz: SequenceFile doesn't work with GzipCodec without native-hadoop code!
****************************** HFile *****************************
File Type: HFile
Writing HFile with codecName: none cipherName: none
2015-01-19 21:50:34 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/HFile.Performance
HFile write method: 
2015-01-19 21:50:34,634 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
2015-01-19 21:50:34,676 ERROR [main] util.AbstractHBaseTool: Error running command-line tool
java.io.IOException: Added a key not lexically larger than previous. Current cell = 8U\x86\xB4\x1F\xEF\xF2\x05\xB4\xE0Z\x00\x08#\xF7\x8B_\x8F\x5C\x02C\x9C\xE8\xF6zx\x1D\x90\xCB\xE6\xBF\x1A\xE7\xF2\xBC@\xA4\x97\x09\xA0l\x0E1I\x9B\xF0)i\xCAB\xD2\x03\xE5f\xBC\xC6\x96\xDE\x08\xFA\x01\x02\xA0\xFD.#0\xB0\x96J\xBB|D0
\xDE\x1C\xAD\x09\xBF\xD68\x1F\xFB\x94\xDA\xAF\xBB\x90\xC4\xED\x91\xA0a:\xD1\xDCKG\x03//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0,
lastCell = `\xB4 \xBB8Q\xD9\xD4z\xCB\x93=\xBEp9\x9B\xF6\xC9-\xA3:\xF0\x1DO\xB7p\xE9\x8C\x03%\xF4\x1D>\xBA\xF8\x98m\xA7\x12\xC8+\xCDMUK\xF0\xB5@#\xC2\x9BbM\xE9\xEF\x9C/\x93\x1E\xFCX\x0F\x9A\xFB\x08\x1B\x12\xE1\x07\xB1\xE8\x05\xF2\xB4\xF5\xF0\xF1\xD0\x0C-\x0FbcFp\x92\x1CPXg\xFF
\xF6\xA83^\x98\xAF\x87%//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0
	at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:201)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:261)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeWrite(TestHFilePerformance.java:200)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:380)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
{noformat}

Same result on branch-1:
{noformat}
****************************** HFile *****************************
File Type: HFile
Writing HFile with codecName: none cipherName: none
2015-01-19 21:53:46 Started timing.
/home/apurtell/src/hbase/target/test-data/37e20539-2afb-4f8a-8c95-15ee6b9ed7cd/TestHFilePerformance/HFile.Performance
HFile write method: 
2015-01-19 21:53:46,953 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
2015-01-19 21:53:46,997 ERROR [main] util.AbstractHBaseTool: Error running command-line tool
java.io.IOException: Added a key not lexically larger than previous. Current cell = 8U\x86\xB4\x1F\xEF\xF2\x05\xB4\xE0Z\x00\x08#\xF7\x8B_\x8F\x5C\x02C\x9C\xE8\xF6zx\x1D\x90\xCB\xE6\xBF\x1A\xE7\xF2\xBC@\xA4\x97\x09\xA0l\x0E1I\x9B\xF0)i\xCAB\xD2\x03\xE5f\xBC\xC6\x96\xDE\x08\xFA\x01\x02\xA0\xFD.#0\xB0\x96J\xBB|D0
\xDE\x1C\xAD\x09\xBF\xD68\x1F\xFB\x94\xDA\xAF\xBB\x90\xC4\xED\x91\xA0a:\xD1\xDCKG\x03//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0,
lastCell = `\xB4 \xBB8Q\xD9\xD4z\xCB\x93=\xBEp9\x9B\xF6\xC9-\xA3:\xF0\x1DO\xB7p\xE9\x8C\x03%\xF4\x1D>\xBA\xF8\x98m\xA7\x12\xC8+\xCDMUK\xF0\xB5@#\xC2\x9BbM\xE9\xEF\x9C/\x93\x1E\xFCX\x0F\x9A\xFB\x08\x1B\x12\xE1\x07\xB1\xE8\x05\xF2\xB4\xF5\xF0\xF1\xD0\x0C-\x0FbcFp\x92\x1CPXg\xFF
\xF6\xA83^\x98\xAF\x87%//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0
	at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:201)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:261)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeWrite(TestHFilePerformance.java:200)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:380)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
{noformat}

Finally, 0.98 produces results as reported in the description of this issue. 
{noformat}
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from
file /home/apurtell/src/hbase/target/test-data/7b0f81cc-fbf8-4b67-b1f5-de77c2ee3074/TestHFilePerformance/HFile.Performance
{noformat}

My guess is the configuration for the tool does not allow for HFileV3, but I haven't dug in
yet.

> TestHFilePerformance is broken
> ------------------------------
>
>                 Key: HBASE-12866
>                 URL: https://issues.apache.org/jira/browse/HBASE-12866
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.98.8
>         Environment: Command
> bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
> Failure observed for test with the following options: Read HFile with codecName: none
cipherName: aes
>            Reporter: Vikas Vishwakarma
>            Assignee: Andrew Purtell
>            Priority: Minor
>             Fix For: 1.0.0, 2.0.0, 1.1.0, 0.98.11
>
>
> Command
> bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
> File Type: HFile
> Writing HFile with codecName: none cipherName: aes
> 2015-01-15 16:54:51 Started timing.
> /home/vvishwakarma/vikas/projects/hbase-src-0.98.8/hbase-0.98.8/target/test-data/03d50949-0185-4fac-b072-957d2da6ae0e/TestHFilePerformanceHFile.Performance
> HFile write method: 
> 2015-01-15 16:54:52 Stopped timing.
> 2015-01-15 16:54:52  Data written: 
> 2015-01-15 16:54:52    rate  = 66MB/s
> 2015-01-15 16:54:52    total = 52200000B
> 2015-01-15 16:54:52  File written: 
> 2015-01-15 16:54:52    rate  = 66MB/s
> 2015-01-15 16:54:52    total = 52303530B
> +++++++
> Reading file of type: HFile
> Input file size: 52303530
> 2015-01-15 16:54:52 Started timing.
> 2015-01-15 16:54:52,680 ERROR [main] util.AbstractHBaseTool: Error running command-line
tool
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer
from file /home/vvishwakarma/vikas/projects/hbase-src-0.98.8/hbase-0.98.8/target/test-data/03d50949-0185-4fac-b072-957d2da6ae0e/TestHFilePerformance/HFile.Performance
> 	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:463)
> 	at org.apache.hadoop.hbase.io.hfile.HFile.createReaderFromStream(HFile.java:517)
> 	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeReading(TestHFilePerformance.java:272)
> 	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:390)
> 	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
> 	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
> Caused by: java.io.IOException: Using no compression but onDiskSizeWithoutHeader=134,
uncompressedSizeWithoutHeader=113, numChecksumbytes=4
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock.assumeUncompressed(HFileBlock.java:561)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1589)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1408)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1248)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1256)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:146)
> 	at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:451)
> 	... 7 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message