hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2434) MapFile.get on HDFS in TRUNK is WAY!!! slower than 0.15.x
Date Sat, 15 Dec 2007 02:21:43 GMT
MapFile.get on HDFS in TRUNK is WAY!!! slower than 0.15.x
---------------------------------------------------------

                 Key: HADOOP-2434
                 URL: https://issues.apache.org/jira/browse/HADOOP-2434
             Project: Hadoop
          Issue Type: Bug
          Components: io
            Reporter: stack


Stall happens down in SequenceFile in the first call to getPos inside readRecordLength.  I
tried the johano patch from HADOOP-2172 that restores the positional cache but that didn't
seem to be the issue here.

Here is data to support my assertion.

I wrote a little program to make a MapFile of 1M records.  I then did 1M random reads from
same file.  Below are timings from a 0.15.0 and TRUNK as of this afternoon run.

0.15.x branch:

{code}
[stack@aa0-000-12 branch-0.15]$ ./bin/hadoop org.apache.hadoop.io.TestMapFile
.07/12/15 01:29:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
07/12/15 01:29:02 INFO io.TestMapFile: Writing 1000000 rows to testMapFileRandomAccess
07/12/15 01:32:04 INFO io.TestMapFile: Writing 1000000 records took 182009ms
07/12/15 01:32:04 INFO io.TestMapFile: Reading 1000000 random rows
07/12/15 01:48:02 INFO io.TestMapFile: Reading 1000000 random records took 958243ms
Time: 1,140.652
OK (1 test)
{code}

For the below test using TRUNK r604352, I amended the test so it output a log message every
100k reads:

{code}
[stack@aa0-000-12 hadoop-trunk]$ ./bin/hadoop org.apache.hadoop.io.TestMapFile
.07/12/15 01:56:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
07/12/15 01:56:34 INFO io.TestMapFile: Writing 1000000 rows to testMapFileRandomAccess
07/12/15 01:59:38 INFO io.TestMapFile: Writing 1000000 records took 183986ms
07/12/15 01:59:38 INFO io.TestMapFile: Reading 1000000 random rows
.........
{code}

After 20 minutes it still hadn't printed out the 'read 100k messages' (I had to leave -- will
fill in final figures later)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message