hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Payne <eric.payne1...@yahoo.com>
Subject Fw: [VOTE] Should we release 0.20.204.0rc2?
Date Mon, 22 Aug 2011 17:46:31 GMT
+1 (non-binding)
 
I downloaded the release candidate 2 onto a 12-nod cluster performed the following tests:
 
##################################################################
Create files using S-Live:
     hadoop --config $HADOOP_CONF_DIR org.apache.hadoop.fs.slive.SliveTest -appendSize
1,67108864 -append 0,uniform -baseDir /user/$USER/S-Live -blockSize 67108864,67108864 -create
100,uniform -delete 0,uniform -dirSize 16 -duration 300 -files 1024 -ls 0,uniform -maps 20
-mkdir 0,uniform -ops 10000 -packetSize 65536 -readSize 1,4294967295 -read 0,uniform -reduces
5 -rename 0,uniform -replication 1,3 -resFile $outFile -seed 12345678 -sleep 100,1000 -writeSize
1,67108864
     Output:
Basic report for operation type CreateOp
-------------
Measurement "bytes_written" = 32046140254
Measurement "milliseconds_taken" = 233234
Measurement "op_count" = 7929
Rate for measurement "bytes_written" = 131.031 MB/sec
Rate for measurement "op_count" = 33.996 operations/sec

##########################################################
 
Do random deletes, reads, mkdirs, lists, and renames using S-Live:
    hadoop --config $HADOOP_CONF_DIR org.apache.hadoop.fs.slive.SliveTest -appendSize
1,67108864 -append 0,uniform -baseDir /user/$USER/S-Live -blockSize 67108864,67108864 -create
0,uniform -delete 20,uniform -dirSize 16 -duration 300 -files 1024 -ls 20,uniform -maps 20
-mkdir 20,uniform -ops 10000 -packetSize 65536 -readSize 1,4294967295 -read 20,uniform -reduces
5 -rename 20,uniform -replication 1,3 -resFile $outFile -seed 12345678 -sleep 100,1000 -writeSize
1,67108864
     Output:
Basic report for operation type DeleteOp 
-------------
Measurement "milliseconds_taken" = 6008007
Measurement "op_count" = 27148
Rate for measurement "op_count" = 4.519 operations/sec
-------------

#########################################################################
 
     I also ran randomwriter.
 
Thanks,
-Eric Payne
 
 
------ Forwarded Message
From: Owen O'Malley <owen@hortonworks.com>
Reply-To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Date: Thu, 18 Aug 2011 00:28:20 -0700
To: Owen O'Malley <owen@hortonworks.com>
Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Subject: Re: [VOTE] Should we release 0.20.204.0rc2?


On Aug 9, 2011, at 8:55 AM, Owen O'Malley wrote:

> All,
>  Matt rolled a 0.20.204.0rc1, but I think it got lost in the previous vote
thread. Unfortunately, it had the version as 0.20.204 and didn't update the
release notes. I've updated it, run the regression tests and I think we should
release it. I've put the tarball up at:
> 
> http://people.apache.org/~omalley/hadoop-0.20.204.0-rc2

This vote is still running with no votes other than mine.

I've tested with and without security on a 60 node cluster and I'm seeing
some failures, but not that many. On a terasort with 15,000 maps and 200
reduces, I ran the following cases:

security + linux task controller : 2 failures (both mr-2651)

no security + default task controller : 6-7 failures (seems to be a race
condition in clean up)

Even in the no security case, it is only losing 0.05% of the time.

It isn't perfect, but this is the code that Yahoo is currently running. I
think we should release it.

-- Owen

------ End of Forwarded Message
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message