hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stephen mulcahy <stephen.mulc...@deri.org>
Subject Re: Slowdown with Hadoop Sort benchmark when using Jumbo frames?
Date Thu, 28 Jan 2010 17:25:11 GMT
Jay Booth wrote:
> Did you set io.file.buffer.size (or whatever the property is) to a large
> value?

Just re-ran the benchmark with that bumped to 65536 (as proposed in 
http://www.cloudera.com/blog/tag/configuration/). The benchmark is still 
slower with jumbo frames than without (but difference was reduced a 
little by bumping io.file.buffer.size.

> Also, even if you do set that, I'm not 100% sure that it will lead to use of
> the jumboframes, I'd have to take a look through the code to be sure.
>  Another issue is that you may lose any networking performance gains as a
> result of increased buffer allocation costs and garbage collection.

I guess there must be something going on anyways - as there is clearly a 
performance drop-off, which surprised me, as lots of apps benefit 
significantly from jumbo frames.

Thanks for the feedback,


Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie    http://webstar.deri.ie    http://sindice.com

View raw message