hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stephen mulcahy <stephen.mulc...@deri.org>
Subject Re: Slowdown with Hadoop Sort benchmark when using Jumbo frames?
Date Thu, 28 Jan 2010 17:25:11 GMT
Jay Booth wrote:
> Did you set io.file.buffer.size (or whatever the property is) to a large
> value?

Just re-ran the benchmark with that bumped to 65536 (as proposed in 
http://www.cloudera.com/blog/tag/configuration/). The benchmark is still 
slower with jumbo frames than without (but difference was reduced a 
little by bumping io.file.buffer.size.

> Also, even if you do set that, I'm not 100% sure that it will lead to use of
> the jumboframes, I'd have to take a look through the code to be sure.
>  Another issue is that you may lose any networking performance gains as a
> result of increased buffer allocation costs and garbage collection.

I guess there must be something going on anyways - as there is clearly a 
performance drop-off, which surprised me, as lots of apps benefit 
significantly from jumbo frames.

Thanks for the feedback,

-stephen

-- 
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie    http://webstar.deri.ie    http://sindice.com

Mime
View raw message