hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Zarochentsev <alexander_zarochent...@xyratex.com>
Subject Hadoop optimization for Lustre FS
Date Wed, 16 May 2012 08:34:28 GMT

there is an optimization for Hadoop on Lustre FS, or any 
high-performance distributed filesystem.

The research paper with test results can be found here
and a presentation for LUG 2011:

Basically the optimization is a replacement for http transport in 
shuffle phase by simple linking target file to the source one. I 
attached a draft patch against hadoop-1.0.0 to illustrate the idea.
How to push this patch upstream?


Alexander "Zam" Zarochentsev

This email may contain privileged or confidential information, which should only be used for
the purpose for which it was sent by Xyratex. No further rights or licenses are granted to
use such information. If you are not the intended recipient of this message, please notify
the sender by return and delete it. You may not use, copy, disclose or rely on the information
contained in it.
Internet email is susceptible to data corruption, interception and unauthorised amendment
for which Xyratex does not accept liability. While we have taken reasonable precautions to
ensure that this email is free of viruses, Xyratex does not accept liability for the presence
of any computer viruses in this email, nor for any losses caused as a result of viruses.
Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office,
Langstone Road, Havant, Hampshire, PO9 1SA.
The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex
International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia,
Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex
Japan Limited registered in Japan.

View raw message