hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyi Lu@cse.osu.edu" <l...@cse.ohio-state.edu>
Subject RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Date Sat, 09 Nov 2013 13:44:47 GMT
The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA 0.9.8
(technology preview) package with the following features and bug-fixes:

* Hadoop-RDMA 0.9.8 Features

   - Based on Apache Hadoop 1.2.1
   - High performance design with native InfiniBand and RoCE support
     at the verbs-level for HDFS, MapReduce, and RPC components
   - Compliant with Apache Hadoop 1.2.1 APIs and applications
   - Easily configurable for native InfiniBand, RoCE and the traditional
     sockets-based support (Ethernet and InfiniBand with IPoIB)
   - HDFS over native InfiniBand and RoCE
       - RDMA-based write
       - RDMA-based replication
       - Tuned packet size
   - MapReduce over native InfiniBand and RoCE
       - RDMA-based shuffle
       - Pre-fetching and caching of map outputs
       - In-memory merge
       - Overlapping of merge and reduce
   - RPC over native InfiniBand and RoCE
       - JVM-bypassed buffer management
       - On-demand InfiniBand connection setup
       - RDMA or send/recv based adaptive communication
   - Tested with
       - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
       - RoCE
       - Various multi-core platforms
       - Different file systems with disks and SSDs

* Bug Fixes (since 0.9.1)

   - MapReduce over InfiniBand
     - Reducing the waiting time for the pre-fetch cache read
     - Enabling Reduce tasks to randomize the shuffle pattern
   - HDFS over InfiniBand
     - Fix an issue in processing multiple blocks concurrently in a
       single task

For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
please visit the following URL:


Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
version can be viewed by visiting the `Performance' tab of the above website.

All questions, feedbacks and bug reports are welcome. Please post it
to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at


The Hadoop-RDMA Team

View raw message