hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milind Bhandarkar <mbhandar...@gopivotal.com>
Subject Re: RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Date Mon, 11 Nov 2013 19:43:01 GMT
Hi Xiaoyi,

This is great !

For clusters that have moved to Hadoop 2.x versions, do you have any work
in progress to rebase Hadoop-RDMA to 2.x ?

Also, I am wondering if the development that you are doing will be
available in Mellanox UDA ? Or is it an independent project ?

- milind


---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)


On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi Lu@cse.osu.edu <
luxi@cse.ohio-state.edu> wrote:

> The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA
> 0.9.8
> (technology preview) package with the following features and bug-fixes:
>
> * Hadoop-RDMA 0.9.8 Features
>
>    - Based on Apache Hadoop 1.2.1
>    - High performance design with native InfiniBand and RoCE support
>      at the verbs-level for HDFS, MapReduce, and RPC components
>    - Compliant with Apache Hadoop 1.2.1 APIs and applications
>    - Easily configurable for native InfiniBand, RoCE and the traditional
>      sockets-based support (Ethernet and InfiniBand with IPoIB)
>    - HDFS over native InfiniBand and RoCE
>        - RDMA-based write
>        - RDMA-based replication
>        - Tuned packet size
>    - MapReduce over native InfiniBand and RoCE
>        - RDMA-based shuffle
>        - Pre-fetching and caching of map outputs
>        - In-memory merge
>        - Overlapping of merge and reduce
>    - RPC over native InfiniBand and RoCE
>        - JVM-bypassed buffer management
>        - On-demand InfiniBand connection setup
>        - RDMA or send/recv based adaptive communication
>    - Tested with
>        - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
>        - RoCE
>        - Various multi-core platforms
>        - Different file systems with disks and SSDs
>
> * Bug Fixes (since 0.9.1)
>
>    - MapReduce over InfiniBand
>      - Reducing the waiting time for the pre-fetch cache read
>      - Enabling Reduce tasks to randomize the shuffle pattern
>    - HDFS over InfiniBand
>      - Fix an issue in processing multiple blocks concurrently in a
>        single task
>
> For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
> please visit the following URL:
>
> http://hadoop-rdma.cse.ohio-state.edu
>
> Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
> version can be viewed by visiting the `Performance' tab of the above
> website.
>
> All questions, feedbacks and bug reports are welcome. Please post it
> to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
> cse.ohio-state.edu).
>
> Thanks,
>
> The Hadoop-RDMA Team
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message