spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zzc (JIRA)" <>
Subject [jira] [Commented] (SPARK-2468) Netty-based block server / client module
Date Thu, 06 Nov 2014 09:57:34 GMT


zzc commented on SPARK-2468:

Hi, Aaron Davidson, I set spark.shuffle.blockTransferService=netty and,
run on CentOS 5.8 with 12G files successfully , but when I set spark.shuffle.blockTransferService=netty
and, there is error:
Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/
/lib64/ version `GLIBC_2.10' not found

I find GLIBC_2.5 on CentOS 5.8 and can not upgrade, how to resolve it. 

> Netty-based block server / client module
> ----------------------------------------
>                 Key: SPARK-2468
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, Spark Core
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>            Priority: Critical
>             Fix For: 1.2.0
> Right now shuffle send goes through the block manager. This is inefficient because it
requires loading a block from disk into a kernel buffer, then into a user space buffer, and
then back to a kernel send buffer before it reaches the NIC. It does multiple copies of the
data and context switching between kernel/user. It also creates unnecessary buffer in the
JVM that increases GC
> Instead, we should use FileChannel.transferTo, which handles this in the kernel space
with zero-copy. See
> One potential solution is to use Netty.  Spark already has a Netty based network module
implemented ( However, it lacks some functionality and is
turned off by default. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message