spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey
Date Wed, 10 Dec 2014 03:23:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240579#comment-14240579
] 

Aaron Davidson edited comment on SPARK-4740 at 12/10/14 3:22 AM:
-----------------------------------------------------------------

I spent another day trying to reproduce the issue. Unfortunately, I was again unable to reproduce
it, even without Renyold's patch in play.

NIO: 35 minutes in reduce
Netty: 34 minutes in reduce

I ended up getting 10GigE, but it was not utilized, as the cores were maxed out with only
around 40MB/s of read throughput. I realized at the end of testing that the SSDs were not
being used because we had 160 GB of buffer cache outside of the JVMs which was able to cache
all the data. I then disabled use of the majority of the buffer cache (down to 30GB outside
of the JVM) and saw the above number of 40 MB/s throughput from the SSDs. Performance did
not decrease significantly when we switched from buffer cache to SSDs, which is not surprising
given the relatively low throughput requirements.

The issue may be prominent when there is sufficient CPU capacity to exceed a few disks' worth
of throughput, but that was not even nearly the case here; on the other hand, you guys are
using HDDs and more cores. Unfortunately, no EC2 instance type has 8 HDDs except hs1 which
has 24, but only 16 cores, and it may take significantly more time to get a spark cluster
started on that since we have not tested the scripts on the hs1 instance type.

I am going to downgrade this from blocker to critical for now. We should still try to diagnose
and fix the issue, but the final fix may go into 1.2.1 instead of blocking 1.2 from being
released. We will feature a notice prominently in the release notes that netty may may cause
a regression for certain cluster configurations.

By the way, I also ran the full data set with 2 connections per peer, but I did not see any
sort of the strange asymmetry that you guys reported from your cluster. All machines ran at
around the same rate, completing very nearly the same number of tasks each time.


was (Author: ilikerps):
I spent another day trying to reproduce the issue. Unfortunately, I was again unable to reproduce
it, even without Renyold's patch in play.

NIO: 35 minutes in reduce
Netty: 34 minutes in reduce

40 minutes

> Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey
> ------------------------------------------------------------------------
>
>                 Key: SPARK-4740
>                 URL: https://issues.apache.org/jira/browse/SPARK-4740
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Zhang, Liye
>            Assignee: Reynold Xin
>         Attachments: (rxin patch better executor)TestRunner  sort-by-key - Thread dump
for executor 3_files.zip, (rxin patch normal executor)TestRunner  sort-by-key - Thread dump
for executor 0 _files.zip, Spark-perf Test Report 16 Cores per Executor.pdf, Spark-perf Test
Report.pdf, TestRunner  sort-by-key - Thread dump for executor 1_files (Netty-48 Cores per
node).zip, TestRunner  sort-by-key - Thread dump for executor 1_files (Nio-48 cores per node).zip,
rxin_patch-on_4_node_cluster_48CoresPerNode(Unbalance).7z
>
>
> When testing current spark master (1.3.0-snapshot) with spark-perf (sort-by-key, aggregate-by-key,
etc), Netty based shuffle transferService takes much longer time than NIO based shuffle transferService.
The network throughput of Netty is only about half of that of NIO. 
> We tested with standalone mode, and the data set we used for test is 20 billion records,
and the total size is about 400GB. Spark-perf test is Running on a 4 node cluster with 10G
NIC, 48 cpu cores per node and each executor memory is 64GB. The reduce tasks number is set
to 1000. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message