hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sekhon (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-13720) HDFS dataset Anti-Affinity Block Placement across DataNodes for data local task optimization (improve Spark executor utilization & performance)
Date Thu, 05 Jul 2018 10:13:00 GMT
Hari Sekhon created HDFS-13720:
----------------------------------

             Summary: HDFS dataset Anti-Affinity Block Placement across DataNodes for data
local task optimization (improve Spark executor utilization & performance)
                 Key: HDFS-13720
                 URL: https://issues.apache.org/jira/browse/HDFS-13720
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: balancer &amp; mover, block placement, performance
    Affects Versions: 2.7.3
         Environment: Hortonworks HDP 2.6
            Reporter: Hari Sekhon


Improvement Request for Anti-Affinity Block Placement across datanodes such that for a given
data set the blocks are distributed evenly across all available datanodes in order to improve
task scheduling while maintaining data locality.

This could be done via a client side write flag as well as via a balancer command switch
combined with giving a target path to files or directories to redistributed as evenly as
possible across all datanodes in the cluster.

See this following Spark issue which causes massive under-utilisation across jobs. Only 30-50%
of executor cores were being used for tasks due to data locality targeting. Many executors
doing literally nothing, while holding significant cluster resources, because the data set,
which in at least one job was large enough to have 30,000 tasks churning though slowly on
only a subset of the available executors. The workaround in the end was to disable data local
tasks in Spark, but if everyone did that the bottleneck would go back to being the network
and it undermines Hadoop's first premise of don't move the data to compute. For performance
critical jobs, returning tasks to Yarn isn't a good idea either, they want the jobs to use
all the resources available, not just the resources on a subset of nodes that hold a given
dataset or pulling half the blocks across the network.

https://issues.apache.org/jira/browse/SPARK-24474



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org


Mime
View raw message