hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5396) YARN large file broadcast service
Date Fri, 07 Jul 2017 20:26:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078617#comment-16078617
] 

Inigo Goiri commented on YARN-5396:
-----------------------------------

We are also interested on this and we may be able to add resources for testing, etc.
[~mingma], can you add a pointer to the Spark bittorrent broadcasting?

> YARN large file broadcast service
> ---------------------------------
>
>                 Key: YARN-5396
>                 URL: https://issues.apache.org/jira/browse/YARN-5396
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Zhiyuan Yang
>            Assignee: Zhiyuan Yang
>         Attachments: slides-prototype.pdf, YARN-broadcast-prototype.patch, YARNFileTransferService-prototype.pdf
>
>
> In Hadoop and related softwares, there are demands of broadcasting large files. For example,
YARN application may localize large jar files on each node; Hive may distribute large tables
in fragment-replicate joins; docker integration may broadcast large container image. The current
local resource based solution is to put the files on HDFS and let each node download from
HDFS, which is inefficient and not scalable. So we want to build a better file transfer service
in YARN so that all applications can use it broadcast large file efficiently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message