Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 26 Apr 2017 01:08:04 +0000 (UTC)
From: "Eugene Kirpichov (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13007515.1474879333000.50476.1493168884110@Atlassian.JIRA>
In-Reply-To: <JIRA.13007515.1474879333000@Atlassian.JIRA>
References: <JIRA.13007515.1474879333000@Atlassian.JIRA> <JIRA.13007515.1474879333320@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-673) Data locality for Read.Bounded
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 26 Apr 2017 01:08:08 -0000


    [ https://issues.apache.org/jira/browse/BEAM-673?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D159839=
50#comment-15983950 ]=20

Eugene Kirpichov commented on BEAM-673:
---------------------------------------

On second thought, this might be related to SDF: processing different restr=
ictions of the same element may have different requirements.

Or more like: a design for DoFn's giving hints to runners about their resou=
rce requirements would need to include some data dependence. I don't have a=
 good idea about how to express it in a way that will be modular and will c=
ombine well with the rest of the Beam model and various tricks runners are =
allowed to do (such as fusion or materialization).

> Data locality for Read.Bounded
> ------------------------------
>
>                 Key: BEAM-673
>                 URL: https://issues.apache.org/jira/browse/BEAM-673
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Isma=C3=ABl Mej=C3=ADa
>             Fix For: First stable release
>
>
> In some distributed filesystems, such as HDFS, we should be able to hint =
to Spark the preferred locations of splits.
> Here is an example of how Spark does that for Hadoop RDDs:
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/a=
pache/spark/rdd/NewHadoopRDD.scala#L249


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)