Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Tue, 28 Feb 2017 17:28:45 +0000 (UTC)
From: "Davor Bonaci (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.13046347.1488074872000.20598.1488302925555@Atlassian.JIRA>
In-Reply-To: <JIRA.13046347.1488074872000@Atlassian.JIRA>
References: <JIRA.13046347.1488074872000@Atlassian.JIRA> <JIRA.13046347.1488074872798@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1556) Spark executors need to register IO
 factories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 28 Feb 2017 17:28:50 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1556?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15888=
502#comment-15888502 ]=20

Davor Bonaci commented on BEAM-1556:
------------------------------------

Ack the higher point -- this is an SDK specific-requirement.

I think it would make sense if this is done by the SDK _if_ we had init & c=
leanup methods at various scopes (per worker, JVM, process, thread, task, s=
tep, etc.). The runner would then call the appropriately scoped init method=
 and the SDK can do this (trivially). However, this is something we need fo=
r other reasons too, but don't have today. Today, the SDK could call it per=
haps before every step (every DoFn, source, etc.), which seems like an over=
kill.

To unblock the user scenario, I'd be in favor of the approach that the runn=
er calls this now on a per-task basis, with the clear understanding this ha=
s to be cleaned up in the future.

> Spark executors need to register IO factories
> ---------------------------------------------
>
>                 Key: BEAM-1556
>                 URL: https://issues.apache.org/jira/browse/BEAM-1556
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Frances Perry
>            Assignee: Jean-Baptiste Onofr=C3=A9
>
> The Spark executors need to call IOChannelUtils.registerIOFactories(optio=
ns) in order to support GCS file and make the default WordCount example wor=
k.
> Context in this thread: https://lists.apache.org/thread.html/469a139c9eb0=
7e64e514cdea42ab8000678ab743794a090c365205d7@%3Cuser.beam.apache.org%3E


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)