Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 27 Sep 2017 12:50:00 +0000 (UTC)
From: "James Xu (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13021388.1479372231000.214678.1506516600086@Atlassian.JIRA>
In-Reply-To: <JIRA.13021388.1479372231000@Atlassian.JIRA>
References: <JIRA.13021388.1479372231000@Atlassian.JIRA> <JIRA.13021388.1479372231782@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-995) Apache Pig DSL
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 27 Sep 2017 12:50:05 -0000


    [ https://issues.apache.org/jira/browse/BEAM-995?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D161825=
02#comment-16182502 ]=20

James Xu commented on BEAM-995:
-------------------------------

[~nielsbasjes] Pig actually is doing the similar thing as Beam:=20

# They both defined an unified data processing API
# They both support several backends.

So do pig-on-beam on either side does not have so much difference. I prefer=
 to do it on beam-side because:

# BEAM is already doing the `support several backends` thing, let's just le=
t BEAM do it, make pig focus more on its primary advantage: the friendly AP=
I.
# To align with other extension like SQL.


For the pros to do it on pig-side you mentioned:

1. Builtin facilities for loading UDFs and UDAFs

> Yes, I agree, the existing UDFs and UDAFs are very important. If we do pi=
g-on-beam on beam-side, we will have something like `UDFAdapter` which will=
 adapt all existing UDFs, so we can use them in the new pig-on-beam.

2. Execution flow optimizer(s)

> There is pipeline optimizer in BEAM, and also an optimizer in underline e=
ngine(Spark, MapReduce), will pig optimizer matter so much in this context?=
 (I am not familiar with Pig, correct me if I am wrong)

3. A selection of execution backends.

> Beam itself supports all the different backends.

> Apache Pig DSL
> --------------
>
>                 Key: BEAM-995
>                 URL: https://issues.apache.org/jira/browse/BEAM-995
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-ideas
>            Reporter: Jean-Baptiste Onofr=C3=A9
>            Assignee: Jean-Baptiste Onofr=C3=A9
>
> Apache Pig is still popular and the language is not so large.
> Providing a DSL using the Pig language would potentially allow more peopl=
e to use Beam (at least during a transition period).


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)