pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Will Lauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-4608) FOREACH ... UPDATE
Date Wed, 17 Jan 2018 19:25:03 GMT

    [ https://issues.apache.org/jira/browse/PIG-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329258#comment-16329258
] 

Will Lauer commented on PIG-4608:
---------------------------------

{quote}This is Yahoo-centric, but would it be possible to grep our logs for existing pig jobs
and see how many of them have keyword conflicts with 'update', 'delete', 'drop', etc? I'm
indifferent on 'delete' versus 'drop', but it'd be interesting to know which one would impact
fewer existing scripts.{quote}
Keyword conflicts shouldn't really be an issue give the syntax we are talking about. Given
the way the syntax works, it will always be obvious to the parser whether "drop"/"delete"
is refering to a column or is a keyword.

> FOREACH ... UPDATE
> ------------------
>
>                 Key: PIG-4608
>                 URL: https://issues.apache.org/jira/browse/PIG-4608
>             Project: Pig
>          Issue Type: New Feature
>            Reporter: Haley Thrapp
>            Priority: Major
>
> I would like to propose a new command in Pig, FOREACH...UPDATE.
> Syntactically, it would look much like FOREACH … GENERATE.
> Example:
> Input data:
> (1,2,3)
> (2,3,4)
> (3,4,5)
> -- Load the data
> three_numbers = LOAD 'input_data'
> USING PigStorage()
> AS (f1:int, f2:int, f3:int);
> -- Sum up the row
> updated = FOREACH three_numbers UPDATE
> 5 as f1,
> f1+f2 as new_sum
> ;
> Dump updated;
> (5,2,3,3)
> (5,3,4,5)
> (5,4,5,7)
> Fields to update must be specified by alias. Any fields in the UPDATE that do not match
an existing field will be appended to the end of the tuple.
> This command is particularly desirable in scripts that deal with a large number of fields
(in the 20-200 range). Often, we need to only make modifications to a few fields. The FOREACH
... UPDATE statement, allows the developer to focus on the actual logical changes instead
of having to list all of the fields that are also being passed through.
> My team has prototyped this with changes to FOREACH ... GENERATE. We believe this can
be done with changes to the parser and the creation of a new LOUpdate. No physical plan changes
should be needed because we will leverage what LOGenerate does.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message