falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shwetha G S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-30) Enable embedding pig scripts directly in a process
Date Mon, 08 Jul 2013 11:01:50 GMT

    [ https://issues.apache.org/jira/browse/FALCON-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701930#comment-13701930
] 

Shwetha G S commented on FALCON-30:
-----------------------------------

Looked at FALCON-30.r2.patch. Few comments:
1. In OozieProcessMapper, we shouldn't do prepare delete as there can be usecases where users
don't want to do prepare delete(may be for incremental processing or some other random usecase).
2. In OozieProcessMapper.addInputOutputFeedsAsParams(), use ${wf:conf('<param>')} instead
of ${<param>}. Later one doesn't work if param has '.'
3. Feed properties go into just replication and retention wfs. Process properties go into
process parent wf. This is because we didn't see any usecase where feed properties are required
in process. Pig wf doesn't follow this. If you see a usecase for having feed properties in
process, please add it in parent wf for oozie action as well so that its consistent.
4. Feed/process properties are added to conf and will not be available to pig scripts. Should
these be added as params as well? For example, if you want to pass currentHour as param to
pig script, how will you do it?
5. Process lib path is a directory. Will adding the directory as archive, add the files in
it to hadoop distributed cache?
                
> Enable embedding pig scripts directly in a process
> --------------------------------------------------
>
>                 Key: FALCON-30
>                 URL: https://issues.apache.org/jira/browse/FALCON-30
>             Project: Falcon
>          Issue Type: Improvement
>          Components: process
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>             Fix For: 0.3
>
>         Attachments: FALCON-30.patch, FALCON-30.r2.patch, FALCON-30.rev.patch
>
>
> Falcon allows users to express processing as a oozie workflow. This will enable users
to embed pig or hive scripts with out having to express them in a oozie workflow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message