falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-287) Record lineage information in post processing
Date Fri, 28 Feb 2014 15:56:22 GMT

    [ https://issues.apache.org/jira/browse/FALCON-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915916#comment-13915916
] 

Srikanth Sundarrajan commented on FALCON-287:
---------------------------------------------

There seems to be redundant work in the current form. Perhaps it might make sense to keep
the JMS message light weight and rely on FS state or we can use a more light weight message
format for the JMS messaging and piggy back the data for lineage as well. As far as availability
of JMS is concerned, there are more issues should there be an availability issue around JMS
service. 

My personal choice would be to send the entire message over, as the re-runs / late data handling
etc can't be effectively managed without adequate details. We should certainly move away from
the current form of sending data via ARGS and adopt a json format for the message.

Also noticed a element userWorkflowVersion. How is this obtained ?

> Record lineage information in post processing
> ---------------------------------------------
>
>                 Key: FALCON-287
>                 URL: https://issues.apache.org/jira/browse/FALCON-287
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.5
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>              Labels: lineage
>         Attachments: FALCON-287.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message