falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1102) Gather data transfer detail of replication job submitted from HDFS recipe
Date Wed, 25 Mar 2015 10:15:53 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379639#comment-14379639

Peeyush Bishnoi commented on FALCON-1102:

To gather data transfer details of replication job submitted from HDFS recipe, FeedReplicator
class will be extended to access distcp job counter "BYTESCOPIED" and store it to new parameter
“COUNTERS"  in WorkflowExecutionContext. Once JMS invocation happen from FalconPostProcessing
class,  value of "BYTESCOPIED" counter (stored in WorkflowExecutionContext) will be obtained
and added to respective process instance vertex in GraphDB so that UI can access. Distcp job
counter "BYTESCOPIED" will only be accessed when some optional argument passed to FeedReplicator
class through HDFS DR recipe workflow.xml. Hence, with this no changes are required to make
in existing feed replication workflow.xml.

Thoughts please.

> Gather data transfer detail of replication job submitted from HDFS recipe
> -------------------------------------------------------------------------
>                 Key: FALCON-1102
>                 URL: https://issues.apache.org/jira/browse/FALCON-1102
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.7
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>             Fix For: 0.7
> Falcon UI has a requirement to show data transfer details from replication job invoked
through HDFS recipe. To carry out this, we need to capture the bytes transferred from source
to destination of replication job and then populate to backend store from where UI can access.

This message was sent by Atlassian JIRA

View raw message