falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1102) Gather data transfer detail of replication job submitted from HDFS recipe
Date Wed, 22 Apr 2015 12:46:58 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507001#comment-14507001

Peeyush Bishnoi commented on FALCON-1102:

Patch is attached that will populate the GraphDB with BYTESCOPIED once data transfer completed
from HDFS recipe. In the given patch, distcp job counter "BYTESCOPIED" will be temporarily
stored in the counter file inside the workflow log directory once Replication workflow finished.
After this, FalconPostProcessing (invoked through different workflow) will make a check for
the counter file to read the counter information. Once counter information is available, it
will be added to WorkflowExecutionContext class. 
On JMS invocation from FalconPostProcessing class, value of "BYTESCOPIED" counter (stored
in WorkflowExecutionContext) will be accessed in InstanceRelationshipGraphBuilder class and
added to respective process instance vertex in GraphDB so that UI can access. Corresponding
test cases "testLineageForDRCounter" has been added for this in MetadataMappingServiceTest

Please review.

> Gather data transfer detail of replication job submitted from HDFS recipe
> -------------------------------------------------------------------------
>                 Key: FALCON-1102
>                 URL: https://issues.apache.org/jira/browse/FALCON-1102
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.7
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>             Fix For: 0.7
>         Attachments: FALCON-1102.patch
> Falcon UI has a requirement to show data transfer details from replication job invoked
through HDFS recipe. To carry out this, we need to capture the bytes transferred from source
to destination of replication job and then populate to backend store from where UI can access.

This message was sent by Atlassian JIRA

View raw message