falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1480) Gather data transfer details of Hive DR
Date Thu, 24 Sep 2015 09:55:57 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906128#comment-14906128
] 

Peeyush Bishnoi commented on FALCON-1480:
-----------------------------------------

Approach for this to create custom mapreduce counters for the Hive replication job. If distcp
jobs are invoked from Hive replication map tasks, populate custom MR counters in each map
task with required counters value obtained from the completed distcp jobs.
Once Hive replication job completed successfully, obtain and store aggregate custom counters
from Hive replication job in a counter file. Once counter file created, details for populating
the GraphDB is similar as I have done in FALCON-1102. 


> Gather data transfer details of Hive DR
> ---------------------------------------
>
>                 Key: FALCON-1480
>                 URL: https://issues.apache.org/jira/browse/FALCON-1480
>             Project: Falcon
>          Issue Type: Sub-task
>          Components: replication
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>
> Hive DR invokes replication job to transfer Hive metadata and data from source to destination.
It is required to capture the data transfer details like bytes transferred, files copied of
replication job and then populate the the backend store from where UI can access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message