falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Ramesh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-20) Remove dependency on custom InMobi DistCp
Date Mon, 13 Oct 2014 17:39:34 GMT

    [ https://issues.apache.org/jira/browse/FALCON-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169608#comment-14169608
] 

Sowmya Ramesh commented on FALCON-20:
-------------------------------------

bq. Replication will pick distcp jar from oozie sharelib. Is falcon's FeedReplicator guaranteed
to work with any distcp in terms of code signature?

Few changes was required in CustomReplicator and only change in FeedReplicator was to use
setMapBandwidth. Code changes has been tested and replication works.

bq. If latest distcp doesn't support mapBandwidth in KB, we should rename falcon's param as
well accordingly. But this is useful, not sure why it was removed

In Inmobi DistCpOptions code for setMapBandwidthKB was

{noquote}
public void setMapBandwidthKB(int mapBandwidth) {
    if (mapBandwidth <= 0) {
      throw new IllegalArgumentException("Bandwidth " +
          mapBandwidth + " is invalid (should be > 0)");
    }
    this.mapBandwidth = mapBandwidth;
  }
{noquote}

In Hadoop tools DistCP code for setMapBandwidth is
{noquote}
  public void setMapBandwidth(int mapBandwidth) {
    assert mapBandwidth > 0 : "Bandwidth " + mapBandwidth + " is invalid (should be >
0)";
    this.mapBandwidth = mapBandwidth;
  }
{noquote}

As both methods have same logic and mapBandwidth is int I just retained the same param name.




> Remove dependency on custom InMobi DistCp
> -----------------------------------------
>
>                 Key: FALCON-20
>                 URL: https://issues.apache.org/jira/browse/FALCON-20
>             Project: Falcon
>          Issue Type: Sub-task
>          Components: replication
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Sowmya Ramesh
>             Fix For: 0.6
>
>         Attachments: FALCON-20-C.patch, FALCON-20.patch, FALCON-20.r1.patch, FALCON-20.v2.patch,
cluster01.xml, cluster02.xml, feed.xml
>
>
> Falcon depends on custom DistCp 0.3 since DistCp v2 was part of Hadoop 2.x line. 
> https://issues.apache.org/jira/browse/MAPREDUCE-5081 has back ported DistCp v2 to Hadoop
1.x line and Falcon should instead depend on that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message