hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
Date Thu, 01 Sep 2016 15:59:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455849#comment-15455849
] 

Andrew Purtell commented on HBASE-16375:
----------------------------------------

Correct, you'd need to make a patch per branch. However in this case I think it's fine to
just put up a patch just for master. Precommit results will generally apply and the committer
can easily run the new test when applying the patch to the other target branches to check
there is not unexpected variation  

> Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager
and jobhistory webapp address of MapReduceTestingShim  
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-16375
>                 URL: https://issues.apache.org/jira/browse/HBASE-16375
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.2.0
>            Reporter: Loknath Priyatham Teja Singamsetty 
>            Assignee: Loknath Priyatham Teja Singamsetty 
>            Priority: Minor
>             Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
>         Attachments: HBASE-16375.master.001.patch, HBASE-16375_0.98_and_above.patch,
HBASE-16375_0.98_and_above_with_tests.patch, HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting "yarn.resourcemanager.webapp.address"
and "mapreduce.jobhistory.webapp.address" which are required for getting the submitted yarn
apps using mapreduce webapp. These properties are not being copied from jobConf of MapReduceTestingShim
resulting in default values.
> {quote}
> HBaseTestingUtility.java
>     // Allow the user to override FS URI for this map-reduce cluster to use.
>     mrCluster = new MiniMRCluster(servers,
>       FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
>       null, null, new JobConf(this.conf));
>     JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
>     if (jobConf == null) {
>       jobConf = mrCluster.createJobConf();
>     }
>     jobConf.set("mapreduce.cluster.local.dir",
>       conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites this while
it should not
>     LOG.info("Mini mapreduce cluster started");
>     // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance and updates
settings.
>     // Our HBase MR jobs need several of these settings in order to properly run.  So
we copy the
>     // necessary config properties here.  YARN-129 required adding a few properties.
>     conf.set("mapreduce.jobtracker.address", jobConf.get("mapreduce.jobtracker.address"));
>     // this for mrv2 support; mr1 ignores this
>     conf.set("mapreduce.framework.name", "yarn");
>     conf.setBoolean("yarn.is.minicluster", true);
>     String rmAddress = jobConf.get("yarn.resourcemanager.address");
>     if (rmAddress != null) {
>       conf.set("yarn.resourcemanager.address", rmAddress);
>     }
>     String historyAddress = jobConf.get("mapreduce.jobhistory.address");
>     if (historyAddress != null) {
>       conf.set("mapreduce.jobhistory.address", historyAddress);
>     }
>     String schedulerAddress =
>       jobConf.get("yarn.resourcemanager.scheduler.address");
>     if (schedulerAddress != null) {
>       conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
>     }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to be added
as well
> {quote}
>     String rmWebappAddress = jobConf.get("yarn.resourcemanager.webapp.address");
>     if (rmWebappAddress != null) {
>       conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
>     }
>     String historyWebappAddress = jobConf.get("mapreduce.jobhistory.webapp.address");
>     if (historyWebappAddress != null) {
>       conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
>     }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties to HBaseTestingUtility
conf object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message