hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Garner (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2578) HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with defaults
Date Sat, 12 Jan 2008 20:11:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558286#action_12558286
] 

Thomas Garner commented on HADOOP-2578:
---------------------------------------

I'm not sure I understand your logic.  Are you saying that when hbase is reading hbase-default
and hbase-site, that it's not finding them on disk (they are there and properly configured
in /usr/local/hadoop/src/contrib/hbase/conf on both of my mapreduce nodes), and then injecting
defaults from somewhere other than the xml files?

To test this, I edited hbase-default on one node, and I am now seeing the change (hbase.master
to default:60000):

2008-01-12 19:56:59,675 DEBUG org.apache.hadoop.mapred.TaskTracker: Child starting
2008-01-12 19:57:00,038 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
with processName=MAP, sessionId=
2008-01-12 19:57:00,284 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:57:00,353 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
super(c): hbase.master: master:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
hbase-default.xml: hbase.master: default:60000
2008-01-12 19:57:00,420 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
hbase-site.xml: hbase.master: default:60000
2008-01-12 19:57:00,428 DEBUG org.apache.hadoop.hbase.HConnectionManager: DEBUG: HConnection
getConnection(): hbase.master: default:60000
2008-01-12 19:57:00,438 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
super(c): hbase.master: default:60000
2008-01-12 19:57:00,444 DEBUG org.apache.hadoop.hbase.HConnectionManager$TableServers: No
servers for wordcount. Doing a find...
2008-01-12 19:57:10,119 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt
0 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:20,132 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt
1 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:30,144 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt
2 of 5 failed with . Retrying after sleep of 10000
2008-01-12 19:57:40,153 INFO org.apache.hadoop.hbase.HConnectionManager$TableServers: Attempt
3 of 5 failed with . Retrying after sleep of 10000

Here's the log from an earlier mr job/task, where you see the original hbase.master value
be correct (master:60000), then change to local (as was the value in the hbase-default file):

2008-01-12 19:16:49,005 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
with processName=MAP, sessionId=
2008-01-12 19:16:49,253 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2008-01-12 19:16:49,333 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
super(c): hbase.master: master:60000
2008-01-12 19:16:49,403 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
hbase-default.xml: hbase.master: local
2008-01-12 19:16:49,404 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
hbase-site.xml: hbase.master: local
2008-01-12 19:16:49,421 INFO org.apache.hadoop.conf.Configuration: HBaseConfiguration after
super(c): hbase.master: local
2008-01-12 19:16:49,453 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000.
Already tried 1 time(s).
2008-01-12 19:16:50,458 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000.
Already tried 2 time(s).
2008-01-12 19:16:51,462 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000.
Already tried 3 time(s).
2008-01-12 19:16:52,466 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000.
Already tried 4 time(s).

I don't have an issue allowing mr nodes to read hbase default and site configs (say for optimizations
applicable to all hbase mr jobs that are pushed to disk for every node), so long as config
values specific to my job override any default values.

I'm surprised that there's no set method in the configuration class that takes a configuration
as a parameter, then all you'd have to do is super on the default, then set the site and set
the passed config, each overriding any existing parameters.

> HBaseConfiguration(Configuration c) constructor shouldn't overwrite passed conf with
defaults
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2578
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2578
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: Thomas Garner
>
> While testing out mapreduce with hbase, in the map portion of a task, the map would try
to connect to an hbase master at localhost/127.0.0.1.  The config passed to the hbaseconfiguration
contained the necessary hbase configuration information, but I assume was being overwritten
by the defaults in the config files during addResource, as commenting out addHbaseResources
in the constructor fixed the symptom.  I would expect the configs to be layered on top of
each other, e.g. default, then site, then the config passed as a parameter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message