hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Stack <st...@duboce.net>
Subject Re: Hadoop/HBase setup
Date Mon, 05 Nov 2007 15:05:02 GMT
I'm glad you got it working.  Ping if any new issues.

Hadoop/hbase passes the configured address to InetSocketAddress 
(http://java.sun.com/j2se/1.4.2/docs/api/java/net/InetSocketAddress.html).  
It attempts resolving the given hostname.  I should have suggested you 
use an IP.

St.Ack

jonathan doklovic wrote:
> Hey Michael,
>
> First, thanks for your help on this list and IRC, it was very helpful.
>
> I'm happy to report that my problem is resolved, although I'm unhappy
> with the cause.
>
> Basically, our DHCP server is not configured properly and so it's
> assigning me a hostname that doesn't map to my actual IP address.
>
> I do still think there's some weirdness in the hadoop internals
> though....  Since I'm telling it to start on localhost, *everything*
> should be binding and connecting to 127.0.0.1, however, it seems that
> some parts of hadoop or hbase may actually be grabbing my hostname and
> doing a reverse lookup on it and then trying to connect to the IP
> returned.
>
> Anyway, I was able to get everything running by mapping localhost to my
> actual IP in my /etc/hosts file.
>
> Thanks,
>
> - Jonathan
>
>
> On Thu, 2007-11-01 at 13:43 -0700, Michael Stack wrote:
>   
>> Hey Jonathan. 
>>
>>  From the below, the regionserver looks to have reported into the master 
>> fine and even gotten instruction that it should deploy the -ROOT- region 
>> but then when master tried to talk back later, it couldn't.  I have seen 
>> this previous when hosts were confused on how to reach each other -- is 
>> there a bogus entry in an /etc/hosts?
>>
>> But it looks like you are trying the hbase from the hadoop 0.14.x 
>> branch.  IMO, you'll get further if you try the first hbase release 
>> bundled with the hadoop 0.15.0 candidate -- 
>> http://people.apache.org/~cutting/hadoop-0.15.0-candidate-1/ -- or go 
>> get a recent nightly build from here: 
>> http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/.
>>
>> St.Ack
>>
>>
>> jonathan doklovic wrote:
>>     
>>> Hi,
>>>
>>> I'm trying to evaluate hadoop/hbase for a project I'm on that requires
>>> filtering massive amounts of RSS data.
>>>
>>> I've been trying to follow the simple tutorials, but I can't seem to get
>>> anything to work.
>>>
>>> So far, I've formatted hadoop storage, that went fine.
>>> Then I start hadoop:
>>>
>>> jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/bin> ./start-dfs.sh 
>>> starting namenode, logging
>>> to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-namenode-msp-na214982.out
>>> localhost: starting datanode, logging
>>> to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-datanode-msp-na214982.out
>>> localhost: starting secondarynamenode, logging
>>> to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-secondarynamenode-msp-na214982.out
>>>
>>> next I start hbase:
>>>
>>> jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/src/contrib/hbase/bin> ./start-hbase.sh

>>> starting master, logging
>>> to /home/jdoklovic/apps/hadoop-0.14.3/src/contrib/hbase/bin/../../../..//logs/hbase-jdoklovic-master-msp-na214982.out
>>> localhost: starting regionserver, logging
>>> to /home/jdoklovic/apps/hadoop-0.14.3/src/contrib/hbase/bin/../../../..//logs/hbase-jdoklovic-regionserver-msp-na214982.out
>>>
>>> right after this, I tail the master and region server logs.
>>> In the region server log, everything seems fine....
>>>
>>> jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/logs> tail
>>> -f ./hbase-jdoklovic-regionserver-msp-na214982.log 
>>> 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 60010: starting
>>> 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 4 on 60010: starting
>>> 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 60010: starting
>>> 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 6 on 60010: starting
>>> 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 7 on 60010: starting
>>> 2007-11-01 14:48:34,617 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 8 on 60010: starting
>>> 2007-11-01 14:48:34,617 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 9 on 60010: starting
>>> 2007-11-01 14:48:34,617 INFO org.apache.hadoop.hbase.HRegionServer:
>>> HRegionServer started at: 10.0.4.29:60010
>>> 2007-11-01 14:48:34,808 INFO org.apache.hadoop.hbase.HStore: HStore
>>> online for --ROOT--,,0/info
>>> 2007-11-01 14:48:34,811 INFO org.apache.hadoop.hbase.HRegion: region
>>> --ROOT--,,0 available
>>>
>>> However, the master server goes crazy and can't seem to connect to the
>>> region server:
>>>
>>> jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/logs> tail -f
>>> hbase-jdoklovic-master-msp-na214982.log 
>>> 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 1 on 60000: starting
>>> 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 2 on 60000: starting
>>> 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 3 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 4 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 6 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 7 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 8 on 60000: starting
>>> 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 9 on 60000: starting
>>> 2007-11-01 14:48:34,618 INFO org.apache.hadoop.hbase.HMaster: received
>>> start message from: 10.0.4.29:60010
>>> 2007-11-01 14:48:49,667 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 1 time(s).
>>> 2007-11-01 14:48:50,669 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 2 time(s).
>>> 2007-11-01 14:48:51,673 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 3 time(s).
>>> 2007-11-01 14:48:52,678 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 4 time(s).
>>> 2007-11-01 14:48:53,680 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 5 time(s).
>>> 2007-11-01 14:48:54,681 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 6 time(s).
>>> 2007-11-01 14:48:55,686 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 7 time(s).
>>> 2007-11-01 14:48:56,690 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 8 time(s).
>>> 2007-11-01 14:48:57,692 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 9 time(s).
>>> 2007-11-01 14:48:58,696 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect to server: /10.0.4.29:60010. Already tried 10 time(s).
>>> 2007-11-01 14:48:59,698 INFO org.apache.hadoop.ipc.RPC: Server
>>> at /10.0.4.29:60010 not available yet, Zzzzz...
>>>
>>> this repeats forever.
>>>
>>> Can anyone help me get this running properly?
>>>
>>> My config files are really simple:
>>>
>>> hadoop-site.xml
>>> ---------------
>>> <configuration>
>>> <property>
>>>      <name>hadoop.tmp.dir</name>
>>>      <value>/tmp/hadoop</value>
>>> </property>
>>> <property>
>>>      <name>fs.default.name</name>
>>>      <value>localhost:8585</value>
>>> </property>
>>> <property>
>>>      <name>mapred.job.tracker</name>
>>>      <value>localhost:8686</value>
>>> </property>
>>> <property>
>>>   <name>mapred.child.java.opts</name>
>>>   <value>-Xmx512m</value>
>>> </property>
>>> </configuration>
>>>
>>> hbase-site.xml
>>> ---------------
>>> <configuration>
>>> <property>
>>>      <name>hbase.master</name>
>>>      <value>localhost:60000</value>
>>>    </property>
>>> <property>
>>>      <name>hbase.regionserver</name>
>>>      <value>localhost:60010</value>
>>>    </property>
>>>
>>> </configuration>
>>>
>>>
>>> Thanks,
>>>
>>> - Jonathan
>>>
>>>
>>>   
>>>       


Mime
View raw message