hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jonathan doklovic <jdoklo...@ibsys.com>
Subject Re: Hadoop/HBase setup
Date Fri, 02 Nov 2007 14:02:10 GMT
Hey Michael,

Thanks for the tips....  my /etc/hosts is fine, nothing strange in
there, just localhost.

I downloaded 0.15.0 and copied over my config files.
Going through the same process as before, I now get a different error.
It looks like the master boots ok, then the region boots ok, then the
master freaks out about some file and the region can't connect to it.

here's what i get in the master's log:

jdoklovic@msp-na214982:~/apps/hadoop-0.15.0/logs>
more ./hbase-jdoklovic-master-msp-na214982.log 
2007-11-02 08:56:03,973 INFO org.apache.hadoop.hbase.HMaster: Root
region dir: /tmp/hadoop/hbase/hregion_-ROOT-,,0
2007-11-02 08:56:04,040 INFO org.apache.hadoop.hbase.HMaster: bootstrap:
creating ROOT and first META regions
2007-11-02 08:56:04,263 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2007-11-02 08:56:04,264 INFO
org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded &
initialized native-zlib
 library
2007-11-02 08:56:04,268 INFO org.apache.hadoop.hbase.HLog: new log
writer created at /tmp/hadoop/hbase/hregion_-ROOT-,,0/l
og/hlog.dat.000
2007-11-02 08:56:04,375 INFO org.apache.hadoop.hbase.HRegion: region
-ROOT-,,0 available
2007-11-02 08:56:04,511 INFO org.apache.hadoop.hbase.HLog: new log
writer created at /tmp/hadoop/hbase/hregion_.META.,,1/l
og/hlog.dat.000
2007-11-02 08:56:04,695 INFO org.apache.hadoop.hbase.HRegion:
region .META.,,1 available
2007-11-02 08:56:04,948 ERROR org.apache.hadoop.hbase.HMaster: bootstrap
org.apache.hadoop.hbase.DroppedSnapshotException: java.io.IOException:
File /tmp/hadoop/hbase/hregion_-ROOT-,,0/info/mapfi
les/6670111347878566293/data could only be replicated to 0 nodes,
instead of 1
        at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1003)
        at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:293)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

Any suggestions?

- Jonathan


On Thu, 2007-11-01 at 13:43 -0700, Michael Stack wrote:
> Hey Jonathan. 
> 
>  From the below, the regionserver looks to have reported into the master 
> fine and even gotten instruction that it should deploy the -ROOT- region 
> but then when master tried to talk back later, it couldn't.  I have seen 
> this previous when hosts were confused on how to reach each other -- is 
> there a bogus entry in an /etc/hosts?
> 
> But it looks like you are trying the hbase from the hadoop 0.14.x 
> branch.  IMO, you'll get further if you try the first hbase release 
> bundled with the hadoop 0.15.0 candidate -- 
> http://people.apache.org/~cutting/hadoop-0.15.0-candidate-1/ -- or go 
> get a recent nightly build from here: 
> http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/.
> 
> St.Ack
> 
> 
> jonathan doklovic wrote:
> > Hi,
> >
> > I'm trying to evaluate hadoop/hbase for a project I'm on that requires
> > filtering massive amounts of RSS data.
> >
> > I've been trying to follow the simple tutorials, but I can't seem to get
> > anything to work.
> >
> > So far, I've formatted hadoop storage, that went fine.
> > Then I start hadoop:
> >
> > jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/bin> ./start-dfs.sh 
> > starting namenode, logging
> > to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-namenode-msp-na214982.out
> > localhost: starting datanode, logging
> > to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-datanode-msp-na214982.out
> > localhost: starting secondarynamenode, logging
> > to /home/jdoklovic/apps/hadoop-0.14.3/bin/../logs/hadoop-jdoklovic-secondarynamenode-msp-na214982.out
> >
> > next I start hbase:
> >
> > jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/src/contrib/hbase/bin> ./start-hbase.sh

> > starting master, logging
> > to /home/jdoklovic/apps/hadoop-0.14.3/src/contrib/hbase/bin/../../../..//logs/hbase-jdoklovic-master-msp-na214982.out
> > localhost: starting regionserver, logging
> > to /home/jdoklovic/apps/hadoop-0.14.3/src/contrib/hbase/bin/../../../..//logs/hbase-jdoklovic-regionserver-msp-na214982.out
> >
> > right after this, I tail the master and region server logs.
> > In the region server log, everything seems fine....
> >
> > jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/logs> tail
> > -f ./hbase-jdoklovic-regionserver-msp-na214982.log 
> > 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 60010: starting
> > 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 60010: starting
> > 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 60010: starting
> > 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 60010: starting
> > 2007-11-01 14:48:34,613 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 60010: starting
> > 2007-11-01 14:48:34,617 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 60010: starting
> > 2007-11-01 14:48:34,617 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 60010: starting
> > 2007-11-01 14:48:34,617 INFO org.apache.hadoop.hbase.HRegionServer:
> > HRegionServer started at: 10.0.4.29:60010
> > 2007-11-01 14:48:34,808 INFO org.apache.hadoop.hbase.HStore: HStore
> > online for --ROOT--,,0/info
> > 2007-11-01 14:48:34,811 INFO org.apache.hadoop.hbase.HRegion: region
> > --ROOT--,,0 available
> >
> > However, the master server goes crazy and can't seem to connect to the
> > region server:
> >
> > jdoklovic@msp-na214982:~/apps/hadoop-0.14.3/logs> tail -f
> > hbase-jdoklovic-master-msp-na214982.log 
> > 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 60000: starting
> > 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 60000: starting
> > 2007-11-01 14:48:33,246 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 60000: starting
> > 2007-11-01 14:48:33,247 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 60000: starting
> > 2007-11-01 14:48:34,618 INFO org.apache.hadoop.hbase.HMaster: received
> > start message from: 10.0.4.29:60010
> > 2007-11-01 14:48:49,667 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 1 time(s).
> > 2007-11-01 14:48:50,669 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 2 time(s).
> > 2007-11-01 14:48:51,673 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 3 time(s).
> > 2007-11-01 14:48:52,678 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 4 time(s).
> > 2007-11-01 14:48:53,680 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 5 time(s).
> > 2007-11-01 14:48:54,681 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 6 time(s).
> > 2007-11-01 14:48:55,686 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 7 time(s).
> > 2007-11-01 14:48:56,690 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 8 time(s).
> > 2007-11-01 14:48:57,692 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 9 time(s).
> > 2007-11-01 14:48:58,696 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect to server: /10.0.4.29:60010. Already tried 10 time(s).
> > 2007-11-01 14:48:59,698 INFO org.apache.hadoop.ipc.RPC: Server
> > at /10.0.4.29:60010 not available yet, Zzzzz...
> >
> > this repeats forever.
> >
> > Can anyone help me get this running properly?
> >
> > My config files are really simple:
> >
> > hadoop-site.xml
> > ---------------
> > <configuration>
> > <property>
> >      <name>hadoop.tmp.dir</name>
> >      <value>/tmp/hadoop</value>
> > </property>
> > <property>
> >      <name>fs.default.name</name>
> >      <value>localhost:8585</value>
> > </property>
> > <property>
> >      <name>mapred.job.tracker</name>
> >      <value>localhost:8686</value>
> > </property>
> > <property>
> >   <name>mapred.child.java.opts</name>
> >   <value>-Xmx512m</value>
> > </property>
> > </configuration>
> >
> > hbase-site.xml
> > ---------------
> > <configuration>
> > <property>
> >      <name>hbase.master</name>
> >      <value>localhost:60000</value>
> >    </property>
> > <property>
> >      <name>hbase.regionserver</name>
> >      <value>localhost:60010</value>
> >    </property>
> >
> > </configuration>
> >
> >
> > Thanks,
> >
> > - Jonathan
> >
> >
> >   
> 

Mime
View raw message