hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wilcox <Tom.Wil...@cse.org.uk>
Subject RE: Not a host:port issue
Date Wed, 11 Apr 2012 12:14:11 GMT
I forgot to mention, that I am working with Royston on this issue. We have made some progress...

We have managed to get rid of our version/classpath issue by compiling the Java client class
on the namenode with the HBase Classpath and running it from there (however, I then get errors
due to missing client classes where the class has not been distributed to the map nodes).

So if we compile and run on the namenode as follows:

[hadoop1@namenode src]$ javac uk/org/cse/ingestion/SampleUploader.java -cp  `hbase classpath`
[hadoop1@namenode src]$ java -cp `hbase classpath` uk.org.cse.ingestion.SampleUploader sample.10.csv
tomstable dat no 1>class_output 2>&1

The output from this can be found here: http://pastebin.com/jn5e7E2K

Note that, in contrast to the following and previous runs using jars, we get the appropriate
zookeeper version in the client output:

12/04/11 12:20:12 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972,
built on 02/06/2012 10:48 GMT

In contrast to this, if we build a jar from the same place on the namenode and deploy using
hadoop as follows:

jar cf SampleUploader.jar src/uk
HADOOP_CLASSPATH=`hbase classpath` hadoop jar SampleUploader.jar uk.org.cse.ingestion.SampleUploader
sample.10.csv tomstable dat no 1>jar_output 2>&1 

We get the following output: http://pastebin.com/xG98KfYe

With the offending client output showing that we are using the old version of zookeeper:

12/04/11 12:03:25 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
built on 02/23/2011 22:27 GMT

It seems that the "hadoop" command is introducing the older client API jars into the classpath
in place of/ahead of the latest. By testing the successful class run above using the "hadoop"
command in place of "java" we get the same error as with the jar.
HADOOP_CLASSPATH=`hbase classpath` hadoop uk.org.cse.ingestion.SampleUploader sample.10.csv
tomstable dat no 1>class_output_with_hadoop 2>&1

The output (http://pastebin.com/8tfCqGgV) shows the incorrect zookeeper client in use.

Our hadoop-env.sh file has the following additions to the HADOOP_CLASSPATH: 

# Extra Java CLASSPATH elements.  Optional.

Commenting out these lines and rerunning gets rid of the version errors! We are now getting
what appears to be an unrelated local disk error from the job output, but our issue with the
client version mismatch appears to have been resolved by 2 things:

1) Removed all references to HADOOP_CLASSPATH in hadoop-env.sh and replaced with the following
so that any initial HADOOP_CLASSPATH settings have precedence:

# Extra Java CLASSPATH elements.  Optional.

2) Ran the job with the following (so that HADOOP_CLASSPATH contained all appropriate HBase
API jars):

HADOOP_CLASSPATH=`hbase classpath` hadoop jar SampleUploader.jar uk.org.cse.ingestion.SampleUploader
sample.10.csv tomstable dat no

We are now dealing with the following error:

[sshexec] org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid
local directory for taskTracker/hadoop1/distcache/-6735763131868259398_188156722_559071878/namenode/tmp/mapred/staging/hadoop1/.staging/job_201204111219_0013/libjars/hbase-0.95-SNAPSHOT.jar
  [sshexec] 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
  [sshexec] 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
  [sshexec] 	at org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(TrackerDistributedCacheManager.java:172)
  [sshexec] 	at org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDistributedCacheManager.java:187)
  [sshexec] 	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1212)
  [sshexec] 	at java.security.AccessController.doPrivileged(Native Method)
  [sshexec] 	at javax.security.auth.Subject.doAs(Subject.java:396)
  [sshexec] 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
  [sshexec] 	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
  [sshexec] 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
  [sshexec] 	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
  [sshexec] 	at java.lang.Thread.run(Thread.java:662)


-----Original Message-----
From: Tom Wilcox [mailto:Tom.Wilcox@cse.org.uk] 
Sent: 11 April 2012 10:36
To: user@hbase.apache.org
Subject: RE: Not a host:port issue

I am not sure how to confirm which version of the HBase API the client is using. Although
we are referencing the HBase-0.95-SNAPSHOT and zookeeper 3.4.3 jars, we are still seeing the
following message in the program output when building a job:

[sshexec] 12/04/11 10:35:13 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
built on 02/23/2011 22:27 GMT

Not that it is stating the zookeeper version as 3.3.3 and it was built in February (whereas
our referenced jars were built yesterday).

How should I be using the HBase jars with our HBase Client Java program to ensure that it
is the latest version (and how can I properly confirm this)?


-----Original Message-----
From: Royston Sellman [mailto:royston.sellman@googlemail.com] 
Sent: 10 April 2012 18:38
To: user@hbase.apache.org
Subject: RE: Not a host:port issue

The CLASSPATH(S) are here: http://pastebin.com/wbwEL9Li
Looks to me like the client is 0.95-SNAPSHOT as is our HBase server.
However I just noticed the client is built with ZK 3.4.3 but our ZK server is 3.3.3. Is there
any incompatibility between those versions of ZK? (I'm going to make them the same but that
will take a few minutes :)


-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: 10 April 2012 17:08
To: user@hbase.apache.org
Subject: Re: Not a host:port issue

On Tue, Apr 10, 2012 at 2:58 AM, Royston Sellman <royston.sellman@googlemail.com> wrote:
>  [sshexec] java.lang.IllegalArgumentException: Not a host:port pair: 
>  [][][]

We changed how we persist names to zookeeper in 0.92.x.  It used to be a host:port but now
is a ServerName which is host comma port comma startcode and all is prefixed with zk sequenceid.

It looks like your mapreduce job is using an old hbase client.  Is that possible?  Can you
check its CLASSPATH?


View raw message