hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taylor, Ronald C" <ronald.tay...@pnl.gov>
Subject hadoop-hbase failure - could use some help, a class is apparently not being found by Hadoop
Date Fri, 17 Sep 2010 23:19:13 GMT

Hi folks,

Got a problem in basic Hadoop-Hbase communication. My small test program ProteinCounter1.java
- shown in full below - reports out this error

   java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)

The full invocation and error msgs are shown at bottom.

We are using Hadoop 20.2 with HBase0.89.2010726 on a 24-node cluster. Hadoop and Hbase each
appears to work fine separately. That is, I've created programs that run MapReduce on files,
and programs that import data into Hbase tables and manipulate such. Both types of programs
have gone quite smoothly.

Now I want to combine the two - use MapReduce programs on data drawn from an Hbase table,
with results placed back into an Hbase table.

But my test program for such, as you see from the error msg, is not working. Apparently the
   org.apache.hadoop.hbase.mapreduce.TableOutputFormat
 class is not found.

However, I have added these paths, including the relevant Hbase *.jar, to HADOOP_CLASSPATH,
so the missing class should have been found, as you can see:

 export HADOOP_CLASSPATH=/home/hbase/hbase/conf: /home/hbase/hbase/hbase-0.89.20100726.jar:
/home/rtaylor/HadoopWork/log4j-1.2.16.jar: /home/rtaylor/HadoopWork/zookeeper-3.3.1.jar

 This change was made in the ../hadoop/conf/hadoop-env.sh file.

I checked the manifest of /home/hbase/hbase/hbase-0.89.20100726.jar and
    org/apache/hadoop/hbase/mapreduce/TableOutputFormat.class
 is indeed present that Hbase *.jar file.

Also, I have restarted both Hbase and Hadoop after making this change.

Don't understand why the TableOutputFormat class is not being found. Or is the error msg misleading,
and something else is going wrong? I would very much appreciate any advice people have as
to what is going wrong. Need to get this working very soon.

   Regards,
     Ron T.

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

contents of the "ProteinCounter1.java" file:



//  to compile
// javac ProteinCounter1.java
// jar cf ProteinCounterTest.jar  *.class

// to run
//   hadoop jar ProteinCounterTest.jar ProteinCounter1


import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.io.IntWritable;

import java.util.*;
import java.io.*;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.io.*;
import org.apache.hadoop.hbase.util.*;
import org.apache.hadoop.hbase.mapreduce.*;


// %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

/**
 * counts the number of times each protein appears in the proteinTable
 *
 */
public class ProteinCounter1 {


    static class ProteinMapper1 extends TableMapper<ImmutableBytesWritable, IntWritable>
{

        private int numRecords = 0;
        private static final IntWritable one = new IntWritable(1);

        @Override
            public void map(ImmutableBytesWritable row, Result values, Context context) throws
IOException {

            // retrieve the value of proteinID, which is the row key for each protein in the
proteinTable
            ImmutableBytesWritable proteinID_Key = new ImmutableBytesWritable(row.get());
            try {
                context.write(proteinID_Key, one);
            } catch (InterruptedException e) {
                throw new IOException(e);
            }
            numRecords++;
            if ((numRecords % 100) == 0) {
                context.setStatus("mapper processed " + numRecords + " proteinTable records
so far");
            }
        }
    }

    public static class ProteinReducer1 extends TableReducer<ImmutableBytesWritable,
                                               IntWritable, ImmutableBytesWritable> {

        public void reduce(ImmutableBytesWritable proteinID_key, Iterable<IntWritable>
values,
                            Context context)
            throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }

            Put put = new Put(proteinID_key.get());
            put.add(Bytes.toBytes("resultFields"), Bytes.toBytes("total"), Bytes.toBytes(sum));
            System.out.println(String.format("stats : proteinID_key : %d, count : %d",
                                           Bytes.toInt(proteinID_key.get()), sum));
            context.write(proteinID_key, put);
        }
    }

    public static void main(String[] args) throws Exception {

        org.apache.hadoop.conf.Configuration conf;
           conf = org.apache.hadoop.hbase.HBaseConfiguration.create();

        Job job = new Job(conf, "HBaseTest_Using_ProteinCounter");
        job.setJarByClass(ProteinCounter1.class);

        org.apache.hadoop.hbase.client.Scan scan = new Scan();

        String colFamilyToUse = "proteinFields";
        String fieldToUse = "Protein_Ref_ID";

        // retreive this one column from the specified family
        scan.addColumn(Bytes.toBytes(colFamilyToUse), Bytes.toBytes(fieldToUse));

           org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter filterToUse =
                 new org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter();
        scan.setFilter(filterToUse);

        TableMapReduceUtil.initTableMapperJob("proteinTable", scan, ProteinMapper1.class,
                              ImmutableBytesWritable.class,
                                              IntWritable.class, job);
        TableMapReduceUtil.initTableReducerJob("testTable", ProteinReducer1.class, job);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


session output:

[rtaylor@h01 Hadoop]$ javac ProteinCounter1.java

[rtaylor@h01 Hadoop]$ jar cf ProteinCounterTest.jar  *.class

[rtaylor@h01 Hadoop]$ hadoop jar ProteinCounterTest.jar ProteinCounter1

10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.job.tracker;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.local.dir;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.system.dir;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.map.tasks.maximum;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.reduce.tasks.maximum;  Ignoring.
10/09/17 15:46:18 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.job.tracker;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.local.dir;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.system.dir;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.map.tasks.maximum;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.reduce.tasks.maximum;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.name.dir;  Ignoring.
10/09/17 15:46:18 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.data.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.job.tracker;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.local.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.system.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.map.tasks.maximum;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.reduce.tasks.maximum;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.name.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.data.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.job.tracker;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.local.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.system.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.map.tasks.maximum;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: mapred-default.xml:a attempt to override final
parameter: mapred.tasktracker.reduce.tasks.maximum;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.name.dir;  Ignoring.
10/09/17 15:46:19 WARN conf.Configuration: hdfs-default.xml:a attempt to override final parameter:
dfs.data.dir;  Ignoring.
10/09/17 15:46:19 INFO zookeeper.ZooKeeperWrapper: Reconnecting to zookeeper
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.1-942149,
built on 05/07/2010 17:14 GMT
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:host.name=h01.emsl.pnl.gov
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_21
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems
Inc.
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.6.0_21/jre
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadoop/hadoop/bin/../conf:/usr/java/default/lib/tools.jar:/home/hadoop/hadoop/bin/..:/home/hadoop/hadoop/bin/../hadoop-0.20.2-core.jar:/home/hadoop/hadoop/bin/../lib/commons-cli-1.2.jar:/home/hadoop/hadoop/bin/../lib/commons-codec-1.3.jar:/home/hadoop/hadoop/bin/../lib/commons-el-1.0.jar:/home/hadoop/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/home/hadoop/hadoop/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/home/hadoop/hadoop/bin/../lib/commons-net-1.4.1.jar:/home/hadoop/hadoop/bin/../lib/core-3.1.1.jar:/home/hadoop/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/home/hadoop/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/home/hadoop/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/home/hadoop/hadoop/bin/../lib/jets3t-0.6.1.jar:/home/hadoop/hadoop/bin/../lib/jetty-6.1.14.jar:/home/hadoop/hadoop/bin/../lib/jetty-util-6.1.14.jar:/home/hadoop/hadoop/bin/../lib/junit-3.8.1.jar:/home/hadoop/hadoop/bin/../lib/kfs-0.2.2.jar:/home/hadoop/hadoop/bin/../lib/log4j-1.2.15.jar:/home/hadoop/hadoop/bin/../lib/mockito-all-1.8.0.jar:/home/hadoop/hadoop/bin/../lib/oro-2.0.8.jar:/home/hadoop/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/hadoop/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/home/hadoop/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/hadoop/hadoop/bin/../lib/xmlenc-0.52.jar:/home/hadoop/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/home/hadoop/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/home/hbase/hbase/conf:/home/hbase/hbase/hbase-0.89.20100726.jar:/home/rtaylor/HadoopWork/log4j-1.2.16.jar:/home/rtaylor/HadoopWork/zookeeper-3.3.1.jar
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop/bin/../lib/native/Linux-i386-32
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-194.11.1.el5
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:user.name=rtaylor
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/rtaylor
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/rtaylor/HadoopWork/Hadoop
10/09/17 15:46:19 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=h05:2182,h04:2182,h03:2182,h02:2182,h10:2182,h09:2182,h08:2182,h07:2182,h06:2182
sessionTimeout=60000 watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@dcb03b
10/09/17 15:46:19 INFO zookeeper.ClientCnxn: Opening socket connection to server h04/192.168.200.24:2182
10/09/17 15:46:19 INFO zookeeper.ClientCnxn: Socket connection established to h04/192.168.200.24:2182,
initiating session
10/09/17 15:46:19 INFO zookeeper.ClientCnxn: Session establishment complete on server h04/192.168.200.24:2182,
sessionid = 0x22b21c04c330002, negotiated timeout = 60000
10/09/17 15:46:20 INFO mapred.JobClient: Running job: job_201009171510_0004
10/09/17 15:46:21 INFO mapred.JobClient:  map 0% reduce 0%

10/09/17 15:46:27 INFO mapred.JobClient: Task Id : attempt_201009171510_0004_m_000002_0, Status
: FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
        at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:193)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:413)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:288)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
        ... 4 more

10/09/17 15:46:33 INFO mapred.JobClient: Task Id : attempt_201009171510_0004_r_000051_0, Status
: FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
        at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:193)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:413)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
        ... 4 more

I terminated the program here via <Control><C>, since the error msgs were simply
repeating.

[rtaylor@h01 Hadoop]$

Mime
View raw message