giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Orlando <kyle.r.orla...@gmail.com>
Subject Re: Getting Giraph to work with HBase
Date Thu, 01 Aug 2013 15:57:08 GMT
Update:

The NullPointerException being thrown was an error on my part: I had
accidentally deleted the table I was using minutes prior. But
exceptions were still being thrown after I fixed that.  Eventually, I
found this in the logs:

FATAL org.apache.giraph.graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread
org.apache.giraph.master.MasterThread, msg =
com/google/protobuf/Message, exiting...
java.lang.NoClassDefFoundError: com/google/protobuf/Message

I fixed this by adding protobuf-java-2.4.0a.jar in $HBASE_HOME/lib to
the hadoop classpath. In my case, I just copied the jar into
$HADOOP_HOME/lib. I started her up again, and it worked fine.  Also, I
needed to add this to my class that extends HBaseVertexInputFormat to
make it work:

private ImmutableClassesGiraphConfiguration<I, V, E> conf;
	
	@Override
	  public void setConf(ImmutableClassesGiraphConfiguration<I, V, E> conf) {
		conf.set(TableInputFormat.INPUT_TABLE, "edges");
		this.conf = conf;
	  }
	
	@Override
		public ImmutableClassesGiraphConfiguration<I, V, E> getConf() {
		return conf;
	}

On Wed, Jul 31, 2013 at 7:13 PM, Kyle Orlando <kyle.r.orlando@gmail.com> wrote:
> Hello,
>
> I've been having some problems getting Giraph to work with a table
> that is stored in HBase.  I've created my own classes that extend
> HBaseVertexInputFormat and HBaseVertexReader (respectively) and
> implemented nextVertex() and getVertex() accordingly, but I still
> can't get it to work. Here is what I typed into the command line after
> specifying the GiraphRunner class:
>
> org.apache.giraph.examples.SimplePageRankComputation -vif
> HBasePractice.LongDoubleFloatHBaseVertexInputFormat -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /hbaseOut
> -w 2 -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>
> This is the error that I'm getting from the logs:
>
> 2013-07-31 17:20:28,880 ERROR org.apache.giraph.master.MasterThread:
> masterThread: Master algorithm failed with NullPointerException
> java.lang.NullPointerException
>         at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
>         at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
>         at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
>         at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
>         at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
>         at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
>         at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
> 2013-07-31 17:20:28,881 FATAL org.apache.giraph.graph.GraphMapper:
> uncaughtException: OverrideExceptionHandler on thread
> org.apache.giraph.master.MasterThread, msg =
> java.lang.NullPointerException, exiting...
> java.lang.IllegalStateException: java.lang.NullPointerException
>         at org.apache.giraph.master.MasterThread.run(MasterThread.java:185)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
>         at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
>         at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
>         at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
>         at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
>         at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
>         at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
>
>  I don't know how/why a NullPointerException is being thrown.  Do I
> need to specify the -vip option? I didn't think so, because in my code
> for my class that extends HBaseVertexInputFormat I do the following:
>
> private ImmutableClassesGiraphConfiguration<LongWritable,
> DoubleWritable, FloatWritable> conf;
>
> @Override
>           public void setConf(ImmutableClassesGiraphConfiguration<LongWritable,
> DoubleWritable, FloatWritable> conf) {
>                 conf.set(TableInputFormat.INPUT_TABLE, "edges");
>                 this.conf = conf;
>           }
>
> Help would be much appreciated.  I actually was a bit hesitant to post
> this at first; I was determined to figure it out on my own, but then I
> noticed that there have been virtually no questions asked regarding
> HBase + Giraph on this mailing list, so I figured that posting this
> might help someone out with similar problems later on in the future.
>
> Thanks,
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland



-- 
Kyle Orlando
Computer Engineering Major
University of Maryland

Mime
View raw message