giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Ho <csz...@comp.polyu.edu.hk>
Subject Re: NPE encountered when HiveApiInputFormat is computing the InputSplits
Date Mon, 18 Nov 2013 03:41:49 GMT
The problem remains even we submit the job by:
hive --service jar giraph-hive-1.0.0-jar-with-dependencies.jar
org.apache.giraph.hive.HiveGiraphRunner

We are using hadoop-1.0.4, hive-0.11 and giraph-1.0.0. Are they
incompatible? Do we need to configure
any environment variable for Giraph?


On Sat, Nov 16, 2013 at 10:44 PM, Andy Ho <cszahe@comp.polyu.edu.hk> wrote:

> Hi folks,
>
> Currently, we are going to use Giraph to replace some graph processing in
> our Hive workflow.
> We did't use HiveGiraphRunner to submit the job directly, but customize
> and submit it in our own program.
>
> However, after the job is submitted to hadoop, NPE encountered when
> HiveApiInputFormat is computing the InputSplits:
>
> Below is the scala code snippet about the job configuration:
>
> ==================================================
>
> val hive_config_copy = new HiveConf(hive_config)
> val workers = 1
> val dbName = "default"
> val edgeInputTableStr = "transitMatrix"
> val vertexInputTableStr = "initialRank"
> val vertexOutputTableStr = "twitterRank"
>
> HIVE_TO_VERTEX_CLASS.set(hive_config_copy, classOf[InitialRankToVertex])
> HIVE_TO_EDGE_CLASS.set(hive_config_copy, classOf[TransitMatrixToEdge])
> hive_config_copy.setClass(HiveVertexWriter.VERTEX_TO_HIVE_KEY, classOf[TRVertexToHive],
> classOf[VertexToHive[Text, DoubleWritable, Writable]])
>
> val job = new GiraphJob(hive_config_copy, getClass().getName())
> var giraphConf = job.getConfiguration()
> giraphConf.setVertexClass(classOf[TwitterRankVertex])
>
> var hiveVertexInputDescription = new HiveInputDescription()
> var hiveEdgeInputDescription = new HiveInputDescription()
> var hiveOutputDescription = new HiveOutputDescription()
>
> /**
> * Initialize hive input db and tables
> */
>
> hiveVertexInputDescription.setDbName(dbName)
> hiveEdgeInputDescription.setDbName(dbName)
> hiveOutputDescription.setDbName(dbName)
> hiveEdgeInputDescription.setTableName(edgeInputTableStr)
> hiveVertexInputDescription.setTableName(vertexInputTableStr)
> hiveOutputDescription.setTableName(vertexOutputTableStr)
>
>
> /**
> * Initialize the hive input settings
> */
>
> hiveVertexInputDescription.setNumSplits(HIVE_VERTEX_SPLITS.get(giraphConf))
> HiveApiInputFormat.setProfileInputDesc(giraphConf,
> hiveVertexInputDescription, VERTEX_INPUT_PROFILE_ID)
> giraphConf.setVertexInputFormatClass(classOf[HiveVertexInputFormat[Text,
> DoubleWritable, Writable]])
> HiveTableSchemas.put(giraphConf,
> VERTEX_INPUT_PROFILE_ID,hiveVertexInputDescription.hiveTableName())
>
> hiveEdgeInputDescription.setNumSplits(HIVE_EDGE_SPLITS.get(giraphConf))
> HiveApiInputFormat.setProfileInputDesc(giraphConf,
> hiveEdgeInputDescription,EDGE_INPUT_PROFILE_ID)
> giraphConf.setEdgeInputFormatClass(classOf[HiveEdgeInputFormat[Text,
> DoubleWritable]]);
> HiveTableSchemas.put(giraphConf,
> EDGE_INPUT_PROFILE_ID,hiveEdgeInputDescription.hiveTableName())
>
> /**
> * Initialize the hive output settings
> */
>
> HiveApiOutputFormat.initProfile(giraphConf,
> hiveOutputDescription,VERTEX_OUTPUT_PROFILE_ID)
> giraphConf.setVertexOutputFormatClass(classOf[HiveVertexOutputFormat[Text,
> DoubleWritable, Writable]])
> HiveTableSchemas.put(giraphConf,
> VERTEX_OUTPUT_PROFILE_ID,hiveOutputDescription.hiveTableName())
>
> /**
> * Set number of workers
> */
>
> giraphConf.setWorkerConfiguration(workers, workers, 100.0f)
>
> /**
> * Run the job
> */
>
> if (job.run(true)) return true else return false
>
>
> =========================================
>
> Here are the task logs:
>
> 2013-11-16 12:19:19,032 INFO
> com.facebook.giraph.hive.input.HiveApiInputFormat: getSplits for profile
> vertex_input_profile
> 2013-11-16 12:19:19,034 WARN org.apache.hadoop.hive.conf.HiveConf:
> hive-site.xml not found on CLASSPATH
> 2013-11-16 12:19:19,161 INFO org.apache.hadoop.mapred.FileInputFormat:
> Total input paths to process : 1
> 2013-11-16 12:19:19,164 ERROR org.apache.giraph.master.MasterThread:
> masterThread: Master algorithm failed with NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.TextInputFormat.isSplitable(TextInputFormat.java:42)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:232)
> at
> com.facebook.giraph.hive.input.HiveApiInputFormat.computeSplits(HiveApiInputFormat.java:183)
> at
> com.facebook.giraph.hive.input.HiveApiInputFormat.getSplits(HiveApiInputFormat.java:166)
> at
> com.facebook.giraph.hive.input.HiveApiInputFormat.getSplits(HiveApiInputFormat.java:147)
> at
> org.apache.giraph.hive.input.vertex.HiveVertexInputFormat.getSplits(HiveVertexInputFormat.java:60)
> at
> org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:314)
> at
> org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:626)
> at
> org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:692)
>
>
>  at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
>
> Not sure if anything is missing in the job configuration.
> Can anybody help? Thanks in advance.
>
> Best Wishes,
> ~Andy
>



-- 
~Andy

Mime
View raw message