giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: IntIntNullTextInputFormat problem
Date Tue, 04 Feb 2014 06:10:33 GMT
Hi Alexander,

the only inputformat that we have which is compatible with 
SimpleShortestPathsComputation is the

org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat

which is used in

org.apache.giraph.examples.SimpleShortestPathsComputationTest#testToyData

It's not as nice to use as the other inputformats we have, so I opened 
https://issues.apache.org/jira/browse/GIRAPH-844 to ensure that we have 
a nice easy to use TextInputFormat for the shortest paths example.

--sebastian

On 02/03/2014 06:52 PM, Alexander Frolov wrote:
> It seems that I understood the problem. PageRank and
> SingleSourceShortestPath both need weighted graph to compute.  I just
> missed it somehow.
> Thank you for help.
>
>
> On Mon, Feb 3, 2014 at 9:25 PM, Alexander Frolov
> <alexndr.frolov@gmail.com>wrote:
>
>>
>>
>>
>> On Mon, Feb 3, 2014 at 9:06 PM, Sebastian Schelter <ssc@apache.org> wrote:
>>
>>> If you just wanna play with Giraph, I suggest you start with the PageRank
>>> example:
>>>
>>> https://giraph.apache.org/pagerank.html
>>
>>
>> I have seen it.  I think there is no much difference in SSSP and PageRank
>> examples. By the way, I have not found LongDoubleNullTextInputFormat in my
>> sources (I am using release-1.0.0-RC2-152-g585511f) which is mentioned
>> there.
>> The same issue with SimplePageRankComputation.
>>
>> [hduser@localhost giraph]$ $HADOOP_HOME/bin/hadoop jar
>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimplePageRankComputation -vif
>> org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip
>> /user/hduser/input/test -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/hduser/output/amazon-shortestpaths -w 1
>> 14/02/03 21:24:16 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 14/02/03 21:24:16 INFO utils.ConfigurationUtils: No edge output format
>> specified. Ensure your OutputFormat does not require one.
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> checkClassTypes: vertex value types not assignable, computation - class
>> org.apache.hadoop.io.DoubleWritable, VertexInputFormat - class
>> org.apache.hadoop.io.LongWritable
>>   at
>> org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(GiraphConfigurationValidator.java:381)
>> at
>> org.apache.giraph.job.GiraphConfigurationValidator.verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.java:228)
>>   at
>> org.apache.giraph.job.GiraphConfigurationValidator.validateConfiguration(GiraphConfigurationValidator.java:141)
>> at
>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:214)
>>   at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>   at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>>
>>>
>>> --sebastian
>>>
>>>
>>> On 02/03/2014 05:49 PM, Alexander Frolov wrote:
>>>
>>>> I've tried .LongLongNullTextInputFormat and got this:
>>>>
>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop jar
>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>> SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip
>>>> /user/hduser/input/test -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/hduser/output/amazon-shortestpaths -w 1
>>>> 14/02/03 20:48:41 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 14/02/03 20:48:41 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>> checkClassTypes: vertex value types not assignable, computation - class
>>>> org.apache.hadoop.io.DoubleWritable, VertexInputFormat - class
>>>> org.apache.hadoop.io.LongWritable
>>>> at
>>>> org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(
>>>> GiraphConfigurationValidator.java:381)
>>>> at
>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>> verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.
>>>> java:228)
>>>> at
>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>> validateConfiguration(GiraphConfigurationValidator.java:141)
>>>> at
>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(
>>>> ConfigurationUtils.java:214)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>> NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>> DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Feb 3, 2014 at 8:38 PM, Alexander Frolov
>>>> <alexndr.frolov@gmail.com>wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 3, 2014 at 8:30 PM, Sebastian Schelter <ssc@apache.org>
>>>>> wrote
>>>>>
>>>>>   What types does the vertex class that you use in your computation
>>>>>> require?
>>>>>>
>>>>>
>>>>>
>>>>> I see.
>>>>>
>>>>> public class SimpleShortestPathsComputation extends BasicComputation<
>>>>>       LongWritable, DoubleWritable, FloatWritable, DoubleWritable>
{..}
>>>>>
>>>>> So that is a mistake.
>>>>> Thank you very much.
>>>>>
>>>>>
>>>>   Is there any means to import data from file like this (amazon0601.txt
>>>>> from
>>>>> SNAP data collection):
>>>>>
>>>>> 0   1
>>>>> 0   2
>>>>> 0   3
>>>>> 0   4
>>>>> 0   5
>>>>> 0   6
>>>>> 0   7
>>>>> 0   8
>>>>> 0   9
>>>>> 0   10
>>>>> ...
>>>>>
>>>>> and to perform basic tests from giraph-examples? Most of the tests are
>>>>> using LongWritable for vertex id.
>>>>>
>>>>>
>>>>>
>>>>>> On 02/03/2014 05:28 PM, Alexander Frolov wrote:
>>>>>>
>>>>>>   On Mon, Feb 3, 2014 at 8:20 PM, Sebastian Schelter <ssc@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>    Hi Alexander,
>>>>>>>
>>>>>>>>
>>>>>>>> what do you use as type for your vertex ids? It looks that
you are
>>>>>>>> trying
>>>>>>>> to use longs, while IntIntNullTextInputFormat only provides
ints,
>>>>>>>> that
>>>>>>>> could be the error.
>>>>>>>>
>>>>>>>> --sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>>    I also thought so, but I could not understand why it regard
it as
>>>>>>>> long.
>>>>>>>>
>>>>>>> I just use  -vif org.apache.giraph.io.formats.
>>>>>>> IntIntNullTextInputFormat
>>>>>>> -vip /user/hduser/input/test -vof
>>>>>>>
>>>>>>> and file consists only of two lines:
>>>>>>>
>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop dfs -cat
>>>>>>> /user/hduser/input/test
>>>>>>> 0 1
>>>>>>> 1 0
>>>>>>>
>>>>>>> I dont know any other means to specify type of vertex ids.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>   On 02/03/2014 05:06 PM, Alexander Frolov wrote:
>>>>>>>>
>>>>>>>>    Hello,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am trying to load data in from file in IntIntNullTextInputFormat.
>>>>>>>>> But I
>>>>>>>>> get is the following message:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop jar
>>>>>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>>>>>>> SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
-vif
>>>>>>>>> org.apache.giraph.io.formats.IntIntNullTextInputFormat
-vip
>>>>>>>>> /user/hduser/input/test -vof
>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op
>>>>>>>>> /user/hduser/output/amazon-shortestpaths -w 1
>>>>>>>>>
>>>>>>>>> 14/02/03 19:46:08 INFO utils.ConfigurationUtils: No edge
input
>>>>>>>>> format
>>>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>>>> 14/02/03 19:46:08 INFO utils.ConfigurationUtils: No edge
output
>>>>>>>>> format
>>>>>>>>> specified. Ensure your OutputFormat does not require
one.
>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>>> checkClassTypes: vertex index types not assignable, computation
-
>>>>>>>>> class
>>>>>>>>> org.apache.hadoop.io.LongWritable, VertexInputFormat
- class
>>>>>>>>> org.apache.hadoop.io.IntWritable
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(
>>>>>>>>> GiraphConfigurationValidator.java:381)
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>>>>>> verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.
>>>>>>>>> java:226)
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>>>>>> validateConfiguration(
>>>>>>>>> GiraphConfigurationValidator.java:141)
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(
>>>>>>>>> ConfigurationUtils.java:214)
>>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>> at
>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>>> NativeMethodAccessorImpl.java:57)
>>>>>>>>> at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The input file is here (just for testing):
>>>>>>>>>
>>>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop dfs -cat
>>>>>>>>> /user/hduser/input/test
>>>>>>>>> 0 1
>>>>>>>>> 1 0
>>>>>>>>>
>>>>>>>>> Could you please help with this issue?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>       Alex
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


Mime
View raw message