giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Frolov <alexndr.fro...@gmail.com>
Subject Re: IntIntNullTextInputFormat problem
Date Tue, 04 Feb 2014 08:25:57 GMT
Sebastian,

thank you very much for your help!


On Tue, Feb 4, 2014 at 10:10 AM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi Alexander,
>
> the only inputformat that we have which is compatible with
> SimpleShortestPathsComputation is the
>
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>
> which is used in
>
> org.apache.giraph.examples.SimpleShortestPathsComputationTest#testToyData
>
> It's not as nice to use as the other inputformats we have, so I opened
> https://issues.apache.org/jira/browse/GIRAPH-844 to ensure that we have a
> nice easy to use TextInputFormat for the shortest paths example.
>
> --sebastian
>
>
> On 02/03/2014 06:52 PM, Alexander Frolov wrote:
>
>> It seems that I understood the problem. PageRank and
>> SingleSourceShortestPath both need weighted graph to compute.  I just
>> missed it somehow.
>> Thank you for help.
>>
>>
>> On Mon, Feb 3, 2014 at 9:25 PM, Alexander Frolov
>> <alexndr.frolov@gmail.com>wrote:
>>
>>
>>>
>>>
>>> On Mon, Feb 3, 2014 at 9:06 PM, Sebastian Schelter <ssc@apache.org>
>>> wrote:
>>>
>>>  If you just wanna play with Giraph, I suggest you start with the
>>>> PageRank
>>>> example:
>>>>
>>>> https://giraph.apache.org/pagerank.html
>>>>
>>>
>>>
>>> I have seen it.  I think there is no much difference in SSSP and PageRank
>>> examples. By the way, I have not found LongDoubleNullTextInputFormat in
>>> my
>>> sources (I am using release-1.0.0-RC2-152-g585511f) which is mentioned
>>> there.
>>> The same issue with SimplePageRankComputation.
>>>
>>> [hduser@localhost giraph]$ $HADOOP_HOME/bin/hadoop jar
>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>> SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimplePageRankComputation -vif
>>> org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip
>>> /user/hduser/input/test -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/hduser/output/amazon-shortestpaths -w 1
>>> 14/02/03 21:24:16 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>> 14/02/03 21:24:16 INFO utils.ConfigurationUtils: No edge output format
>>> specified. Ensure your OutputFormat does not require one.
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>> checkClassTypes: vertex value types not assignable, computation - class
>>> org.apache.hadoop.io.DoubleWritable, VertexInputFormat - class
>>> org.apache.hadoop.io.LongWritable
>>>   at
>>> org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(
>>> GiraphConfigurationValidator.java:381)
>>> at
>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>> verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.
>>> java:228)
>>>   at
>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>> validateConfiguration(GiraphConfigurationValidator.java:141)
>>> at
>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(
>>> ConfigurationUtils.java:214)
>>>   at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>> NativeMethodAccessorImpl.java:57)
>>>   at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>>
>>>
>>>> --sebastian
>>>>
>>>>
>>>> On 02/03/2014 05:49 PM, Alexander Frolov wrote:
>>>>
>>>>  I've tried .LongLongNullTextInputFormat and got this:
>>>>>
>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop jar
>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>>> SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner
>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>> org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip
>>>>> /user/hduser/input/test -vof
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>> /user/hduser/output/amazon-shortestpaths -w 1
>>>>> 14/02/03 20:48:41 INFO utils.ConfigurationUtils: No edge input format
>>>>> specified. Ensure your InputFormat does not require one.
>>>>> 14/02/03 20:48:41 INFO utils.ConfigurationUtils: No edge output format
>>>>> specified. Ensure your OutputFormat does not require one.
>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>> checkClassTypes: vertex value types not assignable, computation - class
>>>>> org.apache.hadoop.io.DoubleWritable, VertexInputFormat - class
>>>>> org.apache.hadoop.io.LongWritable
>>>>> at
>>>>> org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(
>>>>> GiraphConfigurationValidator.java:381)
>>>>> at
>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>> verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.
>>>>> java:228)
>>>>> at
>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>> validateConfiguration(GiraphConfigurationValidator.java:141)
>>>>>
>>>>> at
>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(
>>>>> ConfigurationUtils.java:214)
>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>> NativeMethodAccessorImpl.java:57)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 3, 2014 at 8:38 PM, Alexander Frolov
>>>>> <alexndr.frolov@gmail.com>wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Mon, Feb 3, 2014 at 8:30 PM, Sebastian Schelter <ssc@apache.org>
>>>>>> wrote
>>>>>>
>>>>>>   What types does the vertex class that you use in your computation
>>>>>>
>>>>>>> require?
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I see.
>>>>>>
>>>>>> public class SimpleShortestPathsComputation extends BasicComputation<
>>>>>>       LongWritable, DoubleWritable, FloatWritable, DoubleWritable>
>>>>>> {..}
>>>>>>
>>>>>> So that is a mistake.
>>>>>> Thank you very much.
>>>>>>
>>>>>>
>>>>>>    Is there any means to import data from file like this
>>>>> (amazon0601.txt
>>>>>
>>>>>> from
>>>>>> SNAP data collection):
>>>>>>
>>>>>> 0   1
>>>>>> 0   2
>>>>>> 0   3
>>>>>> 0   4
>>>>>> 0   5
>>>>>> 0   6
>>>>>> 0   7
>>>>>> 0   8
>>>>>> 0   9
>>>>>> 0   10
>>>>>> ...
>>>>>>
>>>>>> and to perform basic tests from giraph-examples? Most of the tests
are
>>>>>> using LongWritable for vertex id.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  On 02/03/2014 05:28 PM, Alexander Frolov wrote:
>>>>>>>
>>>>>>>   On Mon, Feb 3, 2014 at 8:20 PM, Sebastian Schelter <ssc@apache.org
>>>>>>> >
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>    Hi Alexander,
>>>>>>>>
>>>>>>>>
>>>>>>>>> what do you use as type for your vertex ids? It looks
that you are
>>>>>>>>> trying
>>>>>>>>> to use longs, while IntIntNullTextInputFormat only provides
ints,
>>>>>>>>> that
>>>>>>>>> could be the error.
>>>>>>>>>
>>>>>>>>> --sebastian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    I also thought so, but I could not understand why
it regard it
>>>>>>>>> as
>>>>>>>>> long.
>>>>>>>>>
>>>>>>>>>  I just use  -vif org.apache.giraph.io.formats.
>>>>>>>> IntIntNullTextInputFormat
>>>>>>>> -vip /user/hduser/input/test -vof
>>>>>>>>
>>>>>>>> and file consists only of two lines:
>>>>>>>>
>>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop dfs -cat
>>>>>>>> /user/hduser/input/test
>>>>>>>> 0 1
>>>>>>>> 1 0
>>>>>>>>
>>>>>>>> I dont know any other means to specify type of vertex ids.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>   On 02/03/2014 05:06 PM, Alexander Frolov wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>    Hello,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I am trying to load data in from file in
>>>>>>>>>> IntIntNullTextInputFormat.
>>>>>>>>>> But I
>>>>>>>>>> get is the following message:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop jar
>>>>>>>>>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-
>>>>>>>>>> SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
-vif
>>>>>>>>>> org.apache.giraph.io.formats.IntIntNullTextInputFormat
-vip
>>>>>>>>>> /user/hduser/input/test -vof
>>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op
>>>>>>>>>> /user/hduser/output/amazon-shortestpaths -w 1
>>>>>>>>>>
>>>>>>>>>> 14/02/03 19:46:08 INFO utils.ConfigurationUtils:
No edge input
>>>>>>>>>> format
>>>>>>>>>> specified. Ensure your InputFormat does not require
one.
>>>>>>>>>> 14/02/03 19:46:08 INFO utils.ConfigurationUtils:
No edge output
>>>>>>>>>> format
>>>>>>>>>> specified. Ensure your OutputFormat does not require
one.
>>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>>>> checkClassTypes: vertex index types not assignable,
computation -
>>>>>>>>>> class
>>>>>>>>>> org.apache.hadoop.io.LongWritable, VertexInputFormat
- class
>>>>>>>>>> org.apache.hadoop.io.IntWritable
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>>>>>>> checkAssignable(
>>>>>>>>>> GiraphConfigurationValidator.java:381)
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>>>>>>> verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.
>>>>>>>>>> java:226)
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.job.GiraphConfigurationValidator.
>>>>>>>>>> validateConfiguration(
>>>>>>>>>> GiraphConfigurationValidator.java:141)
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(
>>>>>>>>>> ConfigurationUtils.java:214)
>>>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>>>> NativeMethodAccessorImpl.java:57)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The input file is here (just for testing):
>>>>>>>>>>
>>>>>>>>>> [hduser@localhost ~]$ $HADOOP_HOME/bin/hadoop dfs
-cat
>>>>>>>>>> /user/hduser/input/test
>>>>>>>>>> 0 1
>>>>>>>>>> 1 0
>>>>>>>>>>
>>>>>>>>>> Could you please help with this issue?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>       Alex
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message