giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukas Nalezenec <lukas.naleze...@firma.seznam.cz>
Subject Re: using partition class
Date Thu, 03 Apr 2014 08:38:31 GMT
Hi
It depends on what you want to do with Giraph.
You don't necessary need to write custom PartitionerFactory. You would 
need to turn on PartitionBalancer but its broken. I think I have already 
found the bug, i will commit it soon. I you are really interested in 
balancing I am writing advanced PartitionBalancer with solver inside. I 
would appreciate if somebody would help me.

Regards,
Lukas


On 2.4.2014 18:24, Liannet Reyes wrote:
> Thanks again  for your help Lukas,
>
> It works!
>
> Hereby the full command line for future reference.
>
>>>     hadoop jar
>>>     $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>>>     org.apache.giraph.GiraphRunner-Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.SimpleLongRangePartitionerFactory
>>>     -Dgiraph.vertexKeySpaceSize=5org.apache.giraph.examples.SimpleShortestPathsComputation
>>>     -vif
>>>     org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>     -vip /user/hduser/input/tiny_graph.txt -vof
>>>     org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>     /user/hduser/output/shortestpaths-w 2
>>
>
> I guest the next step is to implement my custom PartitionerFactoryClass.
>
> Best Regards,
>
> Liannet
>
> 2014-04-02 17:32 GMT+02:00 Lukas Nalezenec 
> <lukas.nalezenec@firma.seznam.cz 
> <mailto:lukas.nalezenec@firma.seznam.cz>>:
>
>     Hi,
>
>     1. It have to work if you set giraph.vertexKeySpaceSize just after
>     or before the other property. But it can interfere with the other
>     command line options :(.
>     2. Its Number of Vertexes in your graph.
>
>     Regards,
>     Lukas
>
>
>
>     On 2.4.2014 17:21, Liannet Reyes wrote:
>>     Thank you very much Lukas,
>>
>>     I've notice that using SimpleLongRangePartitionerFactory require
>>     the GiraphConstant giraph.vertexKeySpaceSize to be set,
>>     but I have 2 questions:
>>
>>     1- Where should I set the giraph constant values? I've tried
>>     using -Dgiraph.vertexKeySpaceSize=256 after
>>     -Dgiraph.graphPartitionerFactoryClass but it didn't work
>>     2- Which unit is this PARTITION_VERTEX_KEY_SPACE_SIZE? Bytes??
>>
>>     Regards,
>>
>>     Liannet
>>
>>
>>
>>
>>     2014-04-01 20:55 GMT+02:00 Lukas Nalezenec
>>     <lukas.nalezenec@firma.seznam.cz
>>     <mailto:lukas.nalezenec@firma.seznam.cz>>:
>>
>>         Hi,
>>         Partition is for storing vertexes, Partitioner is for
>>         distributing vertexes between Partitions.
>>
>>         Try this:
>>         -Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.SimpleLongRangePartitionerFactory
>>
>>         Its good idea to switch Partition to ByteArrayPartition (or
>>         better).
>>
>>         Lukas
>>
>>
>>
>>         On 1.4.2014 19:59, Liannet Reyes wrote:
>>>         Hi all,
>>>
>>>         I´m trying to use the giraph partition class to improve my
>>>         graph distribution, however I hardly found info on how to
>>>         use it.
>>>         I´m testing the SimpleShortestPathsComputation example with
>>>         the tiny_graph.txt input file and the
>>>         SimpleLongRangePartitionerFactory out-of-the-box in
>>>         org.apache.giraph.partition but without success. Apparentely
>>>         I need a Partition class and the
>>>         SimpleLongRangePartitionerFactory is not.
>>>
>>>         Can you kindly please let me know the proper way to do it?
>>>         See below my command and console output
>>>
>>>         hadoop jar
>>>         $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>>>         org.apache.giraph.GiraphRunner
>>>         org.apache.giraph.examples.SimpleShortestPathsComputation
>>>         -vif
>>>         org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>         -vip /user/hduser/input/tiny_graph.txt -vof
>>>         org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>         /user/hduser/output/shortestpaths-pc
>>>         org.apache.giraph.partition.SimpleLongRangePartitionerFactory -w
>>>         2
>>>
>>>         Exception in thread "main" java.lang.RuntimeException: class
>>>         org.apache.giraph.partition.SimpleLongRangePartitionerFactory not
>>>         org.apache.giraph.partition.Partition
>>>         at
>>>         org.apache.hadoop.conf.Configuration.setClass(Configuration.java:915)
>>>         at
>>>         org.apache.giraph.conf.ClassConfOption.set(ClassConfOption.java:182)
>>>         at
>>>         org.apache.giraph.conf.GiraphConfiguration.setPartitionClass(GiraphConfiguration.java:594)
>>>         at
>>>         org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:375)
>>>         at
>>>         org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:207)
>>>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>>         sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>         at
>>>         sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>>         Thanks in advance
>>>
>>>         Liannet
>>
>>
>
>


Mime
View raw message