giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavan Kumar A <pava...@outlook.com>
Subject RE: Using a custom graph partitioning stratergy with giraph
Date Sun, 28 Sep 2014 15:29:32 GMT
I worked on this feature sometime back - but I only worked on inputting hive file & not
hdfs
You can use logic outside giraph to select which partition file to use - this is possible
because you input the number of workers anyway.For instance in the script that you use to
launch a giraph job have a selection logic for the partition file
You can take a look at : https://issues.apache.org/jira/browse/GIRAPH-908You might have to
extend upon the jira for your specific use case - I only added support for case when id =
longwritable
Here is a list of options you might want to explore
 # Mapping Store related information    -Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongByteMappingStore
   -Dgiraph.lbMappingStoreUpper=1987000    -Dgiraph.lbMappingStoreLower=4096    # Mapping
tore ops information    -Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultEmbeddedLongByteOps
   # Embed mapping information    -Dgiraph.edgeTranslationClass=org.apache.giraph.mapping.translate.LongByteTranslateEdge
   # PartitionerFactory to be used    -Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongMappingStorePartitionerFactory
So the partition map is stored here as map of byte arrays. with .lbMappingStoreUpper being
size of map and lbMappingStoreLower being size of individual arrays
Please explore code & tell me what else you need.ThanksDate: Sat, 27 Sep 2014 22:51:29
-0700
Subject: Re: Using a custom graph partitioning stratergy with giraph
From: charith.dhanushka@gmail.com
To: user@giraph.apache.org

Also adding some more information. 
My current understanding is I should be able to do this by  my own org.apache.giraph.partition.WorkerGraphPartitioner
implementation.
But my question is, Is there are a way to get some outside input inside the WorkerGraphPartitioner?In
my case it will be an hdfs file location. 

Thanks,Charith







 
On Sat, Sep 27, 2014 at 10:13 PM, Charith Wickramarachchi <charith.dhanushka@gmail.com>
wrote:
Hi, 
I m trying to use giraph with a custom graph partitioner that I have. In my case i want to
assign vertices to workers based on a custom partitioner input. 
In my case partitioner will take number of workers as an input parameter and give me a file
which maps each vertex id to a worker. I m trying load this file to a hdfs location and use
it as an input to the giraph and do the vertex assignment.  
Any suggestions or pointers on best way to this will be highly appricated (Use the current
extention points of giraph as much as possible to avoid random hacks). 
I m currently using giraph-1.0.0.
Thanks,Charith



-- 
Charith Dhanushka Wickramaarachchi
Tel  +1 213 447 4253Web  http://apache.org/~charithBlog  http://charith.wickramaarachchi.org/Twitter
 @charithwiki
This communication may contain privileged or other confidential information and is intended
exclusively for the addressee/s. If you are not the intended recipient/s, or believe that
you may havereceived this communication in error, please reply to the sender indicating that
fact and delete the copy you received and in addition, you should not print, copy, retransmit,
disseminate, or otherwise use the information contained in this communication. Internet communications
cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept
liability for any errors or omissions




-- 
Charith Dhanushka Wickramaarachchi
Tel  +1 213 447 4253Web  http://apache.org/~charithBlog  http://charith.wickramaarachchi.org/Twitter
 @charithwiki
This communication may contain privileged or other confidential information and is intended
exclusively for the addressee/s. If you are not the intended recipient/s, or believe that
you may havereceived this communication in error, please reply to the sender indicating that
fact and delete the copy you received and in addition, you should not print, copy, retransmit,
disseminate, or otherwise use the information contained in this communication. Internet communications
cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept
liability for any errors or omissions

 		 	   		  
Mime
View raw message