I will write a detailed explanat= ion in weekend. thanks for your interest.

Date: Wed=2C 1 Oct 2014 10:56:16 -0700
Subject: Re: Using a custom gr= aph partitioning stratergy with giraph
From: charith.dhanushka@gmail.com=
To: user@giraph.apache.org

Thanks Pavan=2C = =3B

I get the high level level idea. I am =3Bstill n= ew to Giraph code base so I am still trying to understand the overall desig= n. =3B

So I have few questions regarding this = feature. =3B

Can we use this feature with a ve= rtex input format without using edge translation? =3B
(Since =  =3B =3BgetPartition =3Bin MappingStoreOps =  =3Bcan be used to get the partition of any target vertex)

Also=2C since I have mapping information in a separate fil= e  =3Bdo I =3Bneed to embed =3Btarget information in the vertex= ?  =3B

It will be great if you could explain y= our scenario with dataformat =3Byou used and what ex= tension =3Bpoints you used so that I can understand it better and adapt= to my scenario. =3B

Thanks=2C<= /div>
Charith

On Mon=2C Sep 29=2C 2014 at 3:34 PM=2C Pavan Kumar A <=3Bpavanka@o= utlook.com>=3B wrote:
=0A= =0A= =0A=
we have two inputs - vertex &=3B edges
if we p= artition edges vertices based on a map=2C then when we want to send message= s we should be able to know which partition a vertex is on.

<= /div>
typically we send messages to targetIds of outgoing edges=2C edge= transation helps encode mapping information into targetIds=2C so knowing w= hich partition to send a message can be done by just looking at the targeti= d

Date: Mon=2C 29 Sep 2014 14:37:22 -0700

Subject: Re: Using a custom graph partitioning stratergy with giraphFrom: ch= arith.dhanushka@gmail.com
To: user@giraph.apache.org

Hi= Pavan=2C =3B

Thanks for the details. I went through= the code specially the extension =3Bpoints you mentioned. =3B
I am =3Bnot clear about the function of the edge Translation (org= .apache.giraph.mapping<= span>.translate.TranslateEdge) class. Could you please = explain the idea of this translation process. =3B

In my case I will have a mapping file which maps each vertex to a p= artition.
=3B
ex:

v1 part1
v2 part2
v3 part3 =3B
.
.
.

So I was thinking of passing= this as a parameter and reading inside my own MappingStore Implementation&= nbsp=3B
-Dgiraph.mappingFilePath=3D/use= r/charith/input/mapping.txt

Is there = a better approach? =3B

Thanks=2C
Charith

On= Sun=2C Sep 28=2C 2014 at 8:29 AM=2C Pavan Kumar A <=3B= pavanka@outlook.co= m>=3B wrote:
=0A= =0A= =0A=
I worked on this feature sometime back - but I only w= orked on inputting hive file &=3B not hdfs

You can use logi= c outside giraph to select which partition file to use - this is possible b= ecause you input the number of workers anyway.
For instance in th= e script that you use to launch a giraph job have a selection logic for the= partition file

You can take a look at : =3Bhttps://issues.apache.org/jira/browse/GIRAPH-908
You might = have to extend upon the jira for your specific use case - I only added supp= ort for case when id =3D longwritable

Here is a li= st of options you might want to explore

= =3B# Mapping Store related information
=3B  =3B -Dgiraph= .mappingStoreClass=3Dorg.apache.giraph.mapping.LongByteMappingStore
=3B  =3B -Dgiraph.lbMappingStoreUpper=3D1987000
= =3B  =3B -Dgiraph.lbMappingStoreLower=3D4096
=3B  = =3B # Mapping tore ops information
=3B  =3B -Dgiraph.map= pingStoreOpsClass=3Dorg.apache.giraph.mapping.DefaultEmbeddedLongByteOps
=3B  =3B # Embed mapping information
=3B &n= bsp=3B -Dgiraph.edgeTranslationClass=3Dorg.apache.giraph.mapping.translate.= LongByteTranslateEdge
=3B  =3B # PartitionerFactory to b= e used
=3B  =3B -Dgiraph.graphPartitionerFactoryClass=3D= org.apache.giraph.partition.LongMappingStorePartitionerFactory
So the partition map is stored here as map of byte arrays. wit= h =3B.lbMappingStoreUpper being size = of map and =3BlbMappingStoreLo= wer being size of individual arrays

Please explore code &=3B tell me what = else you need.
Thanks

Date: Sat=2C 27 Sep 2014 22:5= 1:29 -0700
Subject: Re: Using a custom graph partitioning stratergy with= giraph
From: charith.dhanushka@gmail.com
To: user@giraph.apache.org

=

<= div>My current understanding is I should be able to do this by  =3Bmy o= wn org.apache.giraph.partition.WorkerGraphPartitioner implementation.
=

But my question is=2C Is there are a way to get some ou= tside input inside the WorkerGraphPartitioner?In my case it will be an hdfs= file location. =3B

Thanks=2C
Ch= arith

=3B
=

On Sat=2C Sep 27=2C 2014 at 10:13 PM=2C Charith Wickram= arachchi <=3Bcharith.dhanushka@gmail.com>=3B wrote:<= br>
<= div dir=3D"ltr">Hi=2C =3B

I m trying to use giraph w= ith a custom graph partitioner that I have. In my case i want to assign ver= tices to workers based on a custom partitioner input. =3B
In my case partitioner will take number of workers as an input = parameter and give me a file which maps each vertex id to a worker. I m try= ing load this file to a hdfs location and use it as an input to the giraph = and do the vertex assignment.  =3B

Any suggest= ions or pointers on best way to this will be highly appricated (Use the cur= rent extention points of giraph as much as possible to avoid random hacks).=  =3B

I m currently using giraph-1.0.0.

Thanks=2C
Charith

-- <= br>
Charith Dhanushka Wickramaarachchi
Tel =3B =3B+1 213 447 4253

This communication may contain privilege= d or other confidential =3Binformation and is intended exclusively for the addressee/s. If you&n= bsp=3Bare not the intende= d recipient/s=2C or believe that you may have
received= this communication in error=2C please reply to the sender =3Bindicating that fact and de= lete the copy you received and in addition=2C =3Byou should not print=2C copy=2C retransmit=2C d= isseminate=2C or otherwise use =3Bthe information contained in this communication. Internet = =3Bcommunications cannot = be guaranteed to be timely=2C secure=2C error or =3Bvirus-free. The sender does not accept liab= ility for any errors or =3Bomissions
=0A=
=0A=

--
Charith Dhanushka Wickramaarachchi

Tel =3B&n= bsp=3B+1 213 447= 4253

<= /div>
This communication may contain privileged or other confid= ential =3Binformation= and is intended exclusively for the addressee/s. If you =3Bare not the intended recipient/s=2C = or believe that you may have
received this comm= unication in error=2C please reply to the sender =3Bindicating that fact and delete the copy yo= u received and in addition=2C =3Byou should not print=2C copy=2C retransmit=2C disseminate=2C or= otherwise use =3Bthe= information contained in this communication. Internet =3Bcommunications cannot be guaranteed to= be timely=2C secure=2C error or =3Bvirus-free. The sender does not accept liability for any er= rors or =3Bomissions<= /span>
=0A=
=0A=

--
Charith Dhanushka Wickramaarachchi

Tel =3B&n= bsp=3B+1 213 447= 4253

<= /div>
This communication may contain privileged or other confid= ential =3Binformation= and is intended exclusively for the addressee/s. If you =3Bare not the intended recipient/s=2C = or believe that you may have
received this comm= unication in error=2C please reply to the sender =3Bindicating that fact and delete the copy yo= u received and in addition=2C =3Byou should not print=2C copy=2C retransmit=2C disseminate=2C or= otherwise use =3Bthe= information contained in this communication. Internet =3Bcommunications cannot be guaranteed to= be timely=2C secure=2C error or =3Bvirus-free. The sender does not accept liability for any er= rors or =3Bomissions<= /span>
=0A=
=0A=

--
Charith Dhanushka Wickramaarachchi

Tel =3B&n= bsp=3B+1 213 447= 4253