giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ameya Vilankar <ameya.vilan...@gmail.com>
Subject Running Giraph with out of core messages and combiners at the same time
Date Mon, 28 Oct 2013 18:46:03 GMT
I have the following questions:
1. Can we use the following parameters:
-D giraph.useOutOfCoreMessages=true and -D
giraph.useBigDataIOForMessages=true at the same time. Can we use the unsafe
serialization and the out of core messaging at the same time?

2. Also whenever I use the -D giraph.useOutOfCoreMessages=true, the
combiners don't run while sending the messages. I need the combiners to run
in all the other steps where I am using them except for the similarity
computation step where I cannot combine the messages being sent. In my code
I just read the first message received in the message list because I know
that running the combiners will result in only one message being received
by all the vertices. But, running out of core causes the combiners not to
run and so I read only the first message and do the processing which is
wrong.

Is there any way to run the program with out of core messages and combiners
working on those messages.
I have to use run with out of core messages enabled because in the second
superstep I cannot combine the messages and I want to run this on 100
Million users and the 100 Million items.

Please Help me out.
Thanks in advance.
Ameya Vilankar
Software Engineer Intern
Zynga

Mime
View raw message