giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Han JU <ju.han.fe...@gmail.com>
Subject What if the resulting graph is larger than the memory?
Date Fri, 17 May 2013 15:38:21 GMT
Hi,

It's me again.
After a day's work I've coded a Giraph solution for my problem at hand. I
gave it a run on a medium dataset and it's notably faster than other
approaches.

However the goal is to process larger inputs, for example I've a larger
dataset that the result graph is about 400GB when represented in edge
format and in text file. And I think the edges that the algorithm created
all reside in the cluster's memory. So it means that for this big dataset,
I need a cluster with ~ 400GB main memory to run? Is there any
possibilities that I can output "on the go" that means I don't need to
construct the whole graph, an edge is outputed to HDFS immediately instead
of being created in main memory then be outputed?

Thanks!
-- 
*JU Han*

Software Engineer Intern @ KXEN Inc.
UTC   -  Université de Technologie de Compiègne
*     **GI06 - Fouille de Données et Décisionnel*

+33 0619608888

Mime
View raw message