Return-Path: X-Original-To: apmail-hama-dev-archive@www.apache.org Delivered-To: apmail-hama-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DEAF2E5BD for ; Thu, 31 Jan 2013 07:20:25 +0000 (UTC) Received: (qmail 69210 invoked by uid 500); 31 Jan 2013 07:20:25 -0000 Delivered-To: apmail-hama-dev-archive@hama.apache.org Received: (qmail 69076 invoked by uid 500); 31 Jan 2013 07:20:24 -0000 Mailing-List: contact dev-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hama.apache.org Delivered-To: mailing list dev@hama.apache.org Received: (qmail 69037 invoked by uid 99); 31 Jan 2013 07:20:23 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jan 2013 07:20:23 +0000 Received: from localhost (HELO mail-oa0-f43.google.com) (127.0.0.1) (smtp-auth username surajsmenon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jan 2013 07:20:22 +0000 Received: by mail-oa0-f43.google.com with SMTP id l10so2659222oag.2 for ; Wed, 30 Jan 2013 23:20:22 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.60.169.240 with SMTP id ah16mr6073572oec.9.1359616821954; Wed, 30 Jan 2013 23:20:21 -0800 (PST) Received: by 10.76.170.231 with HTTP; Wed, 30 Jan 2013 23:20:21 -0800 (PST) In-Reply-To: References: Date: Thu, 31 Jan 2013 02:20:21 -0500 Message-ID: Subject: Re: Discussion for memory and scalability issues of Graph package. From: Suraj Menon To: dev@hama.apache.org Content-Type: multipart/alternative; boundary=bcaec54a367ab409c004d490754e --bcaec54a367ab409c004d490754e Content-Type: text/plain; charset=ISO-8859-1 Thanks for bringing up our discussion online. For 1. Let's implement something withing bsp-core that could be re-used by graph package. [HAMA-724] For 2. For sorted queue, It would be expensive to do all the sorting on the sender side. We need to have a send protocol and the receive protocol (merge sort) [HAMA-722][HAMA-723] Regards, Suraj On Wed, Jan 30, 2013 at 3:05 AM, Edward J. Yoon wrote: > Hi devs, > > As you know, many people reports OOM problems with graph algorithms. > It is about handling messages. I roughly think that every vertex can > send or receive as many messages as the number of outgoing or incoming > links. For example, you know, Barack Obama has an 26,000,000+ > followers. > > I believe the issue of message queue will be fixed by adding spilling > queue. Another issue is the grouping messages by vertex ID[1]. To > solve this issue, I'm thinking about two ways: 1) Support grouping > function of key-value pair messages in BSP framework (like > Map/Reduce). 2) Write messages and Sort by vertex ID on local disk > (external merge sort). > > If you have any ideas or suggestions, Pls let me know. > > 1. https://issues.apache.org/jira/browse/HAMA-704 > > -- > Best Regards, Edward J. Yoon > @eddieyoon > --bcaec54a367ab409c004d490754e--