incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [Hama Wiki] Trivial Update of "Architecture" by Edward J. Yoon
Date Tue, 11 May 2010 02:19:27 GMT
In this mailing
(http://markmail.org/search/?q=hama-dev+discuss#query:hama-dev%20discuss+page:1+mid:amlvccbptom3yro3+state:results)
and others, we discussed about:

1) We can't implement all of matrix algebra. 2) Even if we do that,
the code complexity based on M/R model is too high. As we learned, the
M/R model isn't fit well for the complex numerical/relational data
processing required for many communications between each others. 3)
Admittedly, BSP is flexible than M/R.

Therefore, We've aligned our goal: to provide easy *matrix/graph
computing* programming environment on the distributed system (hadoop),
such as M/R framework. (NOT library) So, we won't develop only the BSP
framework if there is other solutions.

On Tue, May 11, 2010 at 10:29 AM, Hyunsik Choi <hyunsik.choi@gmail.com> wrote:
> According to you,  the main objective of hama appears to be developing only
> the BSP framework. Is it right?
>
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea Univ.
> http://diveintodata.org
>
>
> On Mon, May 10, 2010 at 8:34 PM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>
>> > In the distant future, the graph package would use both MR and BSP. But,
>>
>> Where will M/R use for?
>>
>> I'd like to cut M/R out of HAMA project and leave only the pure BSP &
>> Graph computing framework as simple as possible. Then, the rest are
>> user's matrix/graph applications.
>>
>> And, I'd like to consider about building an contributor/developer
>> ECOsystem.
>>
>> > now i'll try to develop the graph package with only BSP. How about the
>> > matrix package?
>>
>> As mentioned above, I'd like to show only some example of how you
>> implement your own matrix/graph applications using BSP.
>>
>> I think the block of hadoop families should be added on diagram only
>> to help understand relations between hadoop and hama,
>>
>> On Mon, May 10, 2010 at 7:40 PM, Hyunsik Choi <hyunsik.choi@gmail.com>
>> wrote:
>> > In the distant future, the graph package would use both MR and BSP. But,
>> > now i'll try to develop the graph package with only BSP. How about the
>> > matrix package?
>> >
>> > --
>> > Hyunsik Choi
>> > Database & Information Systems Group, Korea Univ.
>> > http://diveintodata.org
>> >
>> >
>> > On Mon, May 10, 2010 at 7:17 PM, Edward J. Yoon <edwardyoon@apache.org
>> >wrote:
>> >
>> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
>> to
>> >> > external. It is only used inside the BSP.
>> >>
>> >> Right you are. And also, the top of M/R and BSP should be clarified,
>> >> but currently something is odd overall.
>> >>
>> >> On Mon, May 10, 2010 at 6:24 PM, Hyunsik Choi <hyunsik.choi@gmail.com>
>> >> wrote:
>> >> > Good job! But, the matrix package is omitted. Besides, I think that
>> >> > zookeeper should be enclosed in BSP since zookeeper is not disclosed
>> to
>> >> > external. It is only used inside the BSP.
>> >> >
>> >> > Best regards,
>> >> > --
>> >> > Hyunsik Choi
>> >> > Database & Information Systems Group, Korea Univ.
>> >> > http://diveintodata.org
>> >> >
>> >> >
>> >> > On Mon, May 10, 2010 at 3:01 PM, Apache Wiki <wikidiffs@apache.org>
>> >> wrote:
>> >> >
>> >> >> Dear Wiki user,
>> >> >>
>> >> >> You have subscribed to a wiki page or wiki category on "Hama Wiki"
>> for
>> >> >> change notification.
>> >> >>
>> >> >> The "Architecture" page has been changed by Edward J. Yoon.
>> >> >> http://wiki.apache.org/hama/Architecture?action=diff&rev1=93&rev2=94
>> >> >>
>> >> >> --------------------------------------------------
>> >> >>
>> >> >>
>> >> >>  This effort is still a "work in progress". Please feel free to
add
>> >> >> comments.
>> >> >>  ----
>> >> >> +
>> >> >> + == The overall architecture of HAMA ==
>> >> >> + {{{
>> >> >> +      +-----------------------------------------------+
>> >> >> +      |                  HAMA APIs          
         |
>> >> >> +      +-----------------------------------------------+
>> >> >> +   --------------------------------------------------------
>> >> >> +                          +---------------------------+
>> >> >> +                          | Graph Computing Framework
|
>> >> >> +                          +---------------------------+
  Computing
>> >> Engine
>> >> >> +      +-----------+ +---------------------------------+
>> >> >> +      | MapReduce | | BSP (Bulk Synchronous Parallel) |
>> >> >> +      +-----------+ +---------------------------------+
>> >> >> +   --------------------------------------------------------
>> >> >> +      +-----------------------------------------------+
>> >> >> +      |                  Zookeeper          
         |
>> Distributed
>> >> >> Locking Service
>> >> >> +      +-----------------------------------------------+
>> >> >> +   --------------------------------------------------------
>> >> >> +      +-----------------------------------------------+
>> >> >> +      |                                  
            |
>> >> >> +      | +-----------------------------+              
|
>> >> >> +      | | HBase : Distributed Storage |              
|
>> >> >> +      | +-----------------------------+              
|
>> Distributed
>> >> >> Storage Systems
>> >> >> +      | +---------------------------------+          
|
>> >> >> +      | | HDFS : Distributed File System  |          
|
>> >> >> +      | +---------------------------------+          
|
>> >> >> +      +-----------------------------------------------+
>> >> >> + }}}
>> >> >> +
>> >> >>  == BSP ==
>> >> >>
>> >> >>  The BSP package is a implementation of BSP over Hadoop RPC(sockets).
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> edwardyoon@apache.org
>> >> http://blog.udanax.org
>> >>
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>



-- 
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message