hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: How to monitor performance metrics?
Date Thu, 28 Mar 2013 01:33:19 GMT
P.S., you might want to file this on jira and assign it to me or
yourself. if you need some help, I can help you. ;)

On Thu, Mar 28, 2013 at 10:27 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> The cost of a superstep described in Wikipedia[1] is as below:
>
> The cost of a superstep S = max(Wi) + max(hg) + l = max cost of the
> local computation + global communication + cost of the barrier
> synchronization. As you already might know, max(Wi) means just the
> wait time for the slowest process.
>
>> III - In relation to the cost of computation, is there any way to implement on
>> Apache Hama?
>
> Sure, you can calculate the cost of each superstep using Counters, and
> can sum finally. With this, you can also add something like "estimated
> time remaining". Please see our implementation of sync() method[2]
> (especially PeerCounter.TIME_IN_SYNC_MS).
>
> 1. http://en.wikipedia.org/wiki/Bulk_synchronous_parallel
> 2. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java
>
> On Wed, Mar 27, 2013 at 11:26 PM, Júlio Pires <juliocspires@gmail.com> wrote:
>> Hi Suraj,
>>
>> Thanks for the response.
>> I've been by analyzing and I got the following doubts:
>>
>> I - How can I know when a super step ended?
>> II - With the information of finish of a superstep, it is possible to
>> capture the costs discussed in the previous email (computing, communication
>> and synchronization)?
>> III - In relation to the cost of computation, is there any way to implement on
>> Apache Hama?
>>
>> Thanks!
>> Júlio
>>
>>
>> 2013/3/16 Suraj Menon <surajsmenon@apache.org>
>>
>>> Hi Julio,
>>>
>>> - Cost of the longest running local computation;
>>> We don't have this yet per superstep.
>>>
>>> - Cost of global communication between the processors;
>>> We provide information on count/size in bytes of messages sent and received
>>> by peers.( IO_BYTES_READ, MESSAGE_BYTES_TRANSFERED, MESSAGE_BYTES_RECEIVED,
>>> TOTAL_MESSAGES_SENT, TOTAL_MESSAGES_RECEIVED, COMPRESSED_BYTES_SENT,
>>> COMPRESSED_BYTES_RECEIVED)
>>> - Cost of the barrier synchronisation at the end of the superstep.
>>> This is again provided in the final output in TIME_IN_SYNC_MS.
>>>
>>> The stats above are aggregated throughout the lifecycle and we don't have a
>>> breakdown per superstep yet. You may create a JIRA for tracking.
>>>
>>> Thanks,
>>> Suraj
>>>
>>>
>>> On Fri, Mar 15, 2013 at 8:10 PM, Edward J. Yoon <edwardyoon@apache.org
>>> >wrote:
>>>
>>> > Hi, of course.
>>> >
>>> > On Fri, Mar 15, 2013 at 9:39 PM, Júlio Pires <juliocspires@gmail.com>
>>> > wrote:
>>> > > Hi,
>>> > >
>>> > > Thanks for the response.
>>> > >
>>> > > So, thinking in a more abstract level, there are means to achieve these
>>> > > three cost information on Apache Hama (another type of information)
>>> (see
>>> > >
>>> >
>>> http://en.wikipedia.org/wiki/Bulk_synchronous_parallelThe_Cost_of_a_BSP_algorithm
>>> > )?
>>> > >
>>> > > - Cost of the longest running local computation;
>>> > > - Cost of global communication between the processors;
>>> > > - Cost of the barrier synchronisation at the end of the superstep.
The
>>> > cost
>>> > > of one superstep for p processors
>>> > >
>>> > > Thanks!
>>> > >
>>> > >
>>> > > 2013/3/14 Edward J. Yoon <edwardyoon@apache.org>
>>> > >
>>> > >> Unfortunately, we currently don't have any monitoring tool for
metrics
>>> > >> such as CPU, memory.
>>> > >>
>>> > >> For the cluster and BSP job monitoring, there's a web UI but not
>>> mature
>>> > >> yet.
>>> > >>
>>> > >> On Fri, Mar 15, 2013 at 2:17 AM, Júlio Pires <juliocspires@gmail.com>
>>> > >> wrote:
>>> > >> > Hi,
>>> > >> >
>>> > >> > It's possible to retrieve information of performance such
as CPU
>>> > usage or
>>> > >> > memory in each task on each peer? If so, how do?
>>> > >> >
>>> > >> > Also, details of the functioning of job monitoring in Apache
Hama.
>>> > >> >
>>> > >> > Thanks!
>>> > >> > Júlio
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Best Regards, Edward J. Yoon
>>> > >> @eddieyoon
>>> > >>
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>> >
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message