hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John George (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Date Mon, 14 Nov 2011 23:23:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150072#comment-13150072

John George commented on HADOOP-7821:

There has been several JIRAs that talk about 'messaging type' service that could be utilized
by Hadoop clients.
- HDFS-1742 talks about a system that could notify interested clients about major HDFS events
(like file creation, deletion, etc).
- HADOOP-5640 talks about a system that could notify clients on certain Namenode events (like
entering safemode, exiting safemode etc).
- OOZIE-599 talks about Hadoop supporting JMS based (eg: Apache ActiveMQ) messaging service
as an alternative to obtain 'call-back' URLs.
- Some other examples that could utilize a messaging service are:
   -- events/states/status specific to an application (eg: mapper percentage complete)
   -- events/states/status common to any application using YARN

Some high level requirements for creating such a messaging service:
- Provide timely (and/or periodic) state/status/statistics updates to subscribed users.
- Provide the capability to let users specify 'interesting events'.
- Provide a 'customizable' and 'configurable' interface such that user-defined parameters
can also be 'subscribed' by the user. 
- Provide a pluggable security interface.

Such a messaging service will not only reduce the load on Hadoop
components like NN (reduced polling by clients), but also increase the
productivity of developers using hadoop.

Of course, there are challenges as well and this JIRA is filed to provide a platform for discussion
about this feature and possibly act as a parent JIRA to implement a Hadoop Messaging Service.
> Hadoop Messaging Service
> ------------------------
>                 Key: HADOOP-7821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7821
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.24.0
>            Reporter: John George
> It will be good if Hadoop supports some sort of a messaging service that allows users
to subscribe.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message