hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook callbacks into key service events
Date Mon, 14 Sep 2009 06:29:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754841#action_12754841
] 

Todd Lipcon commented on HADOOP-5640:
-------------------------------------

bq. Since Thrift RPC has replaced client-NN rpc, can't you do the same for the DN to NN rpc?


Currently we haven't replaced the Client-NN RPC with anything - for Java clients using DistributedFileSystem,
it continues to use Hadoop RPC and the BlockXceiverProtocol. The Thrift stuff is a second
option for clients not written in Java. Our particular use case for the Thrift interface is
mostly concerned with management and metadata, and less with making a fully functional and
performant client interface replacement.

bq. Look at the lifecycle Jira (HDFS-326) and explain if there is any overlap on the lifecycle
events.

Not to be too pessimistic, but that JIRA has been in the works for over a year and recently
got pushed back again so it won't be in 0.21. We'd like to have this in 0.21. If we go ahead
and mark this API as Unstable as you mentioned, I'd be happy to do the work at a later date
to cut it back out if HDFS-326 covers the use case.

bq. Explain how you are planning to deliver the events. 

The current patch should illustrate this. The event dispatch uses a separate thread to make
the calls on the plugins. Events are dispatched by submitting Runnables to this thread, which
is not blocking. This puts the synchronization onus on the receiver, but earlier in the discussion
on this JIRA we determined that was probably preferable to possibly allowing plugin code to
block daemon code.

bq. Mark the APIs as Unstable for now.

Will do. I'll upload a rebased patch on trunk with this change later this week.

> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin
implementations at NN/DN start/stop. However, this is insufficient integration for some common
use cases.
> We should add some functionality for Plugins to subscribe to events generated by the
service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync
up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin
interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257.
Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time,
and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message