hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-7590) Add a costless notifications mechanism from master to regionservers & clients
Date Mon, 28 Jan 2013 15:39:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564331#comment-13564331
] 

nkeywal edited comment on HBASE-7590 at 1/28/13 3:38 PM:
---------------------------------------------------------

Sorry, I missed your comment.
- znodes+watches was initially my preferred target, but it's not ideal today, we need at least
ZOOKEEPER-1147. 
There are as well differences between the contract we need and the ZK features: we accept
to miss events and we don't need a synchronisation between servers (a single master sending
its view of the world is enough), we don't want to monitor the client. Additionally, we don't
need to write, but that's what ZOOKEEPER-1147 is about. My feeling is that waiting for ZK
can take a long time...

- I read the doc, I'm unclear of the cost. They say it's replaced by a broadcast for the switches
(not the routers) when they don't support a specific protocol. While I guess that it may be
challenging for systems with thousands of messages per second or hundred of thousands of client
application, here it seems quite simple: a maximum number of client in the order of a few
thousands + 1 message every 10 seconds or so.

I don't mind beeing proven wrong here. I've used such mechanisms (with actually much more
messages) in the past, but all the datacenters guys were quite used to this type of architecture
so they had no questions on it and I didn't dig inside the operational impact.

Anyway, there is no doubt it should be an optional mechanism.

                
      was (Author: nkeywal):
    Sorry, I didn't see your comment.
- znodes+watches was initially my preferred target, but it's not ideal today, we need at least
ZOOKEEPER-1147. 
There are as well differences between the contract we need and the ZK features: we accept
to miss events and we don't need a synchronisation between servers (a single master sending
its view of the world is enough), we don't want to monitor the client. Additionally, we don't
need to write, but that's what ZOOKEEPER-1147 is about. My feeling is that waiting for ZK
can take a long time...

- I read the doc, I'm unclear of the cost. They say it's replaced by a broadcast for the switches
(not the routers) when they don't support a specific protocol. While I guess that it may be
challenging for systems with thousands of messages per second or hundred of thousands of client
application, here it seems quite simple: a maximum number of client in the order of a few
thousands + 1 message every 10 seconds or so.

I don't mind beeing proven wrong here. I've used such mechanisms (with actually much more
messages) in the past, but all the datacenters guys were quite used to this type of architecture
so they had no questions on it and I didn't dig inside the operational impact.

Anyway, there is no doubt it should be an optional mechanism.

                  
> Add a costless notifications mechanism from master to regionservers & clients
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-7590
>                 URL: https://issues.apache.org/jira/browse/HBASE-7590
>             Project: HBase
>          Issue Type: Bug
>          Components: Client, master, regionserver
>    Affects Versions: 0.96.0
>            Reporter: nkeywal
>
> t would be very useful to add a mechanism to distribute some information to the clients
and regionservers. Especially It would be useful to know globally (regionservers + clients
apps) that some regionservers are dead. This would allow:
> - to lower the load on the system, without clients using staled information and going
on dead machines
> - to make the recovery faster from a client point of view. It's common to use large timeouts
on the client side, so the client may need a lot of time before declaring a region server
dead and trying another one. If the client receives the information separatly about a region
server states, it can take the right decision, and continue/stop to wait accordingly.
> We can also send more information, for example instructions like 'slow down' to instruct
the client to increase the retries delay and so on.
>  Technically, the master could send this information. To lower the load on the system,
we should:
> - have a multicast communication (i.e. the master does not have to connect to all servers
by tcp), with once packet every 10 seconds or so.
> - receivers should not depend on this: if the information is available great. If not,
it should not break anything.
> - it should be optional.
> So at the end we would have a thread in the master sending a protobuf message about the
dead servers on a multicast socket. If the socket is not configured, it does not do anything.
On the client side, when we receive an information that a node is dead, we refresh the cache
about it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message