hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jurriaan Mous (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12597) Add RpcClient interface and enable changing of RpcClient implementation
Date Mon, 01 Dec 2014 09:08:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229559#comment-14229559

Jurriaan Mous commented on HBASE-12597:

bq. I was just trying to warn you off the implementation to instead work against the Connection
(rather than HCI) if possible. The new Interfaces are kinda sweet. Would be grand if your
new rpc Interface could be in the same style.

Since the RpcClient changes proposed here would probably land on branch-1 and/or master I
think I need to make a branch myself to support the new Hbase version. I will take your recommendation
and base it on the new Api. :)

bq. Can you say more on the above? Why would you use hbase implementation connection pooling
and not Netty's (I'm not sure – I'm guessing it can do this)

While Netty pools the connections (called channels) for the outside world in a connection
pool, inside the application depends on how you want to recycle connections. I based it on
the ConnectionId inside RpcClient so channels get mapped in the same way as in Hbase itself.

ConnectionId takes care of the grouping which is on User ticket, protobuf servicename, the
region server address and the rpc timeout. This is pretty HBase specific. The map it would
end up in is PoolMap<RpcClient.ConnectionId, AsyncRpcChannel> connections. And the client
uses the map to give users an existing or a new connection.

bq. Do you have a proposal for rpcclient interface? 

Will look into it. I would probably extract whatever is used in the HConnectionImplementation
into an interface.

bq. There is our use of the protobuf Service Interfaces taking rpc controllers etc. We just
took it on because it was there and the pb code generation did all the stubbing for us. Just
saying we are amenable to change long as the basic protocol stays the same, for now.

The protobuf Service interface made it easy to make a comparable implementation. Although
I could see that it could be made more optimal since like the Hbase implementation I also
needed to create a new async PayloadCarryingRpcController to pass information around the limiting
interface. Although it was not that big of a problem. I would recommend not to make big changes
in it without taking care of an async implementation. But I don't know if it is worth it to
create new implementations for each Service.

bq. That'd be pretty great. Thinking on it, I suppose it should be ok doing it as netty channel.
You just have to have our idiosyncratic header stuff up front on the connection which I believe
netty will let you just insert. Thats great.

I handle those directly after channel creation. If channel compression is added there needs
to be a new header defining the compression defined. We need something of a header in which
the client announces in the connect header which compressions it supports and the server replying
with a header which compression it uses.

As said I will return with a first suggestion on the RpcClient interface.

> Add RpcClient interface and enable changing of RpcClient implementation
> -----------------------------------------------------------------------
>                 Key: HBASE-12597
>                 URL: https://issues.apache.org/jira/browse/HBASE-12597
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client
>            Reporter: Jurriaan Mous
> Currently HConnectionImplementation works with the included RpcClient which is a direct
implementation and not defined by an interface.
> It would be great to be able to swap out the default RpcClient with another implementation
which can also be controlled by the default HConnectionImplementation. 
> Suggested changes:
> - Create a RpcClient interface which defines all the ways HConnectionImplementation interacts
with an RPC client. Like getting a blocking protobuf service interface or closing the client.
> - Define which RpcClient implementation to construct by setting a configuration variable
which defaults to the current RpcClient.
> - Possibly create an abstract RpcClient class to only load all the basic Rpc layer configurations
to be used in an implementation.
> Why? It enables experimentation with RpcClients which could enable new features or could
be more performant than the included client. 
> I created a new RpcClient implementation based on Netty which can also be called asynchronously.
It would be great to also be able to use this RpcClient in all the default ways and tests
to see if there are any issues with it. 
> https://github.com/jurmous/async-hbase-client/
> https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java

This message was sent by Atlassian JIRA

View raw message