hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-794) Language neutral IPC as a first class component of HBase architecture
Date Sun, 03 May 2009 22:00:30 GMT

    [ https://issues.apache.org/jira/browse/HBASE-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705457#action_12705457

Jonathan Gray commented on HBASE-794:

There are two instances where it would be helpful to have an RPC call that we could access
the received byte[] directly.

Sending of PUTS and receiving of GETS.  This is being changed in 880 to being List<KeyValue>
(does not require serializiation/deserialization).  I believe, by far, the biggest improvement
we could make to help with our GC woes during imports is doing this for writes.

Just thought we should keep this in mind as we explore rpc stuff.

Right now, we allocate data for writes *at least 3 times* on the way in to the memcache. 
The last two times, it is at the individual keyval level (lots of small allocations).  Using
KeyValue, we could read the entire buffer into a single byte[] and pass around KeyValue pointers
all the way to the Memcache.  These would be grouped by row in the case of batching.

I do not think we need to go down the path of pooling.  Changes like above, and moving away
from Trees wherever possible as we're working on now, should help tremendously.  If we do
end up needing more improvements, the optimizations would be isolated to the data structure
backing Memcache.  I think it'd be a fun project...  but i agree that it can bite you hard
with a GC and we should avoid it at all costs (short of rewrites in c)  :)

> Language neutral IPC as a first class component of HBase architecture
> ---------------------------------------------------------------------
>                 Key: HBASE-794
>                 URL: https://issues.apache.org/jira/browse/HBASE-794
>             Project: Hadoop HBase
>          Issue Type: New Feature
>          Components: client, ipc, master, regionserver
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
>            Priority: Minor
> This issue considers making a language neutral IPC mechanism and wire format a first
class component of HBase architecture. Clients could talk to the master and regionserver using
this protocol instead of HRPC at their option.
> Options for language neutral IPC include:
> * Thrift: http://incubator.apache.org/thrift/
> * Protocol buffers: http://code.google.com/p/protobuf/
> * XDR: http://en.wikipedia.org/wiki/External_Data_Representation

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message