hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avik Dey (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-10353) Alternative I/O Engine
Date Wed, 15 Jan 2014 23:06:22 GMT
Avik Dey created HBASE-10353:

             Summary: Alternative I/O Engine
                 Key: HBASE-10353
                 URL: https://issues.apache.org/jira/browse/HBASE-10353
             Project: HBase
          Issue Type: Umbrella
            Reporter: Avik Dey

Increasingly HBase is being asked to operate in server environments where the Java Virtual
Machine is still working to take optimum advantage of the server resources, for example those
backed by solid-state storage, or deployed on large memory systems. Many HBase users also
struggle in environments with demanding and tight request latency SLAs. The Java sandbox does
not yet have facilities for taking advantage of OS level features for high throughput low
latency I/O, nor efficient access to interconnects for high performance computing. Our test
results show that even using G1 GC, Java server processes that must operate within tight latency
bounds must keep their heaps small relative to the kind of RAM we can pack into a server today.
We would like this issue to serve as an umbrella for brainstorming and proposals for an incremental
evolution of HBase, through use of existing plug points and careful introduction of new ones,
to a complete architecture for plugging in alternative I/O engines. As a consequence we will
be able to explore improvements within and outside the JVM. Such an alternative engine would
be expected to provide lower latency and higher throughput than what is possible within the
constraints of today's implementation. Other JIRAs already cover moving allocations for the
I/O path off heap. Once we have I/O code paths built on direct buffers that conveniently provides
substrate for efficient boundary crossings from Java into native code, we can open subtasks
for proposing and discussing where those boundaries can be drawn. We will also open subtasks
for proposing the shape and scope of the first experimental alternative I/O engine.
This work should maintain API and wire compatibility as well preserve data format consistency
with today’s HBase, so HBase users can take advantage of this work without any need to change
client code or massage existing data.

This message was sent by Atlassian JIRA

View raw message