hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-4047) [Coprocessors] Generic external process host
Date Thu, 30 Jun 2011 16:30:28 GMT
[Coprocessors] Generic external process host

                 Key: HBASE-4047
                 URL: https://issues.apache.org/jira/browse/HBASE-4047
             Project: HBase
          Issue Type: New Feature
          Components: coprocessors
            Reporter: Andrew Purtell

Where HBase coprocessors deviate substantially from the design (as I understand it) of Google's
BigTable coprocessors is we've reimagined it as a framework for internal extension. In contrast
BigTable coprocessors run as separate processes colocated with tablet servers. The essential
trade off is between performance, flexibility and possibility, and the ability to control
and enforce resource usage.

Since the initial design of HBase coprocessors some additional considerations are in play:

- Developing computational frameworks sitting directly on top of HBase hosted in coprocessor(s);

- Introduction of the map reduce next generation (mrng) resource management model, and the
probability that limits will be enforced via cgroups at the OS level after this is generally
available, e.g. when RHEL 6 deployments are common;

- The possibility of deployment of HBase onto mrng-enabled Hadoop clusters via the mrng resource
manager and a HBase-specific application controller.

Therefore we should consider developing a coprocessor that is a generic host for another coprocessor,
but one that forks a child process, loads the target coprocessor into the child, establishes
a bidirectional pipe and uses an eventing model and umbilical protocol to provide for the
coprocessor loaded into the child the same semantics as if it was loaded internally to the
parent, and (eventually) use available resource management capabilities on the platform --
perhaps via the mrng resource controller or directly with cgroups -- to limit the child as
desired by system administrators or the application designer.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message