hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Lau (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-14439) New Filesystem Abstraction Layer
Date Tue, 15 Sep 2015 22:02:45 GMT
Ben Lau created HBASE-14439:

             Summary: New Filesystem Abstraction Layer
                 Key: HBASE-14439
                 URL: https://issues.apache.org/jira/browse/HBASE-14439
             Project: HBase
          Issue Type: New Feature
            Reporter: Ben Lau

Ticket for work in progress on new FileSystem abstractions.  Previously, we (Yahoo) submitted
a ticket that would add support for humongous (1 million region+) tables via a hierarchical
layout (HBASE-13991).  However open source is moving in a similar but not identical direction
in the future and so the patch will not be merged.

We will be working with Cloudera on a different patch now.  It will create/add to 2 layers--
a path abstraction layer and a use-oriented abstraction layer.  The path abstraction layer
is epitomized by classes like FsUtils (and in the patch new classes like AFsLayout).  The
use oriented abstraction layer is epitomized by existing classes like MasterFileSystem/HRegionFileSystem
(and possibly new classes later) that build on the path abstraction layer and focus on 'doing
things' (eg creating regions) and less on the gritty details like the paths.

This work on abstracting and isolating the paths from the use cases will help Yahoo not diverge
too much from open source with its internal 'Humongous' table hierarchical layout, while also
helping open source move further towards the eventual goal of redoing the FS layout in a similar
(but different) hierarchical layout later that focuses on data directory uniformity and storing
hierarchy in the meta table instead (see HBASE-14090.)

Attached to this ticket is some work we've done at Yahoo so far that will be put into an open
source HBase branch for further collaboration.  The patch is not meant to be complete yet
and is a work in progress.  (Please wait on patch comments/reviews.)

This message was sent by Atlassian JIRA

View raw message