hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14439) New/Improved Filesystem Abstractions
Date Thu, 22 Sep 2016 16:35:20 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Busbey updated HBASE-14439:
--------------------------------
    Assignee: Sean Busbey  (was: Matteo Bertozzi)

Making another go at this effort. I've moved the experimental implementation from last year
to the branch {{hbase-14439-experimental}} and posted a WIP on {{hbase-14439}} so that I can
work with [~uagashe] in the public repo before getting things to a level of done-ness needed
to patch master.

the WIP has a bunch of gaps. I've got some notes on them in {{TODO}} lines. Most notably,
it currently will pass {{mvn -DskipTests compile}} but won't pass {{mvn -DskipTests test-compile}}.

> New/Improved Filesystem Abstractions
> ------------------------------------
>
>                 Key: HBASE-14439
>                 URL: https://issues.apache.org/jira/browse/HBASE-14439
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Ben Lau
>            Assignee: Sean Busbey
>         Attachments: abstraction.patch
>
>
> Ticket for work in progress on new FileSystem abstractions.  Previously, we (Yahoo) submitted
a ticket that would add support for humongous (1 million region+) tables via a hierarchical
layout (HBASE-13991).  However open source is moving in a similar but not identical direction
in the future and so the patch will not be merged into open source.
> We will be working on a different patch now with folks from open source.  It will create/add
to 2 layers-- a path abstraction layer and a use-oriented abstraction layer.  The path abstraction
layer is epitomized by classes like FsUtils (and in the patch new classes like AFsLayout).
 The use oriented abstraction layer is epitomized by existing classes like MasterFileSystem/HRegionFileSystem
(and possibly new classes later) that build on the path abstraction layer and focus on 'doing
things' (eg creating regions) and less on the gritty details like the paths.
> This work on abstracting and isolating the paths from the use cases will help Yahoo not
diverge too much from open source with its internal 'Humongous' table hierarchical layout,
while also helping open source move further towards the eventual goal of redoing the FS layout
in a similar (but different) hierarchical layout later that focuses on data directory uniformity
(unlike the humongous patch) and storing hierarchy in the meta table instead which enables
new optimizations (see HBASE-14090.)
> Attached to this ticket is some work we've done at Yahoo so far that will be put into
an open source HBase branch for further collaboration.  The patch is not meant to be complete
yet and is a work in progress.  (Please wait on patch comments/reviews.)  It also includes
some Yahoo-specific 'humongous' layout code that will be removed before submission in open
source.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message