hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elliott Clark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8369) MapReduce over snapshot files
Date Fri, 13 Dec 2013 00:22:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846963#comment-13846963
] 

Elliott Clark commented on HBASE-8369:
--------------------------------------

I'm -0.5 on getting this into older releases.  To me it seems like the community drew a line
in the sand on what features made it into 0.96.  We held that line for some things (security,
etc).  So it seems weird that we would pull this into older releases without extraordinary
circumstances, and we were strict other places.

We've started faster release trains and 0.98 is on the way.  I would vote for holding a very
strict no new features into old releases now that we have addressed concerns over release
timings.

> MapReduce over snapshot files
> -----------------------------
>
>                 Key: HBASE-8369
>                 URL: https://issues.apache.org/jira/browse/HBASE-8369
>             Project: HBase
>          Issue Type: New Feature
>          Components: mapreduce, snapshots
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 0.98.0
>
>         Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch,
HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch,
HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch,
hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch
>
>
> The idea is to add an InputFormat, which can run the mapreduce job over snapshot files
directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking
a Scan object from the user, but instead of running from an online table, it runs from a table
snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader.
A RegionScanner is used internally for doing the scan without any HRegionServer bits. 
> Users have been asking and searching for ways to run MR jobs by reading directly from
hfiles, so this allows new use cases if reading from stale data is ok:
>  - Take snapshots periodically, and run MR jobs only on snapshots.
>  - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase
cluster.
>  - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's
snapshot, but read today's data from online hbase cluster. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message