hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13159) Consider RangeReferenceFiles with transformations
Date Sat, 07 Mar 2015 04:59:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351410#comment-14351410
] 

stack commented on HBASE-13159:
-------------------------------

Tell us more about use cases. Give us example transforms (other than identity and 'drop'...
or is that it?) Regards transforms, ain't that just filter at HFile level?

How would you get the 'view' up to high level when transform is at the HFile level?

bq. ...and can be referred from multiple regions

Always shied from supporting this. Was afraid of the complexity and dangling pointers.

I suppose if we had RangeReferenceFiles, could help at log split time. Same count of files
so same NN traffic... we'd just be reading from one file. With current small files, you can
remove them when done but with RangeReferenceFiles, you'd need to do refcounting... in meta?

RangeReferenceFiles would not save NN traffic since we have to drop them in the FS but could
save i/o not replicating Cells.

@enis The index would be persisted in HFile or to a separate RangeReferenceFile?  The benefit
is smaller indices? Transform would add prefix back? Will that work? (maybe I'm not following)






> Consider RangeReferenceFiles with transformations
> -------------------------------------------------
>
>                 Key: HBASE-13159
>                 URL: https://issues.apache.org/jira/browse/HBASE-13159
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: Lars Hofhansl
>
> Currently we have References used by HalfStoreReaders and HFileLinks.
> For various use cases we have here we have need for a RangeReferences with simple transformation
of the keys.
> That would allow us to map HFiles between regions or even tables without copying any
data.
> We can probably combine HalfStores, HFileLinks, and RangeReferences into a single concept:
> * RangeReference = arbitrary start and stop row, arbitrary key transformation
> * HFileLink = start and stop keys set to the linked file's start/stop key, transformation
= identity
> * (HalfStore) References = start/stop key set according to top or bottom reference, transformation
= identity
> Note this is a *brainstorming* issue. :)
> (Could start with just references with arbitrary start/stop keys, and do transformations
later)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message