hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
Date Thu, 16 Jul 2015 15:42:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629861#comment-14629861
] 

Andrew Purtell edited comment on HBASE-12596 at 7/16/15 3:41 PM:
-----------------------------------------------------------------

That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we
are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation
of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade
path from 0.98 to a branch-1 release with a contiguous feature set. 

This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can
accept changes that add features on our minor releases (0.98.x) but not on our patch releases
(0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar
but now the minors are 1.x and the patch versions are 1.x.y. 

We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not
going to retire 0.98 as RM until we have consensus it should be EOL. 

bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well.

You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis
does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense
as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking
for him.

The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions
leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat
became a pain point for downstreamers and Phoenix so now I also check that as part of the
RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes
to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications.
Already the code divergence between 0.98 and branch-1 is significant enough to make all but
trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow
what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because
of change policies applied to the 1.x branches by their RMs, and that divergence will get
a lot bigger. At that point I would advocate that 0.98 is EOLed and retire as its RM since
it would no longer need dedicated attention. At some point those two actions should happen
anyway.

Edit: Consolidated two posts and at-mentions.


was (Author: apurtell):
That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we
are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation
of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade
path from 0.98 to a branch-1 release with a contiguous feature set. 

This is the consequence of adopting semver for 1.0 and up but not earlier. We have contributors
and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98
as RM until we have consensus it should be EOL. 

> bulkload needs to follow locality
> ---------------------------------
>
>                 Key: HBASE-12596
>                 URL: https://issues.apache.org/jira/browse/HBASE-12596
>             Project: HBase
>          Issue Type: Improvement
>          Components: HFile, regionserver
>    Affects Versions: 0.98.8
>         Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7
>            Reporter: Victor Xu
>            Assignee: Victor Xu
>             Fix For: 2.0.0, 0.98.14, 1.3.0
>
>         Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch,
HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch,
HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch,
HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch
>
>
> Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded;
2. Move these HFiles to the right hdfs directory. However, the locality could be loss during
the first step. Why not just write the HFiles directly into the right place? We can do this
easily because StoreFile.WriterBuilder has the "withFavoredNodes" method, and we just need
to call it in HFileOutputFormat's getNewWriter().
> This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false'
to disable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message