Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5AC2A18612 for ; Thu, 9 Jul 2015 03:41:07 +0000 (UTC) Received: (qmail 48316 invoked by uid 500); 9 Jul 2015 03:41:06 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 48259 invoked by uid 500); 9 Jul 2015 03:41:06 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 48144 invoked by uid 99); 9 Jul 2015 03:41:06 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2015 03:41:06 +0000 Date: Thu, 9 Jul 2015 03:41:06 +0000 (UTC) From: "Victor Xu (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-12596) bulkload needs to follow locality MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Xu updated HBASE-12596: ------------------------------ Attachment: HBASE-12596-master-v6.patch Commit new version for master branch. > bulkload needs to follow locality > --------------------------------- > > Key: HBASE-12596 > URL: https://issues.apache.org/jira/browse/HBASE-12596 > Project: HBase > Issue Type: Improvement > Components: HFile, regionserver > Affects Versions: 0.98.8 > Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 > Reporter: Victor Xu > Assignee: Victor Xu > Fix For: 0.98.14 > > Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch > > > Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the "withFavoredNodes" method, and we just need to call it in HFileOutputFormat's getNewWriter(). > This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)