Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 9 Jul 2012 21:58:34 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <1491822856.25518.1341871114485.JavaMail.jiratomcat@issues-vm>
In-Reply-To: <1288403546.25460.1341870635355.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem
 is problematic
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409896#comment-13409896 ] 

Harsh J commented on HBASE-6358:
--------------------------------

I sort of agree, except it is also more of a best-practice thing. If you bulk load remotely with only a single or very few requests per source at a time, and with a high RPC timeout at the client (such that it does not retry too often), then it should be more tolerable.

But in any case, having the RS do FS copies will indeed make it slow.

I ran into a very similar issue and the tweak I had to suggest was to indeed distcp/cp the data first and bulk load next. HBASE-6350 (Logging improvements for ops) and HBASE-6339 (Possible optimization, negative in the end) came out of it.
                
> Bulkloading from remote filesystem is problematic
> -------------------------------------------------
>
>                 Key: HBASE-6358
>                 URL: https://issues.apache.org/jira/browse/HBASE-6358
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.94.0
>            Reporter: Dave Revell
>            Assignee: Dave Revell
>
> Bulk loading hfiles that don't live on the same filesystem as HBase can cause problems for subtle reasons.
> In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its own filesystem if it's not already there. Since this can take a long time for large hfiles, it's likely that the client will timeout and retry. When the client retries repeatedly, there may be several bulkload operations in flight for the same hfile, causing lots of unnecessary IO and tying up handler threads. This can seriously impact performance. In my case, the cluster became unusable and the regionservers had to be kill -9'ed.
> Possible solutions:
>  # Require that hfiles already be on the same filesystem as HBase in order for bulkloading to succeed. The copy could be handled by LoadIncrementalHFiles before the regionserver is called.
>  # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend the timeout or something else.
> I'm willing to write a patch but I'd appreciate recommendations on how to proceed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira