Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Date: Mon, 13 May 2013 22:11:16 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: dev@hbase.apache.org
Message-ID: <JIRA.12643035.1366232813781.312121.1368483076105@arcas>
In-Reply-To: <JIRA.12643035.1366232813781@arcas>
References: <JIRA.12643035.1366232813781@arcas>
Subject: [jira] [Resolved] (HBASE-8367) LoadIncrementalHFiles does not
 return an error code nor throw Exception when failures occur due to
 timeouts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu resolved HBASE-8367.
---------------------------

    Resolution: Fixed
    
> LoadIncrementalHFiles does not return an error code nor throw Exception when failures occur due to timeouts
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-8367
>                 URL: https://issues.apache.org/jira/browse/HBASE-8367
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>    Affects Versions: 0.92.1, 0.92.2
>         Environment: Red Hat 6.2 
> Java 1.6.0_26
> Hadoop 2.0.0-mr1-cdh4.1.1
> HBase 0.92.1-cdh4.1.1
>            Reporter: Brian Dougan
>            Assignee: Brian Dougan
>            Priority: Minor
>             Fix For: 0.98.0, 0.94.8, 0.95.1
>
>         Attachments: 8367-LoadIncrementalFiles-0.94.txt, 8367-LoadIncrementalFiles-long-line-wrapped.patch, LoadIncrementalFiles-HBASE-8367.patch
>
>
> The LoadIncrementalHFiles (completebulkload) command will exit with a success code (or lack of Exception) when one or more of the HFiles fail to be imported through a few ways (mainly when timeouts occur).  Instead, it simply logs error messages to the log which makes it difficult to automate the import of HFiles programmatically.   
> The heart of the LoadIncrementalHFiles class (doBulkLoad) returns void and has essentially the following structure.
> {code:title=LoadIncrementalHFiles.java}
> try {
>       ...
>  
>     } finally {
>       pool.shutdown();
>       if (queue != null && !queue.isEmpty()) {
>         StringBuilder err = new StringBuilder();
>         err.append("-------------------------------------------------\n");
>         err.append("Bulk load aborted with some files not yet loaded:\n");
>         err.append("-------------------------------------------------\n");
>         for (LoadQueueItem q : queue) {
>           err.append("  ").append(q.hfilePath).append('\n');
>         }
>         LOG.error(err);
>       }
>     }
> {code}
> As you can see, instead of returning an error code, a success indicator, or simply throwing an Exception, an error message is sent to the log.  This results in something like the following in the logs.
> {quote}
> Bulk load aborted with some files not yet loaded:
> -------------------------------------------------
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,2.bottom
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,2.top
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,1.bottom
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,1.top
> {quote}
> Without some sort of indication, it's not currently possible to chain this command to another or to programmatically consume this class and be certain of a successful import.
> This class should be enhanced to return non-success in whatever way makes sense to the community.  I don't really have a strong preference, but one of the following should work fine (at least for my needs).
> * boolean return value on doBulkLoad (non-zero on run method)
> * Response object on doBulkLoad detailing the files that failed (non-zero on run method)
> * throw Exception in the finally block when files failed after the error is written to the log (should automatically cause non-zero on run method)
> It would also be nice to get this to the 0.94.x stream so it get included in the next Cloudera release.  Thanks!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira