hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8367) LoadIncrementalHFiles does not return an error code or throw Exception when failures occur due to timeouts.
Date Mon, 13 May 2013 00:17:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655692#comment-13655692
] 

Hadoop QA commented on HBASE-8367:
----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12582872/LoadIncrementalFiles-HBASE-8367.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 lineLengths{color}.  The patch introduces lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.security.access.TestAccessController

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5647//console

This message is automatically generated.
                
> LoadIncrementalHFiles does not return an error code or throw Exception when failures
occur due to timeouts.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-8367
>                 URL: https://issues.apache.org/jira/browse/HBASE-8367
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>    Affects Versions: 0.92.1, 0.92.2
>         Environment: Red Hat 6.2 
> Java 1.6.0_26
> Hadoop 2.0.0-mr1-cdh4.1.1
> HBase 0.92.1-cdh4.1.1
>            Reporter: Brian Dougan
>            Assignee: Brian Dougan
>            Priority: Minor
>             Fix For: 0.94.8
>
>         Attachments: LoadIncrementalFiles-HBASE-8367.patch
>
>
> The LoadIncrementalHFiles (completebulkload) command will exit with a success code (or
lack of Exception) when one or more of the HFiles fail to be imported through a few ways (mainly
when timeouts occur).  Instead, it simply logs error messages to the log which makes it difficult
to automate the import of HFiles programmatically.   
> The heart of the LoadIncrementalHFiles class (doBulkLoad) returns void and has essentially
the following structure.
> {code:title=LoadIncrementalHFiles.java}
> try {
>       ...
>  
>     } finally {
>       pool.shutdown();
>       if (queue != null && !queue.isEmpty()) {
>         StringBuilder err = new StringBuilder();
>         err.append("-------------------------------------------------\n");
>         err.append("Bulk load aborted with some files not yet loaded:\n");
>         err.append("-------------------------------------------------\n");
>         for (LoadQueueItem q : queue) {
>           err.append("  ").append(q.hfilePath).append('\n');
>         }
>         LOG.error(err);
>       }
>     }
> {code}
> As you can see, instead of returning an error code, a success indicator, or simply throwing
an Exception, an error message is sent to the log.  This results in something like the following
in the logs.
> {quote}
> Bulk load aborted with some files not yet loaded:
> -------------------------------------------------
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,2.bottom
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,2.top
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,1.bottom
>   hdfs://prmdprod/user/userxxx/hfile/TABLE-1365721885510/record/_tmp/TABLE,1.top
> {quote}
> Without some sort of indication, it's not currently possible to chain this command to
another or to programmatically consume this class and be certain of a successful import.
> This class should be enhanced to return non-success in whatever way makes sense to the
community.  I don't really have a strong preference, but one of the following should work
fine (at least for my needs).
> * boolean return value on doBulkLoad (non-zero on run method)
> * Response object on doBulkLoad detailing the files that failed (non-zero on run method)
> * throw Exception in the finally block when files failed after the error is written to
the log (should automatically cause non-zero on run method)
> It would also be nice to get this to the 0.94.x stream so it get included in the next
Cloudera release.  Thanks!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message