hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jan Lukavsky (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-5757) TableInputFormat should handle as much errors as possible
Date Tue, 10 Apr 2012 09:59:18 GMT
TableInputFormat should handle as much errors as possible
---------------------------------------------------------

                 Key: HBASE-5757
                 URL: https://issues.apache.org/jira/browse/HBASE-5757
             Project: HBase
          Issue Type: Bug
          Components: mapred, mapreduce
    Affects Versions: 0.90.6
            Reporter: Jan Lukavsky


Prior to HBASE-4196 there was different handling of IOExceptions thrown from scanner in mapred
and mapreduce API. The patch to HBASE-4196 unified this handling so that if exception is caught
a reconnect is attempted (without bothering the mapred client). After that, HBASE-4269 changed
this behavior back, but in both mapred and mapreduce APIs. The question is, is there any reason
not to handle all errors that the input format can handle? In other words, why not try to
reissue the request after *any* IOException? I see the following disadvantages of current
approach
 * the client may see exceptions like LeaseException and ScannerTimeoutException if he fails
to process all fetched data in timeout
 * to avoid ScannerTimeoutException the client must raise hbase.regionserver.lease.period
 * timeouts for tasks is aready configured in mapred.task.timeout, so this seems to me a bit
redundant, because typically one needs to update both these parameters
 * I don't see any possibility to get rid of LeaseException (this is configured on server
side)

I think all of these issues would be gone, if the DoNotRetryIOException would not be rethrown.
On the other hand, handling errors in InputFormat has disadvantage, that it may hide from
the user some inefficiency. Eg. if I have very big scanner.caching, and I manage to process
only a few rows in timeout, I will end up with single row being fetched many times (and will
not be explicitly notified about this). Could we solve this problem by adding some counter
to the InputFormat?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message