hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16117) Fix Connection leak in mapred.TableOutputFormat
Date Mon, 11 Jul 2016 15:21:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370959#comment-15370959

Jonathan Hsieh commented on HBASE-16117:

For the first concern, it is subtle and a good catch. The original version is two nested try
blocks.  The isMasterRunning is already in the first one (ClusterConnection).   If isMasterRunning
fails it turns into a ZK exception which is not what we want. Let me tweak this and see where
it leads.

For the second about adding IOE in registry -- the problem was zk failures would get swallowed
and never surface.  With the unit test there was no way of telling if that we have a bad hbase
connection (e.g. we run out of zk connections and then get a connection that doesn't fail
fast).  In order to surface this kind of error at initialization time we need to have the
exception come out.  An alternative that wouldn't change interface is to convert to a runtime
exception and then inspect to reconvert back which seems gross IMO.

> Fix Connection leak in mapred.TableOutputFormat 
> ------------------------------------------------
>                 Key: HBASE-16117
>                 URL: https://issues.apache.org/jira/browse/HBASE-16117
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>    Affects Versions: 2.0.0, 1.3.0, 1.2.2, 1.1.6
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 2.0.0, 1.3.0, 1.1.6, 1.2.3
>         Attachments: hbase-16117.branch-1.patch, hbase-16117.patch, hbase-16117.v2.branch-1.patch,
hbase-16117.v2.patch, hbase-16117.v3.branch-1.patch, hbase-16117.v3.patch, hbase-16117.v4.patch
> Spark seems to instantiate multiple instances of output formats within a single process.
 When mapred.TableOutputFormat (not mapreduce.TableOutputFormat) is used, this may cause connection
leaks that slowly exhaust the cluster's zk connections.  
> This patch fixes that.

This message was sent by Atlassian JIRA

View raw message