hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kousuke Saruta" <saru...@oss.nttdata.co.jp>
Subject Re: Review Request 14298: Memory leak when using JDBC connections.
Date Fri, 27 Sep 2013 00:12:49 GMT


> On Sept. 26, 2013, 6:36 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 204
> > <https://reviews.apache.org/r/14298/diff/4/?file=356951#file356951line204>
> >
> >     Similar to my other comment (on removing opHandle entry from handleToOperation
map in OperationManager), do you see any use of keeping the stmtHandle on an exception if
we ensure that the operationHandle on the server is removed from the handleToOperation map
in case of an error?

Do you mean we should not put off calling closeClientOperation until next HiveStatement#execute
or until HiveStatement#close will be called but call closeClientOperation immediately on an
exception  right? I think it's reasonable exactly.


> On Sept. 26, 2013, 6:36 p.m., Vaibhav Gumashta wrote:
> > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java, line
197
> > <https://reviews.apache.org/r/14298/diff/4/?file=356952#file356952line197>
> >
> >     In the current (patch) implementation, when an operation throws an exception,
we're making sure that the opHandle gets added to the opHandleSet in HiveSessionImpl for later
cleanup by HiveSessionImpl#close. Wouldn't a better to just remove the opHandle entry from
handleToOperation map in OperationManager but pass on the error to the client?

In the new patch, I modified HiveSessionImpl to remove opHandle from handleToOperation immediately
on an error.


- Kousuke


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14298/#review26414
-----------------------------------------------------------


On Sept. 27, 2013, 12:12 a.m., Kousuke Saruta wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14298/
> -----------------------------------------------------------
> 
> (Updated Sept. 27, 2013, 12:12 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5296
>     https://issues.apache.org/jira/browse/HIVE-5296
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation
as follows.
> 
> 1. When Exceptions are thrown during executing commmand or query, operation handle will
not release.
> 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll
so FileSystem$Cache will continue to increase.
> 
> I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation
handle is needed by OperationManager to remove from handleToOpration.
> Also, I've modified HiveSessionImpl to close FileSystem object at the end of session.
> 
> 
> Diffs
> -----
> 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 
> 
> Diff: https://reviews.apache.org/r/14298/diff/
> 
> 
> Testing
> -------
> 
> I confirmed only not increasing Hashtable$Entry by jmap.
> 
> 
> Thanks,
> 
> Kousuke Saruta
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message