phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3584) Expose metrics for ConnectionQueryServices instances and their allocators in the JVM
Date Tue, 10 Jan 2017 19:31:58 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815937#comment-15815937
] 

Andrew Purtell commented on PHOENIX-3584:
-----------------------------------------

[~lhofhansl] points out we also need to track by connection allocation rate, in case we have
a caller generating a high rate of short lived connections. That is a bit more challenging
to accomplish. One thought is introduce a parametrizable token bucket per allocator identity
(established by hash of stack trace or optional connect string) and increment a counter if
there aren't any available tokens. The nice thing about this idea is it is only a small delta
from there to use the token buckets to rate limit allocators should we choose to do so at
a future time. FWIW

> Expose metrics for ConnectionQueryServices instances and their allocators in the JVM
> ------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3584
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3584
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: Andrew Purtell
>            Assignee: Samarth Jain
>
> In the case a client is leaking Phoenix/HBase connections it would be helpful to have
metrics available on the number of ConnectionQueryServices (ConnectionQueryServicesImpl) instances
and who has allocated them. 
> For the latter, we could get a stacktrace when ConnectionQueryServicesImpls are allocated
(should be a relatively rare) and keep a count by hash of the call stack (and save the call
stack). Then we need a method to dump the hash to callstack map as a string. This method can
be called remotely by JMX when debugging leaks in a live environment. Perhaps after the count
of ConnectionQueryServicesImpls goes over a configurable threshold we can also log warnings
that dump the counts by hash and callstacks corresponding to those hashes. 
> Or, we should only have multiple ConnectionQueryServicesImpls if an optional parameter
is passed in the JDBC connect string. We could keep counts by that parameter string and dump
that instead of call stacks. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message