phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samarth Jain (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-1452) Add Phoenix client-side logging and capture resource utilization metrics
Date Mon, 02 Mar 2015 01:12:05 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Samarth Jain updated PHOENIX-1452:
----------------------------------
    Attachment: PHOENIX-1452.patch

[~jfernando_sfdc][~jamestaylor] - Thanks for your feedback. 

Based on the feedback and offline conversations with [~lhofhansl] and [~jfernando_sfdc], I
have modified the patch to expose the internal metrics as counters and cumulative numbers.
This makes the metrics collection lightweight with no to minimal impact on the normal operation.
The PhoenixMetric interface provides a way of exposing these numbers and a reset method.

As part of this patch we are now exposing:
1) Counter for number of spool files created.
2) Cumulative number of parallel scans.
3) Cumulative number of serial scans.
4) Counter for number of query failures. 
5) Counter for number of query time outs. 
6) Cumulative query time.
7) Cumulative mutation time.
8) Cumulative mutation size in bytes.
9) Cumulative number of mutations.
10) Cumulative bytes allocated by the memory manager.
11) Cumulative time spent by threads waiting for the memory to be allocated.

Apart from the above metrics, the patch also adds an InstrumentedThreadPoolExecutor. It is
configurable for the users to decide which executor they want to use. The following metrics
are exposed by the InstrumentedThreadPoolExecutor: 
1) Cumulative number of tasks submitted to the pool.
2) Cumulative number of tasks rejected.
3) Cumulative time spent by tasks in the queue.
4) Cumulative time taken by tasks to complete - from construction to execution completion.
5) Cumulative time taken by tasks to execute.

> Add Phoenix client-side logging and capture resource utilization metrics
> ------------------------------------------------------------------------
>
>                 Key: PHOENIX-1452
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1452
>             Project: Phoenix
>          Issue Type: Improvement
>    Affects Versions: 5.0.0, 4.2
>            Reporter: Jan Fernando
>            Assignee: Samarth Jain
>         Attachments: PHOENIX-1452.patch, wip.patch
>
>
> For performance testing and tuning of features that use Phoenix and for production monitoring
it would be really helpful to easily be able to extract statistics about Phoenix's client-side
Thread Pool and Queue Depth usage to help with tuning and being able to correlate the impact
of tuning these 2 parameters to query performance.
> For global per JVM logging one of the following would meet my needs, with a preference
for #2:
> 1. A simple log line that that logs the data in ThreadPoolExecutor.toString() at a configurable
interval
> 2. Exposing the ThreadPoolExecutor metrics in PhoenixRuntime or other global client exposed
class and allow client to do their own logging.
> In addition to this it would also be really valuable to have a single log line per query
that provides statistics about the level of parallelism i.e. number of parallel scans being
executed. I don't full explain plan level of data but a good heuristic to be able to track
over time how queries are utilizing the thread pool as data size grows etc. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message