phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-1452) Add Phoenix client-side logging and capture resource utilization metrics
Date Wed, 04 Mar 2015 01:37:05 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346190#comment-14346190
] 

James Taylor commented on PHOENIX-1452:
---------------------------------------

Looking good, [~samarthjain]. Here are some questions/comments:
- Should we have a way of turning metrics collection off, along the lines of logger.isDebugEnabled(),
like a Metrics.isEnabled() call?
- For serial scans, the number of scans is probably not what we want to track, as we execute
serially in order, starting from the first scan until the limit is reached, which is usually
only a single scan. Capturing the total number of scans doesn't really tell us anything, and
there's no good way to track how many of these serial scans get executed (nor is that very
interesting anyway).
{code}
+    
+    private static int numSerialScans(List<List<Scan>> nestedScans) {
+        int numScans = 0;
+        for (List<Scan> scans : nestedScans) {
+            numScans += scans.size();
+        }
+        return numScans;
+    }
+    
     @Override
     protected void submitWork(List<List<Scan>> nestedScans, List<List<Pair<Scan,Future<PeekingResultIterator>>>>
nestedFutures,
             final List<PeekingResultIterator> allIterators, int estFlattenedSize) {
@@ -67,7 +77,7 @@ public class SerialIterators extends BaseResultIterators {
         // will spray the scans across machines as opposed to targeting a
         // single one since the scans are in row key order.
         ExecutorService executor = context.getConnection().getQueryServices().getExecutor();
-        
+        numberOfSerialScans.add(numSerialScans(nestedScans));
{code}


> Add Phoenix client-side logging and capture resource utilization metrics
> ------------------------------------------------------------------------
>
>                 Key: PHOENIX-1452
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-1452
>             Project: Phoenix
>          Issue Type: Improvement
>    Affects Versions: 5.0.0, 4.2
>            Reporter: Jan Fernando
>            Assignee: Samarth Jain
>         Attachments: PHOENIX-1452.patch, PHOENIX-1452_v2.patch, wip.patch
>
>
> For performance testing and tuning of features that use Phoenix and for production monitoring
it would be really helpful to easily be able to extract statistics about Phoenix's client-side
Thread Pool and Queue Depth usage to help with tuning and being able to correlate the impact
of tuning these 2 parameters to query performance.
> For global per JVM logging one of the following would meet my needs, with a preference
for #2:
> 1. A simple log line that that logs the data in ThreadPoolExecutor.toString() at a configurable
interval
> 2. Exposing the ThreadPoolExecutor metrics in PhoenixRuntime or other global client exposed
class and allow client to do their own logging.
> In addition to this it would also be really valuable to have a single log line per query
that provides statistics about the level of parallelism i.e. number of parallel scans being
executed. I don't full explain plan level of data but a good heuristic to be able to track
over time how queries are utilizing the thread pool as data size grows etc. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message