phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
Date Wed, 23 May 2018 19:37:01 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487912#comment-16487912
] 

Hudson commented on PHOENIX-4701:
---------------------------------

FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1885 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1885/])
PHOENIX-4701 Write client-side metrics asynchronously to SYSTEM.LOG (ankitsinghal59: rev 7afaceb7e7355e59ae9465a02b812b230fc58edd)
* (add) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricUtil.java
* (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryLoggerIT.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/LogLevel.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/StatementContext.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/ChunkedResultIterator.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogInfo.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixRecordReader.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java
* (edit) bin/hbase-site.xml
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/SchemaUtil.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/TableLogWriter.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/ParallelIterators.java
* (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryStatus.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricType.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEvent.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MemoryMetricsHolder.java
* (delete) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogState.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ScanMetricsHolder.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MutationMetricQueue.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/LogWriter.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerUtil.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEventTranslator.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/OverAllQueryMetrics.java
* (edit) phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogger.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/QueryUtil.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/TaskExecutionMetricsHolder.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ReadMetricQueue.java
* (edit) phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixMetricsIT.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/iterate/SpoolingResultIteratorTest.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/SerialIterators.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/SpoolingMetricsHolder.java
PHOENIX-4701 Write client-side metrics asynchronously to (ankitsinghal59: rev 50533ce387fc6cabb5aaccdbe6677a97b9debe73)
* (edit) phoenix-core/src/it/java/org/apache/phoenix/monitoring/PhoenixMetricsIT.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/CombinableMetricImpl.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MutationMetricQueue.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricUtil.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/SpoolingMetricsHolder.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/OverAllQueryMetrics.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/CombinableMetric.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/iterate/SpoolingResultIteratorTest.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ReadMetricQueue.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/TaskExecutionMetricsHolder.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/StatementContext.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ScanMetricsHolder.java
PHOENIX-4701 Write client-side metrics asynchronously to (jtaylor: rev b63b8e9e68e36b272479f69a92530b36d80c21a5)
* (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryWithLimitIT.java


> Write client-side metrics asynchronously to SYSTEM.LOG
> ------------------------------------------------------
>
>                 Key: PHOENIX-4701
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4701
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: James Taylor
>            Assignee: Ankit Singhal
>            Priority: Major
>             Fix For: 4.14.0, 5.0.0
>
>         Attachments: PHOENIX-4701.patch, PHOENIX-4701_addendum.patch, PHOENIX-4701_addendum2.patch,
PHOENIX-4701_master.patch, PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch,
PHOENIX-4701_wip3.patch
>
>
> Rather than inventing a new, different set of client-side metrics to persist, we should
just persist our [client metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG.
The metrics captures all the same information as your QueryLogInfo (and much more), rolls
all the information up to a single set of metrics for each Phoenix statement (aggregating/merging
parallel scans, etc), and can emits a single log line (which could be written in a single
upsert statement). At SFDC, we emit this information into a file system log in a layer above
(and use Splunk to produce nifty dashboard for monitoring), but this could easily be emitted
directly in Phoenix and go through your asynchronous write path (and then use Phoenix queries
to produce the same kind of dashboards). The only piece would be to add the concept of a log
level to each metric to enable statically controlling which metrics are output.
> With this approach, the SYSTEM.LOG table could be declared immutable and use our dense
storage format with a single byte for column encoding and get a 3-5x perf gain. This would
also open the door for users to potentially add secondary indexes on the table. See schema
identified in the wip2 patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message