ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Collecting query metrics and statistics
Date Fri, 04 Aug 2017 15:46:31 GMT
NB: I mixed up in places "mapper" and "reducer" in the second part of the
message.

пт, 4 авг. 2017 г. в 17:48, Vladimir Ozerov <vozerov@gridgain.com>:

> Igniters,
>
> At the moment Ignite lacks administrative and management capabilities in
> various places. On such demanded place is SQL queries. I want to start a
> discussion around adding query better metrics to Ignite.
>
> I propose to split the task in two parts: infrastructure and UX.
>
> 1) We start with adding unique UUID to all query requests, to be able to
> glue all pieces together. Then we log the following things (approximately):
> - Query ID
> - Original query
> - Map query
> - Mapper execution time
> - Mapper rows
> - Mapper output bytes (to reducer)
> - Mapper input bytes (for distributed joins)
> - Reduce query
> - Reduce execution time
> - Reduce rows
> - Reduce input bytes (sum of all mapper output bytes)
> - Some info on distributed joins may be
> - Advanced things in future (memory page accesses, disk accesses, etc.)
>
> All these stats are saved to local structures. These structures are
> accessible through some API and typically not exchange between nodes until
> requested.
>
> 2) UX
> The most important - we will made these stats *accessible through SQL*!
>
> SELECT * FROM map_query_stats
> |node|query_id|query              |time_ms|input_bytes|
> -------------------------------------------------------
> |CLI1|*UUID1*   |SELECT AVG(f)      | 50    |100        |
>
> SELECT * FROM reduce_query_stats
> |node|query_id|query                  |time_ms|output_bytes|disk_reads|
> -----------------------------------------------------------------------
> |SRV1|*UUID1*   |SELECT SUM(f), COUNT(f)|00     |20          |35        |
>
> Then we do some UNIONS/JOINS from client/console/driver/whatever, and:
> SELECT ... FROM map_query_stats WHERE query_id = *UUID1*
> UNION
> SELECT ... FROM reduce_query_stats WHERE query_id = *UUID1*
>
> |total_time|total_disk_reads|reduce_node|reduce_time|reduce_disk_reads|
> -----------------------------------------------------------------------
> |100       |180             |           |           |                 |
> -----------------------------------------------------------------------
> |          |                |SRV1       |10         |35               |
> -----------------------------------------------------------------------
> |          |                |SRV2       |90         |130              |
> -----------------------------------------------------------------------
> |          |                |SRV3       |20         |25               |
>
> Makes sense?
>
> Vladimir.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message