drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mehant Baid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1990) Add peak memory allocation in a operator to OperatorStats.
Date Thu, 15 Jan 2015 00:51:35 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278010#comment-14278010
] 

Mehant Baid commented on DRILL-1990:
------------------------------------

Looks good, +1

> Add peak memory allocation in a operator to OperatorStats.
> ----------------------------------------------------------
>
>                 Key: DRILL-1990
>                 URL: https://issues.apache.org/jira/browse/DRILL-1990
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Operators
>            Reporter: Venki Korukanti
>            Assignee: Mehant Baid
>             Fix For: 0.8.0
>
>         Attachments: DRILL-1990-1.patch
>
>
> Currently we have "localMemoryAllocated" which is always set to zero as we try to fill
the stats at the end of fragment execution by calling allocator.getAllocatedMemory() which
at that point has already released the allocated memory. Instead if we have a stat for peak
memory an allocator has seen in the lifetime of the operator execution will be useful (each
operator has its own allocator). 
> Example query on query profile: To find aggregate of peak memory of each operator across
all minor fragments in a major fragment and list them in descending order of peak memory usage
> {code:sql}
> SELECT
>   majorFragmentId,
>   opProfile['operatorType'] opType,
>   sum(opProfile['peakLocalMemoryAllocated']) aggPeakMemoryAcrossAllMinorFragments 
> FROM 
>   (SELECT
>        majorFragmentId,
>        flatten(minorFragProfile['operatorProfile']) opProfile
>    FROM 
>        (SELECT
>              majorFragment['majorFragmentId'] majorFragmentId, 
>              flatten(majorFragment['minorFragmentProfile']) minorFragProfile
>          FROM
>              (SELECT flatten(fragmentProfile) as majorFragment from dfs.`/tmp/a.json`)
>          )
>    )
> -- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular operator
> GROUP BY 
>   majorFragmentId,
>   opProfile['operatorType']
> ORDER BY
>   aggPeakMemoryAcrossAllMinorFragments DESC;
> {code}
> {code}
> +-----------------+------------+--------------------------------------+
> | majorFragmentId |   opType   | aggPeakMemoryAcrossAllMinorFragments |
> +-----------------+------------+--------------------------------------+
> | 1               | 4          | 115065856                            |
> | 1               | 3          | 10027008                             |
> | 0               | 3          | 1671168                              |
> | 3               | 6          | 1536000                              |
> | 2               | 6          | 901120                               |
> | 1               | 6          | 606208                               |
> | 3               | 28         | 393216                               |
> | 2               | 28         | 229376                               |
> | 3               | 10         | 122880                               |
> | 2               | 10         | 81920                                |
> | 0               | 11         | 0                                    |
> | 0               | 10         | 0                                    |
> | 0               | 13         | 0                                    |
> | 1               | 10         | 0                                    |
> | 1               | 11         | 0                                    |
> +-----------------+------------+--------------------------------------+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message