drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacques Nadeau (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-1990) Add peak memory allocation in a operator to OperatorStats.
Date Wed, 14 Jan 2015 07:13:35 GMT

    [ https://issues.apache.org/jira/browse/DRILL-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276587#comment-14276587
] 

Jacques Nadeau commented on DRILL-1990:
---------------------------------------

looks like wrong patch attached.

> Add peak memory allocation in a operator to OperatorStats.
> ----------------------------------------------------------
>
>                 Key: DRILL-1990
>                 URL: https://issues.apache.org/jira/browse/DRILL-1990
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Operators
>            Reporter: Venki Korukanti
>            Assignee: Mehant Baid
>             Fix For: 0.8.0
>
>         Attachments: 0001-DRILL-1991-Code-indendation-and-formatting-cleanup-f.patch
>
>
> Currently we have "localMemoryAllocated" which is always set to zero as the we try to
fill the stats at the end of execution of fragment by calling calling allocator. getAllocatedMemory()
which at that point has already released the allocated memory. Instead having a peak memory
that allocator (each operator has its own allocator) has seen in the lifetime of the operator
execution will be useful.
> Example query on query profile: To find all aggregate peak memory of each operator across
all minor fragments in a major fragment and list them in descending order peak memory usage
> {code:sql}
> SELECT
>   majorFragmentId,
>   opProfile['operatorType'] opType,
>   sum(opProfile['peakLocalMemoryAllocated']) aggPeakMemoryAcrossAllMinorFragments 
> FROM 
>   (SELECT
>        majorFragmentId,
>        flatten(minorFragProfile['operatorProfile']) opProfile
>    FROM 
>        (SELECT
>              majorFragment['majorFragmentId'] majorFragmentId, 
>              flatten(majorFragment['minorFragmentProfile']) minorFragProfile
>          FROM
>              (SELECT flatten(fragmentProfile) as majorFragment from dfs.`/tmp/a.json`)
>          )
>    )
> -- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular operator
> GROUP BY 
>   majorFragmentId,
>   opProfile['operatorType']
> ORDER BY
>   aggPeakMemoryAcrossAllMinorFragments DESC;
> {code}
> {code}
> +-----------------+------------+--------------------------------------+
> | majorFragmentId |   opType   | aggPeakMemoryAcrossAllMinorFragments |
> +-----------------+------------+--------------------------------------+
> | 1               | 4          | 115065856                            |
> | 1               | 3          | 10027008                             |
> | 0               | 3          | 1671168                              |
> | 3               | 6          | 1536000                              |
> | 2               | 6          | 901120                               |
> | 1               | 6          | 606208                               |
> | 3               | 28         | 393216                               |
> | 2               | 28         | 229376                               |
> | 3               | 10         | 122880                               |
> | 2               | 10         | 81920                                |
> | 0               | 11         | 0                                    |
> | 0               | 10         | 0                                    |
> | 0               | 13         | 0                                    |
> | 1               | 10         | 0                                    |
> | 1               | 11         | 0                                    |
> +-----------------+------------+--------------------------------------+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message