drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Victoria Markman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-4266) Possible memory leak (fragmentation ?) in rpc layer
Date Mon, 18 Jan 2016 15:00:44 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Victoria Markman updated DRILL-4266:
------------------------------------
    Attachment: memComsumption_framework.output_Sun_Jan_17_04_jacques_branch_drill-4131

Attached are results of the same test executed with drill built from branch:

{code}
Generated by Git-Commit-Id-Plugin
#Sun Jan 17 03:49:25 UTC 2016
git.commit.id.abbrev=daa89a6
git.commit.user.email=jacques@apache.org
git.commit.message.full=DRILL-4131\: Move RPC allocators under Drill's root allocator &
accounting\n\n- Allow settings to be set to ensure RPC reservation and maximums (currently
unset by default). Defaults set in drill-module.conf\n- Add new metrics to report RPC layer
memory consumption.\n- Check for memory leaks from RPC layer at shutdown.\n
git.commit.id=daa89a68aa416f8718ad005bc57baf4ac58a9c66
git.commit.message.short=DRILL-4131\: Move RPC allocators under Drill's root allocator &
accounting
git.commit.user.name=Jacques Nadeau
git.build.user.name=vmarkman
git.commit.id.describe=0.9.0-560-gdaa89a6
git.build.user.email=vmarkman@maprtech.com
git.branch=daa89a68aa416f8718ad005bc57baf4ac58a9c66
git.commit.time=16.01.2016 @ 00\:04\:59 UTC
git.build.time=17.01.2016 @ 03\:49\:25 UTC
git.remote.origin.url=git@github.com\:jacques-n/drill.git
{code}


> Possible memory leak (fragmentation ?)  in rpc layer
> ----------------------------------------------------
>
>                 Key: DRILL-4266
>                 URL: https://issues.apache.org/jira/browse/DRILL-4266
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC
>    Affects Versions: 1.5.0
>            Reporter: Victoria Markman
>            Assignee: Jacques Nadeau
>         Attachments: drill.log.2016-01-12-16, memComsumption.txt, memComsumption_framework.output_Fri_Jan_15_width_per_node=4.log,
memComsumption_framework.output_Sun_Jan_17_04_jacques_branch_drill-4131, test.tar
>
>
> I have executed 5 tests from Advanced/mondrian test suite in a loop overnight.
> My observation is that direct memory steadily grew from 117MB to 1.8GB and remained on
that level for 14875 iteration of the tests.
> My question is: why do 5 queries that were able to execute with 117MB of memory require
1.8GB of memory after 5 hours of execution ?
> Attached:
> * Memory used after each test iteration : memComsumption.txt
> * Log of the framework run: drill.log.2016-01-12-16
> * Tests: test.tar
> Setup:
> {noformat}
> Single node 32 core box. 
> DRILL_MAX_DIRECT_MEMORY="4G"
> DRILL_HEAP="1G"
> 0: jdbc:drill:schema=dfs> select * from sys.options where status like '%CHANGED%';
> +-----------------------------------+----------+---------+----------+----------+-------------+-----------+------------+
> |               name                |   kind   |  type   |  status  | num_val  | string_val
 | bool_val  | float_val  |
> +-----------------------------------+----------+---------+----------+----------+-------------+-----------+------------+
> | planner.enable_decimal_data_type  | BOOLEAN  | SYSTEM  | CHANGED  | null     | null
       | true      | null       |
> +-----------------------------------+----------+---------+----------+----------+-------------+-----------+------------+
> 1 row selected (1.309 seconds)
> {noformat}
> {noformat}
> Reproduction:
> * tar xvf test.tar into Functional/test directory 
> * ./run.sh -s Functional/test -g regression -t 180 -n 5 -i 10000000 -m
> {noformat}
> This is very similar behavior as Hakim and I observed long time ago with window functions.
Now, that new allocator is in place we rerun this test and we see the similar things, and
allocator does not seem to think that we have a memory leak. Hence the speculation that memory
is leaked in RPC layer.
> I'm going to reduce planner.width.max_per_node and see if it has any effect on memory
allocation (speculating again ...)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message