drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jean-claude (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-4278) Memory leak when using LIMIT
Date Mon, 18 Jan 2016 01:10:39 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104019#comment-15104019
] 

jean-claude edited comment on DRILL-4278 at 1/18/16 1:09 AM:
-------------------------------------------------------------

I did another test using 18 JSON files containing each a single object with a single field.

{
	"name" : "lajsdlfjsklfjsdlkfjsldf jslfslkjfklsdj fklsjflksdjflksdjflja;kjflkdsjfkljdfsf 
d fkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsadkjfksadjfkj"
}

If run this query thousands of times the heap usage goes up and up.
select name from dfs.`/Users/jccote/json/` limit 4;

On the other hand if I run this query (the limit is the same as the size of the db so there
are no rows actually removed) thousands of times there is absolutely no problem.
select name from dfs.`/Users/jccote/json/` limit 18;

There's definitely something leaking when rows are taken out of the result set.





was (Author: jccote):
I did another test using 18 JSON files containing each a single object with a single field.

{
	"name" : "lajsdlfjsklfjsdlkfjsldf jslfslkjfklsdj fklsjflksdjflksdjflja;kjflkdsjfkljdfsf 
d fkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsdfjdalkfjasjfaldsjfljfkldajsfkljaklfjksajfjaslfjakljfkljfkldsajfkldsjfksdjfklsajfkljklasjfkldsjklfjkdlsjfklsdjfkljsdklafjkldjafkldj
asklfjskldafjklsdjfjdsfkjdsfkjsadkjfksadjfkj"
}

If run this query thousands of times the heap usage goes up and up.
select * from dfs.`/Users/jccote/json/` limit 4;

On the other hand if I run this query (the limit is the same as the size of the db so there
are no rows actually removed) thousands of times there is absolutely no problem.
select * from dfs.`/Users/jccote/json/` limit 18;

There's definitely something leaking when rows are taken out of the result set.




> Memory leak when using LIMIT
> ----------------------------
>
>                 Key: DRILL-4278
>                 URL: https://issues.apache.org/jira/browse/DRILL-4278
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC
>    Affects Versions: 1.4.0, 1.5.0
>         Environment: OS X
> 0: jdbc:drill:zk=local> select * from sys.version;
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> | version  |                 commit_id                 |                   commit_message
                   |        commit_time         |        build_email         |         build_time
        |
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> | 1.4.0    | 32b871b24c7b69f59a1d2e70f444eed6e599e825  | [maven-release-plugin] prepare
release drill-1.4.0  | 08.12.2015 @ 00:24:59 PST  | venki.korukanti@gmail.com  | 08.12.2015
@ 01:14:39 PST  |
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> 0: jdbc:drill:zk=local> select * from sys.options where status <> 'DEFAULT';
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> |            name             | kind  |  type   |  status  | num_val  | string_val  |
bool_val  | float_val  |
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> | planner.slice_target        | LONG  | SYSTEM  | CHANGED  | 10       | null        |
null      | null       |
> | planner.width.max_per_node  | LONG  | SYSTEM  | CHANGED  | 5        | null        |
null      | null       |
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> 2 rows selected (0.16 seconds)
>            Reporter: jean-claude
>
> copy the parquet files in the samples directory so that you have a 12 or so
> $ ls -lha /apache-drill-1.4.0/sample-data/nationsMF/
> nationsMF1.parquet
> nationsMF2.parquet
> nationsMF3.parquet
> create a file with a few thousand lines like these
> select * from dfs.`/Users/jccote/apache-drill-1.4.0/sample-data/nationsMF` limit 500;
> start drill
> $ /apache-drill-1.4.0/bin/drill-embeded
> reduce the slice target size to force drill to use multiple fragment/threads
> jdbc:drill:zk=local> system set planner.slice_target=10;
> now run the list of queries from the file your created above
> jdbc:drill:zk=local> !run /Users/jccote/test-memory-leak-using-limit.sql
> the java heap space keeps going up until the old space is at 100% and eventually you
get an OutOfMemoryException in drill
> $ jstat -gccause 86850 5s
>   S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT    LGCC
                GCC                 
>   0.00   0.00 100.00 100.00  98.56  96.71   2279   26.682   240  458.139  484.821 GCLocker
Initiated GC Ergonomics          
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   242  461.347  488.028 Allocation
Failure   Ergonomics          
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   245  466.630  493.311 Allocation
Failure   Ergonomics          
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   247  470.020  496.702 Allocation
Failure   Ergonomics          
> If you do the same test but do not use the LIMIT then the memory usage does not go up.
> If you add a where clause so that no results are returned, then the memory usage does
not go up.
> Something with the RPC layer?
> Also it seems sensitive to the number of fragments/threads. If you limit it to one fragment/thread
the memory usage goes up much slower.
> I have used parquet files and CSV files. In either case the behaviour is the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message