Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Thu, 4 May 2017 17:12:04 +0000 (UTC)
From: "Paul Rogers (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13069203.1493916600000.128627.1493917924157@Atlassian.JIRA>
In-Reply-To: <JIRA.13069203.1493916600000@Atlassian.JIRA>
References: <JIRA.13069203.1493916600000@Atlassian.JIRA> <JIRA.13069203.1493916600577@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-5472) Parquet reader generating
 low-density batches causing Sort operator to spill un-necessarily
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 04 May 2017 17:12:08 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997100#comment-15997100 ] 

Paul Rogers commented on DRILL-5472:
------------------------------------

This is a known issue with Parquet, but one that is not currently a high priority.

The thought here is that this issue will be resolved as a side-effect of the fix for DRILL-5211. For that bug, we must limit vector sizes to 16 MB. At present, the Parquet reader tries, but fails, to limit vector sizes. That failure causes random vector sizes and low density. Fixing the Parquet vector limit to avoid fragmentation will also, perhaps, reduced the low-density issue without the issue itself having to be a high priority.

> Parquet reader generating low-density batches causing Sort operator to spill un-necessarily
> -------------------------------------------------------------------------------------------
>
>                 Key: DRILL-5472
>                 URL: https://issues.apache.org/jira/browse/DRILL-5472
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators, Storage - Parquet
>            Reporter: Rahul Challapalli
>            Assignee: Paul Rogers
>         Attachments: drill5472.log, drill5472.parquet, drill5472.sys.drill
>
>
> git.commit.id.abbrev=1e0a14c
> The parquet file used in the below query is ~20MB. The uncompressed size id ~1.2 GB. Now the below query has a sort which is given ~6GB memory for a single fragment and yet it spills.
> {code}
> select * from (select * from dfs.`/drill/testdata/resource-manager/all_types_large` s order by s.missing12.x) d where d.missing3 is false;
> {code}
> The profile indicates that the above query has spilled twice. Attached the profile and the logs


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)