hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Pachitariu (JIRA)" <>
Subject [jira] [Resolved] (HIVE-20523) Improve table statistics for Parquet format
Date Tue, 19 Feb 2019 15:41:00 GMT


George Pachitariu resolved HIVE-20523.
    Resolution: Duplicate

> Improve table statistics for Parquet format
> -------------------------------------------
>                 Key: HIVE-20523
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Physical Optimizer
>            Reporter: George Pachitariu
>            Assignee: George Pachitariu
>            Priority: Minor
>         Attachments: HIVE-20523.1.patch, HIVE-20523.10.patch, HIVE-20523.11.patch, HIVE-20523.12.patch,
HIVE-20523.2.patch, HIVE-20523.3.patch, HIVE-20523.4.patch, HIVE-20523.5.patch, HIVE-20523.6.patch,
HIVE-20523.7.patch, HIVE-20523.8.patch, HIVE-20523.9.patch, HIVE-20523.patch
> Right now, in the table basic statistics, the *raw data size* for a row with any data
type in the Parquet format is 1. This is an underestimated value when columns are complex
data structures, like arrays.
> Having tables with underestimated raw data size makes Hive assign less containers (mappers/reducers)
to it, making the overall query slower. 
> Heavy underestimation also makes Hive choose MapJoin instead of the ShuffleJoin that
can fail with OOM errors.
> In this patch, I compute the columns data size better, taking into account complex structures.
I followed the Writer implementation for the ORC format.

This message was sent by Atlassian JIRA

View raw message