hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
Date Tue, 15 Dec 2015 22:19:46 GMT

     [ https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gopal V reassigned HIVE-12682:
------------------------------

    Assignee: Gopal V

> Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12682
>                 URL: https://issues.apache.org/jira/browse/HIVE-12682
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Carter Shanklin
>            Assignee: Gopal V
>         Attachments: reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here: https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached graph shows
where time was spent during the reducer phase.
> Problem seems to relate to https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message