hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
Date Tue, 12 Jan 2016 23:01:40 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095163#comment-15095163
] 

Sergey Shelukhin commented on HIVE-12682:
-----------------------------------------

What does the first change do? it gets taskId from taskId

> Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12682
>                 URL: https://issues.apache.org/jira/browse/HIVE-12682
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Carter Shanklin
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-12682.1.patch, HIVE-12682.2.patch, reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here: https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached graph shows
where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message