hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5936) analyze command failing to collect stats with counter mechanism
Date Thu, 05 Dec 2013 22:52:35 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840659#comment-13840659
] 

Ashutosh Chauhan commented on HIVE-5936:
----------------------------------------

I tested HIVE-5916 on a table having 2K partitions. It used to fail earlier because we used
to run out of counters earlier. Now analyze command succeeds since we lowered the number of
unique counters required. However, as you pointed out in HIVE-5916 review we still potentially
can run into counter name limit for which we need to generate unique smaller string which
is implemented in this patch. So, it will be good to shorten counter names when needed.
I also noted that client update the partition object one by one from client to metastore,
which is both slow as well as error-prone. We should batch update partitions from client at
end of job. Thats should be another jira.

> analyze command failing to collect stats with counter mechanism
> ---------------------------------------------------------------
>
>                 Key: HIVE-5936
>                 URL: https://issues.apache.org/jira/browse/HIVE-5936
>             Project: Hive
>          Issue Type: Bug
>          Components: Statistics
>    Affects Versions: 0.13.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Navis
>         Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, HIVE-5936.3.patch.txt,
HIVE-5936.4.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails with NPE.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message