hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <>
Subject [jira] [Commented] (HIVE-4518) Counter Strike: Operation Operator
Date Mon, 16 Dec 2013 10:09:09 GMT


Lefty Leverenz commented on HIVE-4518:

This patch removed *hive.task.progress* from and hive-default.xml.template,
so I revised the Configuration Properties wiki with a version note:  "Removed in: Hive 0.13.0
with HIVE-4518" (see []
and search for "hive.task.progress").

But the patch also added ** to without defining it in
hive-default.xml.template.  Would someone please put a definition in the release note for
this ticket?  Then it can be added to hive-default.xml.template, either with the spelling
fixes in HIVE-5879 or in a follow-up ticket when the Config Props wiki gets synchronized with
hive-default.xml.template, and I'll add it to the wiki.

> Counter Strike: Operation Operator
> ----------------------------------
>                 Key: HIVE-4518
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Gunther Hagleitner
>            Assignee: Gunther Hagleitner
>             Fix For: 0.13.0
>         Attachments: HIVE-4518.1.patch, HIVE-4518.10.patch, HIVE-4518.11.patch, HIVE-4518.2.patch,
HIVE-4518.3.patch, HIVE-4518.4.patch, HIVE-4518.5.patch, HIVE-4518.6.patch.txt, HIVE-4518.7.patch,
HIVE-4518.8.patch, HIVE-4518.9.patch
> Queries of the form:
> from foo
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> Generate a huge amount of counters. The reason is that task.progress is turned on for
dynamic partitioning queries.
> The counters not only make queries slower than necessary (up to 50%) you will also eventually
run out. That's because we're wrapping them in enum values to comply with hadoop 0.17.
> The real reason we turn task.progress on is that we need CREATED_FILES and FATAL counters
to ensure dynamic partitioning queries don't go haywire.
> The counters have counter-intuitive names like C1 through C1000 and don't seem really
useful by themselves.
> With hadoop 20+ you don't need to wrap the counters anymore, each operator can simply
create and increment counters. That should simplify the code a lot.

This message was sent by Atlassian JIRA

View raw message