hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2144) reduce workload generated by JDBCStatsPublisher
Date Mon, 23 May 2011 21:19:49 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038236#comment-13038236
] 

jiraposter@reviews.apache.org commented on HIVE-2144:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/765/#review709
-----------------------------------------------------------



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
<https://reviews.apache.org/r/765/#comment1417>

    can you add a comment on what situation this exception will be thrown? Just for the sake
of reader that didn't notice there is a primary key constraint in the DDL. 



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
<https://reviews.apache.org/r/765/#comment1418>

    remove this?


- Ning


On 2011-05-21 01:49:07, Tomasz Nykiel wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/765/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-05-21 01:49:07)
bq.  
bq.  
bq.  Review request for hive.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Currently, the JDBCStatsPublisher executes two queries per inserted row of statistics,
first query to check if the ID was inserted by another task, and second query to insert a
new or update the existing row.
bq.  The latter occurs very rarely, since duplicates most likely originate from speculative
failed tasks.
bq.  
bq.  Currently the schema of the stat table is the following:
bq.  
bq.  PARTITION_STAT_TABLE ( ID VARCHAR(255), ROW_COUNT BIGINT ) and does not have any integrity
constraints declared.
bq.  
bq.  We amend it to:
bq.  
bq.  PARTITION_STAT_TABLE ( ID VARCHAR(255) PRIMARY KEY , ROW_COUNT BIGINT ).
bq.  
bq.  HIVE-2144 improves on performance by greedily performing the insertion statement.
bq.  Then instead of executing two queries per row inserted, we can execute one INSERT query.
bq.  In the case primary key constraint violation, we perform a single UPDATE query.
bq.  The UPDATE query needs to check the condition, if the currently inserted stats are "newer"
then the ones already in the table.
bq.  
bq.  
bq.  This addresses bug HIVE-2144.
bq.      https://issues.apache.org/jira/browse/HIVE-2144
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 1125468

bq.    trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisher.java PRE-CREATION

bq.  
bq.  Diff: https://reviews.apache.org/r/765/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  TestStatsPublisher JUnit test:
bq.  - basic behaviour
bq.  - multiple updates
bq.  - cleanup of the statistics table after aggregation
bq.  
bq.  Standalone testing on the cluster.
bq.  - insert/analyze queries over non-partitioned/partitioned tables
bq.  
bq.  NOTE. For the correct behaviour, the primary_key index needs to be created, or the PARTITION_STAT_TABLE
table dropped - which triggers creation of the table with the constraint declared.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Tomasz
bq.  
bq.



> reduce workload generated by JDBCStatsPublisher
> -----------------------------------------------
>
>                 Key: HIVE-2144
>                 URL: https://issues.apache.org/jira/browse/HIVE-2144
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ning Zhang
>            Assignee: Tomasz Nykiel
>         Attachments: HIVE-2144.1.patch, HIVE-2144.patch
>
>
> In JDBCStatsPublisher, we first try a SELECT query to see if the specific ID was inserted
by another task (mostly likely a speculative or previously failed task). Depending on if the
ID is there, an INSERT or UPDATE query was issues. So there are basically 2x of queries per
row inserted into the intermediate stats table. This workload could be reduced to 1/2 if we
insert it anyway (it is very rare that IDs are duplicated) and use a different SQL query in
the aggregation phase to dedup the ID (e.g., using group-by and max()). The benefits are that
even though the aggregation query is more expensive, it is only run once per query. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message