hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-24403) change min_history_level schema change to be compatible with previous version
Date Wed, 25 Nov 2020 10:12:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24403?focusedWorklogId=516590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516590
]

ASF GitHub Bot logged work on HIVE-24403:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 25/Nov/20 10:11
            Start Date: 25/Nov/20 10:11
    Worklog Time Spent: 10m 
      Work Description: deniskuzZ commented on a change in pull request #1688:
URL: https://github.com/apache/hive/pull/1688#discussion_r530253708



##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -5094,6 +5153,99 @@ public void countOpenTxns() throws MetaException {
     }
   }
 
+  /**
+   * Add min history level entry for each generated txn record
+   * @param dbConn Connection
+   * @param txnIds new transaction ids
+   * @deprecated Remove this method when min_history_level table is dropped
+   * @throws SQLException ex
+   */
+  @Deprecated
+  private void addTxnToMinHistoryLevel(Connection dbConn, List<Long> txnIds, long minOpenTxnId)
throws SQLException {
+    if (!useMinHistoryLevel) {
+      return;
+    }
+    // Need to register minimum open txnid for current transactions into MIN_HISTORY table.
+    try (Statement stmt = dbConn.createStatement()) {
+
+      List<String> rows = txnIds.stream().map(txnId -> txnId + ", " + minOpenTxnId).collect(Collectors.toList());
+
+      // Insert transaction entries into MIN_HISTORY_LEVEL.
+      List<String> inserts =
+          sqlGenerator.createInsertValuesStmt("\"MIN_HISTORY_LEVEL\" (\"MHL_TXNID\", \"MHL_MIN_OPEN_TXNID\")",
rows);
+      for (String insert : inserts) {
+        LOG.debug("Going to execute insert <" + insert + ">");
+        stmt.execute(insert);
+      }
+      LOG.info("Added entries to MIN_HISTORY_LEVEL for current txns: (" + txnIds + ") with
min_open_txn: " + minOpenTxnId);
+    } catch (SQLException e) {
+      if (dbProduct.isTableNotExists(e)) {
+        // If the table does not exists anymore, we disable the flag and start to work the
new way
+        // This enables to switch to the new functionality without a restart
+        useMinHistoryLevel = false;

Review comment:
       Why is this needed? Are you covering the case that schema change is done while old
HMS is still running? Lot's of code duplication, can you wrap needed methods with aspect?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 516590)
    Time Spent: 1h 50m  (was: 1h 40m)

> change min_history_level schema change to be compatible with previous version
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-24403
>                 URL: https://issues.apache.org/jira/browse/HIVE-24403
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore
>            Reporter: Peter Varga
>            Assignee: Peter Varga
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> In some configurations the HMS backend DB is used by HMS services with different versions. 
>  HIVE-23107 dropped the min_history_level table from the backend DB making the new schema
version incompatible with the older HMS services. 
>  It is possible to modify that change to keep the compatibility
>  * Keep the min_history_level table
>  * Add the new fields for the compaction_queue the same way
>  * Create a feature flag for min_history_level and if it is on
>  * Keep the logic inserting to the table during openTxn
>  * Keep the logic removing the records at commitTxn and abortTxn
>  * Change the logic in the cleaner, to get the highwatermark the old way
>  * But still change it to not start the cleaning before that
>  * The txn_to_write_id table cleaning can work the new way in the new version and the
old way in the old version
>  * This feature flag can be automatically setup based on the existence of the min_history
level table, this way if the table will be dropped all HMS-s can switch to the new functionality
without restart



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message