hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-21115) Add support for object versions in metastore
Date Wed, 16 Jan 2019 01:13:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743500#comment-16743500
] 

Vihang Karajgaonkar edited comment on HIVE-21115 at 1/16/19 1:12 AM:
---------------------------------------------------------------------

Thanks [~odraese]. You make a fair point. I took a look at the code you pointed above and
here are lines which are interesting with respect to generating the version numbers using
transaction handler.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589

{code}
     String s = sqlGenerator.addForUpdateClause("select ntxn_next from NEXT_TXN_ID");
      LOG.debug("Going to execute query <" + s + ">");
      rs = stmt.executeQuery(s);
      if (!rs.next()) {
        throw new MetaException("Transaction database not properly " +
                "configured, can't find next transaction id.");
      }
      long first = rs.getLong(1);
      s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
      LOG.debug("Going to execute update <" + s + ">");
      stmt.executeUpdate(s);
{code}

Although in theory we can use this same logic to generate object versions but this is different
which can have potentially severe performance degradation. Note that the transaction_ids are
unique across all the objects and instances of metastores. This means all the transactions
are serialized to generate these ids in metastore. While this could be used to generate a
globally incrementing version number it may have some adverse performance implications. Having
a version at a per-object level instead reduces the scope of the lock and only the transactions
which are operating on the same object will be serialized. I think as a mid-way we can implement
the versioning for non-acid enabled tables. If the table is ACID enabled, it should use the
transaction ids for its versioning. I am hesitant to use this logic above for all the objects
which indirectly is going to serialize all the DML transactions in metastore. Perhaps someone
more familiar with this code can comment on my thoughts above. [~ekoifman] [~alangates]?


was (Author: vihangk1):
Thanks [~odraese]. You make a fair point. I took a look the you pointed above and here are
lines which are interesting with respect to generating the version numbers using transaction
handler.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589

{code}
     String s = sqlGenerator.addForUpdateClause("select ntxn_next from NEXT_TXN_ID");
      LOG.debug("Going to execute query <" + s + ">");
      rs = stmt.executeQuery(s);
      if (!rs.next()) {
        throw new MetaException("Transaction database not properly " +
                "configured, can't find next transaction id.");
      }
      long first = rs.getLong(1);
      s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
      LOG.debug("Going to execute update <" + s + ">");
      stmt.executeUpdate(s);
{code}

Although in theory we can use this same logic to generate object versions but this is different
which can have potentially severe performance degradation. Note that the transaction_ids are
unique across all the objects and instances of metastores. This means all the transactions
are serialized to generate these ids in metastore. While this could be used to generate a
globally incrementing version number it may have some adverse performance implications. Having
a version at a per-object level instead reduces the scope of the lock and only the transactions
which are operating on the same object will be serialized. I think as a mid-way we can implement
the versioning for non-acid enabled tables. If the table is ACID enabled, it should use the
transaction ids for its versioning. I am hesitant to use this logic above for all the objects
which indirectly is going to serialize all the DML transactions in metastore. Perhaps someone
more familiar with this code can comment on my thoughts above. [~ekoifman] [~alangates]?

> Add support for object versions in metastore
> --------------------------------------------
>
>                 Key: HIVE-21115
>                 URL: https://issues.apache.org/jira/browse/HIVE-21115
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Vihang Karajgaonkar
>            Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg. catName, dbName
and tblName for a table is unique). Once a table or partition is created it could be altered
in many ways. There is no good way currently to identify the version of the object once it
is altered. For example, suppose there are two clients (Hive and Impala) using the same metastore.
Once some alter operations are performed by a client, another client which wants to do a alter
operation has no good way to know if the object which it has is the same as the one stored
in metastore. Metastore updates the {{transient_lastDdlTime}} every time there is a DDL operation
on the object. However, this value cannot be relied for all the clients since after HIVE-1768
metastore updates the value only when it is not set in the parameters. It is possible that
a client which alters the object state, does not remove the {{transient_lastDdlTime}} and
metastore will not update it. Secondly, if there is a clock skew between multiple HMS instances
when HMS-HA is configured, time values cannot be relied on to find out the sequence of alter
operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus  http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html
to generate a incrementing sequence number every time a object is altered. The value of this
object can be set as one of the values in the parameters. The advantage of using Datanucleus
the versioning can be done across HMS instances as part of the database transaction and it
should work for all the supported databases.
> In theory such a version can be used to detect if the client is presenting a object which
is "stale" when issuing a alter request. Metastore can choose to reject such a alter request
since the client may be caching a old version of the object and any alter operation on such
stale object can potentially overwrite previous operations. However, this is can be done in
a separate JIRA.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message