hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16164) Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.
Date Mon, 24 Apr 2017 08:28:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980864#comment-15980864
] 

Lefty Leverenz commented on HIVE-16164:
---------------------------------------

[~spena], this went into branch-2 shortly after branch-2.3 was cut, so it won't be in release
2.3.0 but rather 2.4.0.

If you want it in 2.3.0 you'll have to commit the patch to branch-2.3.

> Provide mechanism for passing HMS notification ID between transactional and non-transactional
listeners.
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16164
>                 URL: https://issues.apache.org/jira/browse/HIVE-16164
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>             Fix For: 2.3.0, 3.0.0
>
>         Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch, HIVE-16164.3.patch, HIVE-16164.6.patch,
HIVE-16164.7.patch, HIVE-16164.8.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS backend DB so
that external applications (such as backup apps) can request incremental notifications based
on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However, there are
sometimes that applications may be required to be in sync with the latest HMS event in order
to process an action. These applications will provide a listener implementation that is called
by the HMS after an HMS transaction happened.
> The problem is that the listener running after the transaction (or during the non-transactional
context) may need the DB event ID in order to sync all events happened previous to that event
ID, but this ID is never passed to the non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on each ListenerEvent
implementations (such as CreateTableEvent), and send the EnvironmentContext to the non-transactional
listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the ObjectStore.addNotificationEvent().
We just need to set this event ID to the EnvironmentContext from each of the event notifications
and make sure that this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
>  ms.createTable(tbl);
>   if (transactionalListeners.size() > 0) {
>     CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
>     createTableEvent.setEnvironmentContext(envContext);
>     for (MetaStoreEventListener transactionalListener : transactionalListeners) {
>       transactionalListener.onCreateTable(createTableEvent);         // <- Here the
notification ID is generated
>     }
>   }
>   success = ms.commitTransaction();
> } finally {
>   if (!success) {
>     ms.rollbackTransaction();
>     if (madeDir) {
>       wh.deleteDir(tblPath, true);
>     }
>   }
>   for (MetaStoreEventListener listener : listeners) {
>     CreateTableEvent createTableEvent =
>         new CreateTableEvent(tbl, success, this);
>     createTableEvent.setEnvironmentContext(envContext);
>     listener.onCreateTable(createTableEvent);                        // <- Here we
would like to consume notification ID
>   }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext, such as
DB_NOTIFICATION_EVENT_ID.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message