incubator-hcatalog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <>
Subject [jira] [Updated] (HCATALOG-546) Rework HCatalog's JMS Notifications
Date Wed, 07 Nov 2012 00:16:16 GMT


Mithun Radhakrishnan updated HCATALOG-546:

    Attachment: sample.Add.Drop.Table.json

I've attached samples of JSON notifications for the following operations:
1. Create/Drop Database.
2. Create/Drop Tables.
3. Add/Drop Partitions.

Please note that the JSON messages don't contain the version-string, format-id or event-id.
I've opted instead to put those in the JMS message (as separate fields), since that makes
deserialization a lot simpler.
> Rework HCatalog's JMS Notifications 
> ------------------------------------
>                 Key: HCATALOG-546
>                 URL:
>             Project: HCatalog
>          Issue Type: Bug
>          Components: notification
>    Affects Versions: 0.4.1
>            Reporter: Mithun Radhakrishnan
>             Fix For: 0.4.1
>         Attachments: sample.Add.Drop.Database.json, sample.Add.Drop.Partition.json, sample.Add.Drop.Table.json
> In 0.4.1, the NotificationListener listens for metastore operations and emits JMS notifications
containing the entire metastore-objects (Database/Table/Partitions) in Java-serialized form.
The assumption at the time was that consumers might need access to the whole object. This
policy poses a couple of problems:
> 1. The notifications are verbose, since it conveys a bunch of information that's available
from querying the metastore anyway.
> 2. Consumers of these JMS notifications (e.g. Oozie) would now be dependent on the Java
class definitions of metastore-objects. If they change, Oozie would also need to be restarted
(with updated libs), to consume the notifications.
> Ideally, the notifications should convey only the minimum information that identifies
the metastore-change unambiguously. (Everything else can be queried for.) They should be backward
compatible. If new fields are added, existing consumers shouldn't break (unless they intend
to consume the new fields). Also, the notification-format ought to be pluggable.
> For the initial rework, we're proposing to use a JSON-string to represent the notification-content.
We're also proposing a helper-class for the likes of Oozie to use, that converts the strings
to POJOs, in a backward-compatible fashion.
> I'll attach sample notifications and a tentative patch shortly.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message