hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roshan Naik (JIRA)" <>
Subject [jira] [Commented] (HIVE-5138) Streaming - Web HCat API
Date Tue, 17 Sep 2013 05:20:56 GMT


Roshan Naik commented on HIVE-5138:

Thanks [~ekoifman] for the comments:

h5. On Pt 1.
 Thanks. I need to take a closer look at this.

h5. On Pt 2.
 I think you mean 'safe to invoke concurrently' instead of 'atomic', since the intermediate
states are going to be visible when an operation spans both file system and meta store. Here
is a summary of the reasons why each operation is concurrency safe:

 - *streamingStatus* : Readonly metastore operation
 - *chunkGet* : This is an atomic metastore operation 
 - *chunkAbort* : Just deletes a file. So no concurrency issues here.
 - *chunkCommit* : Just renames a file. So only one of concurrent operations will succeed.
 - *disableStreaming* : This is an atomic metastore operation 
 - *enableStreaming* : Does a couple of mkdirs (for setup) followed by an atomic metastore
operation. mkdirs() is idempotent, so all concurrent calls succeed. All concurrent invocations
enter a transaction to do the metastore update atomically...only one should update metastore.

 - *partitionRoll* : Creates empty dir for the new current partition & then atomically
updates metastore as follows:
   -# Make note of this new current partition dir
   -# Do an addPartition() on the previous current partition. 

- If concurrent partitionRoll() invocations use same arguments, the addPartition() step will
fail on all but one. If arguments are not same in concurrent invocations, they all succeed
and updates made by the last invocation to exit the metastore transaction would override the

> Streaming - Web HCat  API
> -------------------------
>                 Key: HIVE-5138
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: HCatalog, Metastore, WebHCat
>            Reporter: Roshan Naik
>            Assignee: Roshan Naik
>         Attachments: HIVE-4196.v2.patch

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message