hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot West <tea...@gmail.com>
Subject Re: Synchronizing Hive metastores across clusters
Date Thu, 21 Jan 2016 16:42:16 GMT
Following up on this: I've spent some time trying to evaluate the Hive
replication features but in truth it's more been an exercise in trying to
get them working! I thought I'd share my findings:

   - Conceptually this feature can sync (nearly) all Hive metadata and data
   changes between two clusters.
   - On the source cluster you require at least Hive 1.1.0
   (DbNotificationListener dependency).
   - On the destination cluster you require at least Hive 0.8.0 (IMPORT
   command dependency).
   - The environment in which you execute replication tasks requires at
   least Hive 1.2.0 (ReplicationTask dependency) although this is at the JAR
   level only (i.e. you do not need a 1.2.0 metastore running etc).
   - It is not an 'out of the box solution'; you must still write some kind
   of service that instantiates, schedules, and executes ReplicationTasks.
   This can be quite simple.
   - Exporting into S3 using Hive on EMR (AMI 4.2.0) is currently broken,
   but apparently work is underway to fix it.
   - Data inserted into Hive tables using HCatalog writers will not be
   automatically synced (HIVE-9577).
   - Mappings can be applied to destination database names, table names,
   and table and partition locations.
   - All tables at the destination are managed, even if they are external
   at the source.
   - The source and destination can be running different Hadoop
   distributions and use differing metastore database providers.
   - There is no real user level documentation.
   - It might be nice to add a Kafka based NotificationListener.

In summary it looks like quite a powerful and useful feature. However as
I'm currently running Hive 1.0.0 at my source I cannot use it in a
straightforward manner.

Thanks for your help.

Elliot.

On 18 December 2015 at 14:31, Elliot West <teabot@gmail.com> wrote:

> Eugene/Susanth,
>
> Thank you for pointing me in the direction of these features. I'll
> investigate them further to see if I can put them to good use.
>
> Cheers - Elliot.
>
> On 17 December 2015 at 20:03, Sushanth Sowmyan <khorgath@gmail.com> wrote:
>
>> Also, while I have not wiki-ized the documentation for the above, I
>> have uploaded slides from talks that I've given in hive user group
>> meetup on the subject, and also a doc that describes the replication
>> protocol followed for the EXIM replication that are attached over at
>> https://issues.apache.org/jira/browse/HIVE-10264
>>
>> On Thu, Dec 17, 2015 at 11:59 AM, Sushanth Sowmyan <khorgath@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I think that the replication work added with
>> > https://issues.apache.org/jira/browse/HIVE-7973 is exactly up this
>> > alley.
>> >
>> > Per Eugene's suggestion of MetaStoreEventListener, this replication
>> > system plugs into that and gets you a stream of notification events
>> > from HCatClient for the exact purpose you mention.
>> >
>> > There's some work still outstanding on this task, most notably
>> > documentation (sorry!) but please have a look at
>> > HCatClient.getReplicationTasks(...) and
>> > org.apache.hive.hcatalog.api.repl.ReplicationTask. You can plug in
>> > your implementation of  ReplicationTask.Factory to inject your own
>> > logic for how to handle the replication according to your needs.
>> > (currently there exists an implementation that uses Hive EXPORT/IMPORT
>> > to perform replication - you can look at the code for this, and the
>> > tests for these classes to see how that is achieved. Falcon already
>> > uses this to perform cross-hive-warehouse replication)
>> >
>> >
>> > Thanks,
>> >
>> > -Sushanth
>> >
>> > On Thu, Dec 17, 2015 at 11:22 AM, Eugene Koifman
>> > <ekoifman@hortonworks.com> wrote:
>> >> Metastore supports MetaStoreEventListener and MetaStorePreEventListener
>> >> which may be useful here
>> >>
>> >> Eugene
>> >>
>> >> From: Elliot West <teabot@gmail.com>
>> >> Reply-To: "user@hive.apache.org" <user@hive.apache.org>
>> >> Date: Thursday, December 17, 2015 at 8:21 AM
>> >> To: "user@hive.apache.org" <user@hive.apache.org>
>> >> Subject: Synchronizing Hive metastores across clusters
>> >>
>> >> Hello,
>> >>
>> >> I'm thinking about the steps required to repeatedly push Hive datasets
>> out
>> >> from a traditional Hadoop cluster into a parallel cloud based cluster.
>> This
>> >> is not a one off, it needs to be a constantly running sync process. As
>> new
>> >> tables and partitions are added in one cluster, they need to be synced
>> to
>> >> the cloud cluster. Assuming for a moment that I have the HDFS data
>> syncing
>> >> working, I'm wondering what steps I need to take to reliably ship the
>> >> HCatalog metadata across. I use HCatalog as the point of truth as to
>> when
>> >> when data is available and where it is located and so I think that
>> metadata
>> >> is a critical element to replicate in the cloud based cluster.
>> >>
>> >> Does anyone have any recommendations on how to achieve this in
>> practice? One
>> >> issue (of many I suspect) is that Hive appears to store table/partition
>> >> locations internally with absolute, fully qualified URLs, therefore
>> unless
>> >> the target cloud cluster is similarly named and configured some path
>> >> transformation step will be needed as part of the synchronisation
>> process.
>> >>
>> >> I'd appreciate any suggestions, thoughts, or experiences related to
>> this.
>> >>
>> >> Cheers - Elliot.
>> >>
>> >>
>>
>
>

Mime
View raw message