hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <>
Subject [jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks
Date Wed, 24 Sep 2014 18:00:34 GMT


Sushanth Sowmyan commented on HIVE-8164:

> Since the classes in this patch are really for use outside of Hive by third party system
such as Falcon, it makes sense to note that in the file and give some info
on how a tool might use those files.

Agreed, will add info there.

> ReplicationTask.needsStagingDir - It would make more sense for this to be abstract and
let each subtype say yes/no rather than encoding the logic for each subtype here.
> ReplicationTask.getSrcWhCommands and getDstWhCommands should be abstract as well.

Of all the event types currently present in the replication design, ADD_PARTITION is the only
event type that results in needing the staging directory. Implementing this in ReplicationTask
makes it easy for other ReplicationTasks to avoid having some repetitive code. They can still
choose to @Override, so that flexibility has not been taken from them, it just there is a
default base implementation. I followed the same logic for getSrcWhCommands and getDstWhCommands
to provide a default impl of being empty, but calling verifyActionable(), but these were intended
to be overridden.

I think the default impl for needsStagingDir still has value, and would like to keep it as-is
unless you object strongly to it. The get*WhCommands though, I will go ahead and make abstract.

> At least one real implementation of ReplicationTask would make it easier to determine
whether this is the right interface or not.

Agreed. I'll upload the Add/Drop/Alter Partition ones and their test in a couple of days -
I had to rewire a bunch of them when trying to add in AlterPartition.

> Adding in a ReplicationTask that converts a Notification Event to actionable tasks
> ----------------------------------------------------------------------------------
>                 Key: HIVE-8164
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Import/Export
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: HIVE-8164.patch

This message was sent by Atlassian JIRA

View raw message