falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-93) Replication to handle hive table replication
Date Thu, 10 Oct 2013 22:32:42 GMT

    [ https://issues.apache.org/jira/browse/FALCON-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792056#comment-13792056

Venkatesh Seetharam commented on FALCON-93:

bq. Can prefix be "falcon.source." & "falcon.target." instead of just source & target?

bq. Looks like all tables go through the same export path. Can export & import be avoided
for external tables or does the export/import already take care of the fact that table is
external and allow you to short-circuit this?
Good question. This will be like a split-brain problem. Currently it uses the same path for
both table types. In the future, import and export can employ hdfs snapshots so its 0 copy
and is extremely fast and efficient. IMHO, that would be the place to address this issue.

bq. Seems to be using distcp-v1. This is not desirable.
Yes. I had mentioned this in a previous comment that I could not get FeedReolicator to work.
Will try using FeedReplicator.

bq. Looks like scenario where data from multiple sources each owning a partition getting merged
in the target cluster isn't implemented, as the export need to be specific to the partition
against each of the source cluster. Please confirm.
This is not an issue with HCatalog since the complete partition specification is part of the
URI. Say there are 2 partitions, date and region. The URI will be: hcat://localhost:9083/database/table/ds=${date}/region=${region}

Makes sense?

> Replication to handle hive table replication
> --------------------------------------------
>                 Key: FALCON-93
>                 URL: https://issues.apache.org/jira/browse/FALCON-93
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-93.patch, FALCON-93-r1.patch
> Data and metadata to be replicated atomically.

This message was sent by Atlassian JIRA

View raw message