falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-703) Monitoring plugin that registers catalog partition
Date Wed, 10 Sep 2014 17:10:34 GMT

    [ https://issues.apache.org/jira/browse/FALCON-703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128737#comment-14128737

Venkatesh Seetharam commented on FALCON-703:

In fact, there is a jira FALCON-419 to migrate from Hcat to Hive since it does not provide
anything that we need. 

> Monitoring plugin that registers catalog partition
> --------------------------------------------------
>                 Key: FALCON-703
>                 URL: https://issues.apache.org/jira/browse/FALCON-703
>             Project: Falcon
>          Issue Type: New Feature
>            Reporter: Shwetha G S
>            Assignee: Shwetha G S
> During migration from filesystem based feeds to hcatalog based feeds, there are some
cases which are difficult to handle. For example, if any hourly feed is updated at 10th hour
to hcatalog, how do we update the dependent process that consumes the whole day of data of
that feed. Some instances of that feed for that day will be on hcatalog and some won't be.

> If we can de-couple the partition registration and assume that all partitions are already
on hcatalog even before the migration, then the whole hcatalog migration becomes easier and
the feeds can be migrated independently. We can do hcatalog partition registration by listening
to jms message for succeeded workflow. 

This message was sent by Atlassian JIRA

View raw message