falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-94) Retention to handle hive table eviction
Date Thu, 10 Oct 2013 04:37:45 GMT

    [ https://issues.apache.org/jira/browse/FALCON-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791191#comment-13791191
] 

Srikanth Sundarrajan commented on FALCON-94:
--------------------------------------------

Generic CatalogService (AbstractCatalogService) seems to return HCatPartition, which would
mean that the CatalogService is tied to HCatalog. If that was the intention, you might consider
renaming AbstractCatalogService to AbstractHCatalogService.
{code}
+    public abstract List<HCatPartition> listPartitionsByFilter(String catalogUrl, String
database,
+                                                               String tableName, String filter)
+        throws FalconException;
{code}

We are requiring some hcatalog related jars to be copied to oozie shared lib directory. I
am assuming this will be covered in documentation. Also this would mean that Off the shelf
Oozie patched with falcon config wont work any more. We further require the shared lib dirs
to be setup and the contents copied. Are there any challenges in making these jars available
in the retention lib path by default and not requiring shared libs ? 
{code}
+                <!-- HCatalog jars -->
+                <property>
+                    <name>oozie.use.system.libpath</name>
+                    <value>true</value>
+                </property>
+                <property>
+                    <name>oozie.action.sharelib.for.java</name>
+                    <value>hcatalog</value>
+                </property>
{code}

Is "feedStorageType" a new property added to oozie coordinator for retention. If so, can this
be prefixed with "falcon.". The older ones without prefix need to be fixed some time in future
(Since that would be an incompatible change, we need to hold off on that). Meanwhile, all
new property additions can be prefixed appropriately.
{code}
+            } else if ("feedStorageType".equals(property.getName())) {
+                storageType = property.getValue();
{code}

Can ${ & ?{ be defined as constants?
{code}
+            Assert.assertEquals(feedDataPath, storage.getUriTemplate().replaceAll("\\$\\{",
"\\?\\{"));
{code}

Since the feed storage is abstracted in Storage class either as FileSystemStorage or CatalogStorage,
moreover, behaviors for listing partitions and drop partitions are listed there, why do we
need to hardcode the eviction behavior and instance deletion discovery for filesystem need
to happen in FeedEvictor ? Why can't be implemented in appropriate Storage implementation.
That way FeedEvictor would simpler and lot cleaner. Thoughts ?

> Retention to handle hive table eviction
> ---------------------------------------
>
>                 Key: FALCON-94
>                 URL: https://issues.apache.org/jira/browse/FALCON-94
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-94.patch, FALCON-94-r1.patch, FALCON-94-r2.patch
>
>
> Must handle both hive managed and external tables.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message