falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 32688: FALCON-1091 Monitoring plugin code
Date Wed, 01 Apr 2015 05:59:39 GMT


> On March 31, 2015, 4:36 p.m., Srikanth Sundarrajan wrote:
> > common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java, line
191
> > <https://reviews.apache.org/r/32688/diff/1/?file=911160#file911160line191>
> >
> >     Understand that this will handle the cases of re-run where data is re-added.
In that scenario, if drop partitions are complete, but add partition where to fail mid way,
would it leave the table in a bad state ? This might be a large gap, if this isn't atomic.
If this is indeed a valid issue, we need to address this. But given the largish nature of
the fix, we can file another JIRA and handle this separately.

Hive 0.14 supports transactional insert, delete and update. Meaning, inserts, updates and
deletes can happen in batches - https://issues.apache.org/jira/browse/HIVE-5317. This we should
be leveraging at the earliest.

They also have a plan to come up with support for user demarcated transaction boundaries that
we can leverage later - 
https://issues.apache.org/jira/browse/HIVE-9675

Have filed FALCON-1130 for the same.


- Pallavi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32688/#review78368
-----------------------------------------------------------


On March 31, 2015, 11:35 a.m., Srikanth Sundarrajan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32688/
> -----------------------------------------------------------
> 
> (Updated March 31, 2015, 11:35 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1091
>     https://issues.apache.org/jira/browse/FALCON-1091
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FALCON-1091 Monitoring plugin code
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93

>   common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46 
>   common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95 
>   common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java 1ba7b9d 
>   common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb 
>   common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java 33ec59c 
>   common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java 4bf6e00 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a

>   common/src/main/resources/startup.properties 99dab59 
>   common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc 
>   common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161 
>   hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java
2167375 
>   oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java
7a87919 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 462e26b

>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
545beb1 
>   prism/pom.xml 4a3054a 
>   retention/src/test/java/org/apache/falcon/retention/FeedEvictorTest.java 970d381 
>   test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de 
>   test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b 
>   webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION

>   webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9 
>   webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java 6982b65

>   webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9 
>   webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a 
>   webapp/src/test/resources/cluster-template.xml 16b7c8c 
>   webapp/src/test/resources/feed-template1.xml 456f7ce 
>   webapp/src/test/resources/feed-template2.xml d4901fa 
> 
> Diff: https://reviews.apache.org/r/32688/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Srikanth Sundarrajan
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message