falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shweth...@gmail.com
Subject Re: Review Request 29092: FALCON-943 process update copying user lib is very slow
Date Mon, 22 Dec 2014 10:42:35 GMT


> On Dec. 21, 2014, 4:19 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java,
line 1200
> > <https://reviews.apache.org/r/29092/diff/2/?file=795946#file795946line1200>
> >
> >     Does it make sense to return this effective time and surface it up all the way
back to the user ? It seems like it might be useful, but not fully convinced myself.

If we use coord update in future for entity update, we will not be able to support effective
time. Hence removed it altogether


> On Dec. 21, 2014, 4:19 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java,
line 122
> > <https://reviews.apache.org/r/29092/diff/2/?file=795944#file795944line122>
> >
> >     Are we cleaning up automatically previous installation artefacts ? 
> >     
> >     Generally it might be useful to log when files are deleted.
> >     
> >     This JIRA seems to be slipping in many changes albeit unrelated.

The working path of the cluster that contains the third party jars has just the jar names
without any version and its difficult to debug without jar versions. This just cleans up the
jar without version. I can rollback this and address it in another jira


> On Dec. 21, 2014, 4:19 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java, line 76
> > <https://reviews.apache.org/r/29092/diff/2/?file=795936#file795936line76>
> >
> >     This is a new property. Wouldn't this result in NPE if the config is missed
? How do we ensure that necessary config steps are listed before migrating to the version
with this change.

Yes it does. During the upgrade, the guildeline is that they should take startup properties
and make any changes required


> On Dec. 21, 2014, 4:19 p.m., Srikanth Sundarrajan wrote:
> > common/src/main/java/org/apache/falcon/entity/ProcessHelper.java, line 86
> > <https://reviews.apache.org/r/29092/diff/2/?file=795924#file795924line86>
> >
> >     They dont seem to be related to rollback of the strict process versioning. Why
are they being removed ?

Yes, they are related to copying user workflow and lib to staging directory. Without that,
it will just be process workflow and lib path and hence these functions are not required


> On Dec. 21, 2014, 4:19 p.m., Srikanth Sundarrajan wrote:
> > common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java, line 263
> > <https://reviews.apache.org/r/29092/diff/2/?file=795923#file795923line263>
> >
> >     is this moved to FeedHelper ? If not, can you move it there? Seems like an useful
func.

Its already in Feedhelper


- shwethags


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29092/#review65743
-----------------------------------------------------------


On Dec. 18, 2014, 10:32 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29092/
> -----------------------------------------------------------
> 
> (Updated Dec. 18, 2014, 10:32 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-943
>     https://issues.apache.org/jira/browse/FALCON-943
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Addresses the following:
> 1. Removed copying user workflow/lib to staging directory
> 2. Removed update at specific time
> 3. Copying falcon jars to parent workflow lib copies only specific falcon jars
> 4. Libs copied at cluster submit(3rd party jars) have versions so that its easier to
debug issues
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/cli/FalconCLI.java 93776d3 
>   client/src/main/java/org/apache/falcon/client/FalconClient.java 23c8943 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 59e43fb 
>   common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java bbe274b 
>   common/src/main/java/org/apache/falcon/entity/ProcessHelper.java 174f8f6 
>   common/src/main/java/org/apache/falcon/update/UpdateHelper.java 5a86ae3 
>   common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
ae158f7 
>   common/src/main/resources/startup.properties 433c2a8 
>   common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java ef0d769 
>   docs/src/site/twiki/FalconCLI.twiki 202af63 
>   docs/src/site/twiki/FalconDocumentation.twiki 0d2a692 
>   docs/src/site/twiki/restapi/EntityUpdate.twiki f2c2e7e 
>   oozie-el-extensions/src/main/conf/oozie-site.xml PRE-CREATION 
>   oozie-el-extensions/src/test/resources/oozie-site.xml a106aa6 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java c73401a 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 2ceb91e 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java e341fb8 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
771295f 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 3347fbf 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java
1db4ca4 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java
14668f0 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java
6bd5dd8 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java 8691ee5

>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
d271695 
>   oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java 9567c5f

>   oozie/src/main/java/org/apache/falcon/workflow/LateDataHandler.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 745aaac

>   oozie/src/main/resources/action/post-process.xml 440a131 
>   oozie/src/main/resources/action/pre-process.xml 070c42b 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
3477258 
>   prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java b6e1cec 
>   prism/src/main/java/org/apache/falcon/resource/proxy/SchedulableEntityManagerProxy.java
fb9d9f3 
>   rerun/src/main/java/org/apache/falcon/latedata/LateDataHandler.java d35abfa 
>   rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunConsumer.java 9ba632e

>   src/conf/startup.properties 2db4b1e 
>   src/main/assemblies/distributed-package.xml 7320312 
>   src/main/assemblies/standalone-package.xml f9d6009 
>   src/main/examples/data/generate.sh 74ec3ba 
>   webapp/src/main/java/org/apache/falcon/Debug.java c606074 
>   webapp/src/main/java/org/apache/falcon/resource/ConfigSyncService.java 98b8b3d 
>   webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java a3acbdb 
>   webapp/src/test/java/org/apache/falcon/late/LateDataHandlerIT.java 1885bb7 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java f7e6bdb

> 
> Diff: https://reviews.apache.org/r/29092/diff/
> 
> 
> Testing
> -------
> 
> UTs, end to end test for process, retention and replication 
> 
> 
> Thanks,
> 
> shwethags
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message