falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 36538: Falcon Unit which supports Submit and Schedule of jobs (contributed by Pavan)
Date Tue, 21 Jul 2015 06:30:29 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36538/#review92375
-----------------------------------------------------------



unit/src/main/java/org/apache/falcon/unit/FalconUnit.java (line 50)
<https://reviews.apache.org/r/36538/#comment146504>

    Can we call this LocalFalcon, instead?



unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java (line 45)
<https://reviews.apache.org/r/36538/#comment146509>

    Would be good if this class extends FalconClient and overrides/adds methods. This will
enable users to just switch between LocalFalcon and the remote instance.



unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java (line 47)
<https://reviews.apache.org/r/36538/#comment146505>

    Can we call this LocalFalconClient?



unit/src/main/java/org/apache/falcon/unit/LocalFalconClientProtocolProvider.java (line 56)
<https://reviews.apache.org/r/36538/#comment146513>

    Handle thread safety. Else, you might end up with multiple LocalJobRunners.



unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java (line 112)
<https://reviews.apache.org/r/36538/#comment146515>

    Since colo and cluster are the only 2 props we allow users to modify. Can the method take
explicitly?
    
    The reason I say that is if it is open-ended properties, there will documentation overhead
and validations to ensure users pass the right props.



unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java (line 258)
<https://reviews.apache.org/r/36538/#comment146518>

    Its a good idea to allow for a generic predicate. However, can we add method waitForStatus
that uses this method? Instead of asking users to specify it, because most often the users
will wait on Status.



unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java (line 39)
<https://reviews.apache.org/r/36538/#comment146519>

    As discussed in the demo, we should hide this.



unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java (line 42)
<https://reviews.apache.org/r/36538/#comment146520>

    As mentioned in the comment, we can pull this predicate up into the FalconUnitTestBase
as another method.



unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java (line 51)
<https://reviews.apache.org/r/36538/#comment146521>

    Why assert again, when you have waited on that predicate?



unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java (line 53)
<https://reviews.apache.org/r/36538/#comment146522>

    Don't expose the FS as jailFS, just have a method, getFileSystem that returns the underlying
FS.


- Pallavi Rao


On July 16, 2015, 5:41 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36538/
> -----------------------------------------------------------
> 
> (Updated July 16, 2015, 5:41 a.m.)
> 
> 
> Review request for Falcon and Pallavi Rao.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Submitted to Review Board on behalf of Pavan as he is on vacation for the 3 days.
> 
> Falcon unit will be able to help to validate the process, feed and cluster xml's and
as well as workflows of oozie. It will also help to submit schedule of jobs and check the
job status as well. Falcon Unit supports two modes 
> 1) Local Mode : It will bring up local oozie , use local file system and uses Local Job
Runner to run the jobs
> 2) Cluster Mode: Falcon unit Api's can also be run by pointing to external oozie, hdfs
and jobtracker.
> Documentation of falcon unit is provided in https://issues.apache.org/jira/browse/FALCON-1296
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 63dfb9d 
>   common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java 59b0910

>   common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java b5f531a

>   common/src/main/java/org/apache/falcon/workflow/util/OozieConstants.java PRE-CREATION

>   hadoop-dependencies/src/main/java/org/apache/falcon/hadoop/JailedFileSystem.java 7156bbd

>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 03063f4 
>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieClientFactory.java 622238a

>   oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 4085b8f

>   oozie/src/main/java/org/apache/oozie/client/LocalOozieClientBundle.java PRE-CREATION

>   oozie/src/main/java/org/apache/oozie/client/LocalProxyOozieClient.java PRE-CREATION

>   pom.xml 31997e8 
>   unit/pom.xml PRE-CREATION 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnit.java PRE-CREATION 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java PRE-CREATION 
>   unit/src/main/java/org/apache/falcon/unit/FalconUnitHelper.java PRE-CREATION 
>   unit/src/main/java/org/apache/falcon/unit/LocalFalconClientProtocolProvider.java PRE-CREATION

>   unit/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider
PRE-CREATION 
>   unit/src/main/resources/core-site.xml PRE-CREATION 
>   unit/src/main/resources/deploy.properties PRE-CREATION 
>   unit/src/main/resources/localoozie-log4j.properties PRE-CREATION 
>   unit/src/main/resources/log4j.xml PRE-CREATION 
>   unit/src/main/resources/mapred-site.xml PRE-CREATION 
>   unit/src/main/resources/oozie-site.xml PRE-CREATION 
>   unit/src/main/resources/startup.properties PRE-CREATION 
>   unit/src/test/java/org/apache/falcon/unit/FalconUnitTestBase.java PRE-CREATION 
>   unit/src/test/java/org/apache/falcon/unit/TestFalconUnit.java PRE-CREATION 
>   unit/src/test/java/org/apache/falcon/unit/examples/JavaExample.java PRE-CREATION 
>   unit/src/test/resources/cluster-template.xml PRE-CREATION 
>   unit/src/test/resources/infeed.xml PRE-CREATION 
>   unit/src/test/resources/input.txt PRE-CREATION 
>   unit/src/test/resources/outfeed.xml PRE-CREATION 
>   unit/src/test/resources/process.xml PRE-CREATION 
>   unit/src/test/resources/workflow.xml PRE-CREATION 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 47b51fe

>   webapp/src/test/java/org/apache/falcon/resource/ProcessInstanceManagerIT.java 6e58064

>   webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java 02d1011 
> 
> Diff: https://reviews.apache.org/r/36538/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message