falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi" <bpeey...@yahoo.co.in>
Subject Re: Review Request 38465: FALCON-1459 : Ability to import from database
Date Fri, 18 Sep 2015 14:55:01 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38465/#review99524
-----------------------------------------------------------



client/src/main/java/org/apache/falcon/entity/v0/EntityType.java (line 100)
<https://reviews.apache.org/r/38465/#comment156450>

    Can you please clarify the statement. Feed and process entity can be scheduled but cluster
can't be scheduled.



common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java (line 42)
<https://reviews.apache.org/r/38465/#comment156453>

    In this class, you are trying to fetch the Sqoop jar files and then loading into JVM to
get set to classpath. Can you try any one of the following options:
    1. Package required sqoop jar files with Falcon and then use it locally. 
    2. Try to use distributed cache for jar files.
    
    For the above you can explore the class SharedLibraryHostingService in Falcon.



common/src/test/resources/config/datasource/datasource-0.1.xml (line 44)
<https://reviews.apache.org/r/38465/#comment156457>

    Should not the driver classname and required jar file get handle internally instead of
specifying explictly. To get the required jar file, you can think of using distributed cache
or package required jar file with Falcon to be available locally.



common/src/test/resources/config/datasource/datasource-invalid-0.1.xml (line 43)
<https://reviews.apache.org/r/38465/#comment156454>

    Similar comment as above.



oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java (line 86)
<https://reviews.apache.org/r/38465/#comment156447>

    Sqoop command can be built using command element or arg elements. Is there any downside
of using arg elements instead of command element.



oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java (line 61)
<https://reviews.apache.org/r/38465/#comment156445>

    Uncomment the statement or remove it, if it is not required.



retention/src/test/java/org/apache/falcon/retention/FeedEvictorTest.java (line 374)
<https://reviews.apache.org/r/38465/#comment156444>

    Uncomment the @Test .



webapp/pom.xml (line 546)
<https://reviews.apache.org/r/38465/#comment156455>

    Is localhost required.



webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java (line 78)
<https://reviews.apache.org/r/38465/#comment156443>

    Please remove the commented section, if it is not required.



webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java (line 133)
<https://reviews.apache.org/r/38465/#comment156441>

    Remove the commented statement.



webapp/src/test/resources/datasource-template.xml (line 44)
<https://reviews.apache.org/r/38465/#comment156442>

    See the above comments about handling the driver class name and required jar file in datasource
xml file


- Peeyush Bishnoi


On Sept. 17, 2015, 7:40 p.m., Venkatesan Ramachandran wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38465/
> -----------------------------------------------------------
> 
> (Updated Sept. 17, 2015, 7:40 p.m.)
> 
> 
> Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh,
and Venkat Ranganathan.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FALCON-1459 : Ability to import from database
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c 
>   client/src/main/java/org/apache/falcon/Tag.java beeb812 
>   client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 
>   client/src/main/java/org/apache/falcon/metadata/RelationshipType.java f034772 
>   client/src/main/resources/datasource-0.1.xsd PRE-CREATION 
>   client/src/main/resources/feed-0.1.xsd 4ff8baa 
>   client/src/main/resources/jaxb-binding.xjb f644f40 
>   client/src/main/resources/mysql_database.xml PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 2f05b1f 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b 
>   common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201

>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 992fc51

>   common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java e27187b

>   common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf 
>   common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java bd32852

>   common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java
8c3876c 
>   common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239

>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java 6179855 
>   common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc 
>   common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java
PRE-CREATION 
>   common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java d203b7c

>   common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java 3863b11 
>   common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-invalid-0.1.xml PRE-CREATION

>   common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-invalid-0.1.xml PRE-CREATION 
>   falcon-regression/merlin-core/src/main/java/org/apache/falcon/regression/core/util/HiveAssert.java
2a934b5 
>   oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 85f5330 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
3213a70 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java b819dee 
>   oozie/src/main/resources/action/feed/import-sqoop-database-action.xml PRE-CREATION

>   pom.xml 646de69 
>   retention/src/test/java/org/apache/falcon/retention/FeedEvictorTest.java 72447da 
>   webapp/pom.xml 828f7f5 
>   webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java PRE-CREATION 
>   webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java 220e5a7

>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java f031137 
>   webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java PRE-CREATION 
>   webapp/src/test/resources/datasource-template.xml PRE-CREATION 
>   webapp/src/test/resources/feed-template3.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38465/diff/
> 
> 
> Testing
> -------
> 
> * Unit tests
> * Integration tests
> * Manual tests
>   * Setup MySQL, create table and populate
>   * Create datasource and feed entity with import policy in Falcon  
>   * Made sure the data lands up in the HDFS.
> 
> 
> Thanks,
> 
> Venkatesan Ramachandran
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message