Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Content-Type: multipart/alternative;
 boundary="===============7173081273579966231=="
MIME-Version: 1.0
Subject: Re: Review Request 38465: FALCON-1459 : Ability to import from
 database
From: "Venkatesan Ramachandran" <me.venkatr@gmail.com>
To: "Venkat Ranganathan" <n.r.v@live.com>,
 "Balu Vellanki" <bvellanki@hortonworks.com>,
 "Peeyush Bishnoi" <bpeeyush@yahoo.co.in>,
 "Sowmya Ramesh" <sramesh@hortonworks.com>, "Ajay Yadava" <ajaynsit@gmail.com>
Cc: "Venkatesan Ramachandran" <me.venkatr@gmail.com>,
 "Falcon" <dev@falcon.incubator.apache.org>
Date: Thu, 29 Oct 2015 00:11:05 -0000
Message-ID: <20151029001105.22462.43058@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Venkatesan Ramachandran" <noreply@reviews.apache.org>
References: <20151023203420.1657.85644@reviews.apache.org>
In-Reply-To: <20151023203420.1657.85644@reviews.apache.org>
Reply-To: "Venkatesan Ramachandran" <me.venkatr@gmail.com>

--===============7173081273579966231==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java, line 133
> > <https://reviews.apache.org/r/38465/diff/5/?file=1102932#file1102932line133>
> >
> >     I see that WorkflowExecutionArgs.DATASOURCE_NAME.getName is set with "NONE" or "IGNORE". What is the logic here? When should be None and what cases it will be IGNORE?
> 
> Balu Vellanki wrote:
>     @Sowmya - Look at Jira https://issues.apache.org/jira/browse/FALCON-1401. NONE is the right value to use when there is no input/output exists. IGNORE is supported for backwards compatibility, but should not be used going forward.
> 
> Sowmya Ramesh wrote:
>     @Balu - I know about that jira. I am asking in this jira why different values are used and what is the logic behind that if I am missing something. Also that is for feed names, I am not sure if we check DATASOURCE_NAME.getName() for NONE or IGNORE anywhere. Backward compatibilty makes sense in that jira as it spans multiple apache version. This is new code and hence not required to use both.
> 
> Balu Vellanki wrote:
>     Agree with you. IGNORE should not be used here.
> 
> Venkatesan Ramachandran wrote:
>     NONE is used. The post process xml has the following for datasource argument
>     <arg>-datasource</arg>
>     <arg>${datasource == 'NA' ? 'NONE' : datasource}</arg>

Now, IGNORE is used. The post process xml has the following for datasource argument
<arg>-datasource</arg>
<arg>${datasource == 'NA' ? 'IGNORE' : datasource}</arg>


- Venkatesan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38465/#review103847
-----------------------------------------------------------


On Oct. 28, 2015, 9:55 p.m., Venkatesan Ramachandran wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38465/
> -----------------------------------------------------------
> 
> (Updated Oct. 28, 2015, 9:55 p.m.)
> 
> 
> Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh, and Venkat Ranganathan.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FALCON-1459 : Ability to import from database
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c 
>   client/src/main/java/org/apache/falcon/Tag.java beeb812 
>   client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 
>   client/src/main/java/org/apache/falcon/metadata/RelationshipType.java f034772 
>   client/src/main/resources/datasource-0.1.xsd PRE-CREATION 
>   client/src/main/resources/feed-0.1.xsd 77b8f4b 
>   client/src/main/resources/jaxb-binding.xjb 6f1d6c7 
>   client/src/main/resources/mysql_database.xml PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java ceefb17 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 5c252a8 
>   common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201 
>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java c5cfdd2 
>   common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java 4dd1c68 
>   common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf 
>   common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java bd32852 
>   common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java 8c3876c 
>   common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java f485764 
>   common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java 56fbde0 
>   common/src/main/java/org/apache/falcon/metadata/RelationshipLabel.java 5b312da 
>   common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java ac7140c 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 9bfc51b 
>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java aab9cee 
>   common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java 4020d36 
>   common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java PRE-CREATION 
>   common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 905be68 
>   common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java 3863b11 
>   common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-file-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-invalid-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-exclude-fields-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-invalid-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-noargs-0.1.xml PRE-CREATION 
>   docs/src/site/twiki/EntitySpecification.twiki e07fe12 
>   docs/src/site/twiki/FalconCLI.twiki d994866 
>   messaging/src/main/java/org/apache/falcon/messaging/JMSMessageProducer.java 515562a 
>   oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java a04ae95 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 3213a70 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java 0dc09ee 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 9e55edf 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java f953557 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java 6e2a631 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 3f49adb 
>   oozie/src/main/resources/action/feed/import-sqoop-database-action.xml PRE-CREATION 
>   oozie/src/main/resources/action/post-process.xml df0d286 
>   pom.xml 87c55e3 
>   webapp/pom.xml 9e4dc8f 
>   webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java PRE-CREATION 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java d067dee 
>   webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java PRE-CREATION 
>   webapp/src/test/resources/datasource-template.xml PRE-CREATION 
>   webapp/src/test/resources/feed-template3.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38465/diff/
> 
> 
> Testing
> -------
> 
> * Unit tests
> * Integration tests
> * Manual tests
>   * Setup MySQL, create table and populate
>   * Create datasource and feed entity with import policy in Falcon  
>   * Made sure the data lands up in the HDFS.
> 
> 
> Thanks,
> 
> Venkatesan Ramachandran
> 
>


--===============7173081273579966231==--