falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesan Ramachandran" <me.venk...@gmail.com>
Subject Re: Review Request 38465: FALCON-1459 : Ability to import from database
Date Wed, 28 Oct 2015 21:56:15 GMT


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java,
line 21
> > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083437#file1083437line21>
> >
> >     use org.apache.commons.lang3.StringUtils

Switched to using org.apache.commons.lang3.StringUtils for the following files:
DatasourceEntityParser
FeedImportCoordinatorBuilder
ImportWorkflowBuilder
HdfsClassLoader
However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils.


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java, line
21
> > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083459#file1083459line21>
> >
> >     use org.apache.commons.lang3.StringUtils

Switched to using org.apache.commons.lang3.StringUtils for the following files:
DatasourceEntityParser
FeedImportCoordinatorBuilder
ImportWorkflowBuilder
HdfsClassLoader
However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils.


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java, line 31
> > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083460#file1083460line31>
> >
> >     use org.apache.commons.lang3.StringUtils. Typo?

Switched to using org.apache.commons.lang3.StringUtils for the following files:
DatasourceEntityParser
FeedImportCoordinatorBuilder
ImportWorkflowBuilder
HdfsClassLoader
However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils.


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java,
line 274
> > <https://reviews.apache.org/r/38465/diff/5/?file=1102901#file1102901line274>
> >
> >     For replicaiton and eviction there can be cases where getOutputFeedInstancePaths
or getInputFeedNames can return "NONE" or "IGNORE" i.e. no input or ourput feeds. Can there
be such a scenario for Import?

Import, in its current state, should produce a output path whenever it runs as it will produce
a snapshot of data. Its safe to assume that there will be an output path.


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java, line 69
> > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083460#file1083460line69>
> >
> >     why was srcClusterName changed from cluster.getName() to "NA"?
> >     Do we req both srcClusterName and WorkflowExecutionArgs.CLUSTER_NAME?
> 
> Balu Vellanki wrote:
>     In all workflows that do not do replication, the srcClusterName is set to "NA". So
it makes sense for the import process to set srcClusterName to "NA".

Correct. The post processing xml has an argument srcClusterName that needs to be set to NA
in case of flows other than replication. 
WorkflowExecutionArgs.CLUSTER_NAME is also used in the post processing and so it needs to
be passed on.


> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java,
line 133
> > <https://reviews.apache.org/r/38465/diff/5/?file=1102932#file1102932line133>
> >
> >     I see that WorkflowExecutionArgs.DATASOURCE_NAME.getName is set with "NONE"
or "IGNORE". What is the logic here? When should be None and what cases it will be IGNORE?
> 
> Balu Vellanki wrote:
>     @Sowmya - Look at Jira https://issues.apache.org/jira/browse/FALCON-1401. NONE is
the right value to use when there is no input/output exists. IGNORE is supported for backwards
compatibility, but should not be used going forward.
> 
> Sowmya Ramesh wrote:
>     @Balu - I know about that jira. I am asking in this jira why different values are
used and what is the logic behind that if I am missing something. Also that is for feed names,
I am not sure if we check DATASOURCE_NAME.getName() for NONE or IGNORE anywhere. Backward
compatibilty makes sense in that jira as it spans multiple apache version. This is new code
and hence not required to use both.
> 
> Balu Vellanki wrote:
>     Agree with you. IGNORE should not be used here.

NONE is used. The post process xml has the following for datasource argument
<arg>-datasource</arg>
<arg>${datasource == 'NA' ? 'NONE' : datasource}</arg>


- Venkatesan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38465/#review103847
-----------------------------------------------------------


On Oct. 28, 2015, 9:55 p.m., Venkatesan Ramachandran wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38465/
> -----------------------------------------------------------
> 
> (Updated Oct. 28, 2015, 9:55 p.m.)
> 
> 
> Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh,
and Venkat Ranganathan.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FALCON-1459 : Ability to import from database
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c 
>   client/src/main/java/org/apache/falcon/Tag.java beeb812 
>   client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 
>   client/src/main/java/org/apache/falcon/metadata/RelationshipType.java f034772 
>   client/src/main/resources/datasource-0.1.xsd PRE-CREATION 
>   client/src/main/resources/feed-0.1.xsd 77b8f4b 
>   client/src/main/resources/jaxb-binding.xjb 6f1d6c7 
>   client/src/main/resources/mysql_database.xml PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java ceefb17 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 5c252a8 
>   common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201

>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java c5cfdd2

>   common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java 4dd1c68

>   common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf 
>   common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java bd32852

>   common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java
8c3876c 
>   common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java
f485764 
>   common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java 56fbde0

>   common/src/main/java/org/apache/falcon/metadata/RelationshipLabel.java 5b312da 
>   common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java ac7140c

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 9bfc51b

>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java aab9cee 
>   common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java 4020d36 
>   common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java
PRE-CREATION 
>   common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 905be68

>   common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java 3863b11 
>   common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-file-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-invalid-0.1.xml PRE-CREATION

>   common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-exclude-fields-0.1.xml PRE-CREATION

>   common/src/test/resources/config/feed/feed-import-invalid-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-noargs-0.1.xml PRE-CREATION 
>   docs/src/site/twiki/EntitySpecification.twiki e07fe12 
>   docs/src/site/twiki/FalconCLI.twiki d994866 
>   messaging/src/main/java/org/apache/falcon/messaging/JMSMessageProducer.java 515562a

>   oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java a04ae95 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
3213a70 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java
0dc09ee 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 9e55edf 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java
f953557 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java
6e2a631 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
3f49adb 
>   oozie/src/main/resources/action/feed/import-sqoop-database-action.xml PRE-CREATION

>   oozie/src/main/resources/action/post-process.xml df0d286 
>   pom.xml 87c55e3 
>   webapp/pom.xml 9e4dc8f 
>   webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java PRE-CREATION 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java d067dee 
>   webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java PRE-CREATION 
>   webapp/src/test/resources/datasource-template.xml PRE-CREATION 
>   webapp/src/test/resources/feed-template3.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38465/diff/
> 
> 
> Testing
> -------
> 
> * Unit tests
> * Integration tests
> * Manual tests
>   * Setup MySQL, create table and populate
>   * Create datasource and feed entity with import policy in Falcon  
>   * Made sure the data lands up in the HDFS.
> 
> 
> Thanks,
> 
> Venkatesan Ramachandran
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message