falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesan Ramachandran" <me.venk...@gmail.com>
Subject Re: Review Request 38465: FALCON-1459 : Ability to import from database
Date Wed, 28 Oct 2015 22:03:00 GMT


> On Oct. 24, 2015, 3:24 a.m., Venkat Ranganathan wrote:
> > client/src/main/resources/datasource-0.1.xsd, line 192
> > <https://reviews.apache.org/r/38465/diff/5/?file=1102887#file1102887line192>
> >
> >     Credential types - we support password alias also which allows users to not
have any plaintext password at rest.   Also with wallets, the user and password can be optional.
  These are items to add to the list of enhancements to note lest we lose it

This is a MVP implementation. we can certainly add these enahanements going forward.


> On Oct. 24, 2015, 3:24 a.m., Venkat Ranganathan wrote:
> > client/src/main/resources/datasource-0.1.xsd, line 210
> > <https://reviews.apache.org/r/38465/diff/5/?file=1102887#file1102887line210>
> >
> >     What if any validatation we want to do with this.  Any good reason, this needs
to be an enumeration?

We can take advantage of the enumerations to use datasource specific features.


- Venkatesan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38465/#review103892
-----------------------------------------------------------


On Oct. 28, 2015, 9:55 p.m., Venkatesan Ramachandran wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38465/
> -----------------------------------------------------------
> 
> (Updated Oct. 28, 2015, 9:55 p.m.)
> 
> 
> Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh,
and Venkat Ranganathan.
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FALCON-1459 : Ability to import from database
> 
> 
> Diffs
> -----
> 
>   client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c 
>   client/src/main/java/org/apache/falcon/Tag.java beeb812 
>   client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 
>   client/src/main/java/org/apache/falcon/metadata/RelationshipType.java f034772 
>   client/src/main/resources/datasource-0.1.xsd PRE-CREATION 
>   client/src/main/resources/feed-0.1.xsd 77b8f4b 
>   client/src/main/resources/jaxb-binding.xjb 6f1d6c7 
>   client/src/main/resources/mysql_database.xml PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java ceefb17 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 5c252a8 
>   common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201

>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java c5cfdd2

>   common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java 4dd1c68

>   common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf 
>   common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java bd32852

>   common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java
8c3876c 
>   common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java
f485764 
>   common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java 56fbde0

>   common/src/main/java/org/apache/falcon/metadata/RelationshipLabel.java 5b312da 
>   common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java ac7140c

>   common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 9bfc51b

>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java aab9cee 
>   common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java 4020d36 
>   common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java
PRE-CREATION 
>   common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 905be68

>   common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java 3863b11 
>   common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-file-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/datasource/datasource-invalid-0.1.xml PRE-CREATION

>   common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-exclude-fields-0.1.xml PRE-CREATION

>   common/src/test/resources/config/feed/feed-import-invalid-0.1.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-import-noargs-0.1.xml PRE-CREATION 
>   docs/src/site/twiki/EntitySpecification.twiki e07fe12 
>   docs/src/site/twiki/FalconCLI.twiki d994866 
>   messaging/src/main/java/org/apache/falcon/messaging/JMSMessageProducer.java 515562a

>   oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java PRE-CREATION

>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java a04ae95 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
3213a70 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java
0dc09ee 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 9e55edf 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java
f953557 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java
6e2a631 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
3f49adb 
>   oozie/src/main/resources/action/feed/import-sqoop-database-action.xml PRE-CREATION

>   oozie/src/main/resources/action/post-process.xml df0d286 
>   pom.xml 87c55e3 
>   webapp/pom.xml 9e4dc8f 
>   webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java PRE-CREATION 
>   webapp/src/test/java/org/apache/falcon/resource/TestContext.java d067dee 
>   webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java PRE-CREATION 
>   webapp/src/test/resources/datasource-template.xml PRE-CREATION 
>   webapp/src/test/resources/feed-template3.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38465/diff/
> 
> 
> Testing
> -------
> 
> * Unit tests
> * Integration tests
> * Manual tests
>   * Setup MySQL, create table and populate
>   * Create datasource and feed entity with import policy in Falcon  
>   * Made sure the data lands up in the HDFS.
> 
> 
> Thanks,
> 
> Venkatesan Ramachandran
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message