falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seetharam Venkatesh" <venkat...@innerzeal.com>
Subject Re: Review Request 12977: Ability to ingest data from databases
Date Sun, 28 Jul 2013 07:13:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12977/
-----------------------------------------------------------

(Updated July 28, 2013, 7:13 a.m.)


Review request for Falcon and Srikanth Sundarrajan.


Bugs: FALCON-36
    https://issues.apache.org/jira/browse/FALCON-36


Repository: falcon-git


Description
-------

Adds capability to ingest data from databases into Hadoop.


Diffs
-----

  client/src/main/java/org/apache/falcon/Lifecycle.java PRE-CREATION 
  client/src/main/java/org/apache/falcon/Tag.java beeb812 
  client/src/main/java/org/apache/falcon/cli/FalconCLI.java 086bf4a 
  client/src/main/java/org/apache/falcon/client/FalconClient.java 24a5bfe 
  client/src/main/java/org/apache/falcon/entity/v0/EntityType.java bc7b1f2 
  client/src/main/resources/database-0.1.xsd PRE-CREATION 
  client/src/main/resources/feed-0.1.xsd 8390565 
  client/src/main/resources/jaxb-binding.xjb 93e72af 
  common/src/main/java/org/apache/falcon/entity/DatabaseHelper.java PRE-CREATION 
  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 4897985 
  common/src/main/java/org/apache/falcon/entity/parser/DatabaseEntityParser.java PRE-CREATION

  common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201 
  common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 1c323fd 
  common/src/main/java/org/apache/falcon/util/ClassLoaderForJarsOnHdfs.java PRE-CREATION 
  common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java f7526e4

  common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java e947f69 
  common/src/test/java/org/apache/falcon/entity/parser/DatabaseEntityParserTest.java PRE-CREATION

  common/src/test/resources/config/database/acquisition-feed-0.1.xml PRE-CREATION 
  common/src/test/resources/config/database/acquisition-feed-invalid.xml PRE-CREATION 
  common/src/test/resources/config/database/database-0.1.xml PRE-CREATION 
  docs/src/site/twiki/FalconCLI.twiki 4163a38 
  docs/src/site/twiki/FalconDocumentation.twiki 6312627 
  feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java d954202 
  feed/src/main/java/org/apache/falcon/workflow/OozieFeedWorkflowBuilder.java 7b9095f 
  feed/src/main/resources/config/coordinator/database-export-coordinator.xml PRE-CREATION

  feed/src/main/resources/config/coordinator/database-import-coordinator.xml PRE-CREATION

  feed/src/main/resources/config/workflow/database-acquisition-workflow.xml PRE-CREATION 
  feed/src/main/resources/config/workflow/replication-workflow.xml 7aab158 
  feed/src/main/resources/config/workflow/retention-workflow.xml 422209e 
  feed/src/test/java/org/apache/falcon/converter/OozieFeedMapperTest.java d3d8e91 
  feed/src/test/resources/ingest-feed.xml PRE-CREATION 
  feed/src/test/resources/mysql-db.xml PRE-CREATION 
  messaging/src/main/java/org/apache/falcon/messaging/EntityInstanceMessage.java ddd6781 
  messaging/src/main/java/org/apache/falcon/messaging/MessageProducer.java cf5c2d7 
  messaging/src/test/java/org/apache/falcon/messaging/FalconTopicProducerTest.java 9912678

  messaging/src/test/java/org/apache/falcon/messaging/FeedProducerTest.java 32f731f 
  messaging/src/test/java/org/apache/falcon/messaging/ProcessProducerTest.java 078b9c2 
  oozie/src/main/java/org/apache/falcon/logging/LogMover.java e8a85c6 
  oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java 914828b 
  oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java fcb9d80 
  oozie/src/main/resources/oozie-workflow-0.3.xsd 8cda3d9 
  oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java c6485cd

  prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java d94e8c5 
  prism/src/main/java/org/apache/falcon/resource/proxy/InstanceManagerProxy.java 643f98b 
  process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java 8f75736 
  process/src/main/resources/config/workflow/process-parent-workflow.xml 4f6dd43 
  webapp/pom.xml 54d6938 
  webapp/src/main/java/org/apache/falcon/resource/InstanceManager.java 104dfef 
  webapp/src/test/java/org/apache/falcon/resource/DatabaseIT.java PRE-CREATION 
  webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabase.java PRE-CREATION 
  webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabaseIT.java PRE-CREATION 
  webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabaseTest.java PRE-CREATION 
  webapp/src/test/java/org/apache/falcon/resource/PigProcessIT.java 1cb5825 
  webapp/src/test/java/org/apache/falcon/resource/TestContext.java 6fa07c9 
  webapp/src/test/resources/acquisition-feed-template.xml PRE-CREATION 
  webapp/src/test/resources/apps/pig/data.txt e51d124 
  webapp/src/test/resources/database-template.xml PRE-CREATION 
  webapp/src/test/resources/embedded-acquisition-feed-template.xml PRE-CREATION 
  webapp/src/test/resources/embedded-database-template.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/12977/diff/


Testing
-------

Have added unit and integration tests that cover import. All existing unit and integration
tests pass. 
Also have tested end to end import and export using mysql db and latest sqoop 1.4.4-rc0.


Thanks,

Seetharam Venkatesh


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message