Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 158B810569 for ; Sun, 28 Jul 2013 07:13:22 +0000 (UTC) Received: (qmail 78041 invoked by uid 500); 28 Jul 2013 07:13:22 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 78009 invoked by uid 500); 28 Jul 2013 07:13:21 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 77999 invoked by uid 99); 28 Jul 2013 07:13:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Jul 2013 07:13:19 +0000 X-ASF-Spam-Status: No, hits=-1999.3 required=5.0 tests=ALL_TRUSTED,HTML_MESSAGE,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sun, 28 Jul 2013 07:13:17 +0000 Received: (qmail 77983 invoked by uid 99); 28 Jul 2013 07:12:57 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Jul 2013 07:12:57 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id F168D1C0054; Sun, 28 Jul 2013 07:12:55 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5045319518381505967==" MIME-Version: 1.0 Subject: Re: Review Request 12977: Ability to ingest data from databases From: "Seetharam Venkatesh" To: "Srikanth Sundarrajan" Cc: "Seetharam Venkatesh" , "Falcon" Date: Sun, 28 Jul 2013 07:12:55 -0000 Message-ID: <20130728071255.30929.83193@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Seetharam Venkatesh" X-ReviewGroup: Falcon X-ReviewRequest-URL: https://reviews.apache.org/r/12977/ X-Sender: "Seetharam Venkatesh" References: <20130726235354.32328.634@reviews.apache.org> In-Reply-To: <20130726235354.32328.634@reviews.apache.org> Reply-To: "Seetharam Venkatesh" X-Virus-Checked: Checked by ClamAV on apache.org --===============5045319518381505967== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12977/ ----------------------------------------------------------- (Updated July 28, 2013, 7:12 a.m.) Review request for Falcon and Srikanth Sundarrajan. Changes ------- Adds more integration tests with embedded derby apart from mysql which is disabled by default. Also adds instance management functions with lifecycle as an option. All unit and integration tests pass. Also end to end import, replicate and export functionality works. Excuse me for the size of this patch. Adds quite a few things and more tests. Bugs: FALCON-36 https://issues.apache.org/jira/browse/FALCON-36 Repository: falcon-git Description ------- Adds capability to ingest data from databases into Hadoop. Diffs (updated) ----- client/src/main/java/org/apache/falcon/Lifecycle.java PRE-CREATION client/src/main/java/org/apache/falcon/Tag.java beeb812 client/src/main/java/org/apache/falcon/cli/FalconCLI.java 086bf4a client/src/main/java/org/apache/falcon/client/FalconClient.java 24a5bfe client/src/main/java/org/apache/falcon/entity/v0/EntityType.java bc7b1f2 client/src/main/resources/database-0.1.xsd PRE-CREATION client/src/main/resources/feed-0.1.xsd 8390565 client/src/main/resources/jaxb-binding.xjb 93e72af common/src/main/java/org/apache/falcon/entity/DatabaseHelper.java PRE-CREATION common/src/main/java/org/apache/falcon/entity/EntityUtil.java 4897985 common/src/main/java/org/apache/falcon/entity/parser/DatabaseEntityParser.java PRE-CREATION common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java 5a33201 common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 1c323fd common/src/main/java/org/apache/falcon/util/ClassLoaderForJarsOnHdfs.java PRE-CREATION common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java f7526e4 common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java e947f69 common/src/test/java/org/apache/falcon/entity/parser/DatabaseEntityParserTest.java PRE-CREATION common/src/test/resources/config/database/acquisition-feed-0.1.xml PRE-CREATION common/src/test/resources/config/database/acquisition-feed-invalid.xml PRE-CREATION common/src/test/resources/config/database/database-0.1.xml PRE-CREATION docs/src/site/twiki/FalconCLI.twiki 4163a38 docs/src/site/twiki/FalconDocumentation.twiki 6312627 feed/src/main/java/org/apache/falcon/converter/OozieFeedMapper.java d954202 feed/src/main/java/org/apache/falcon/workflow/OozieFeedWorkflowBuilder.java 7b9095f feed/src/main/resources/config/coordinator/database-export-coordinator.xml PRE-CREATION feed/src/main/resources/config/coordinator/database-import-coordinator.xml PRE-CREATION feed/src/main/resources/config/workflow/database-acquisition-workflow.xml PRE-CREATION feed/src/main/resources/config/workflow/replication-workflow.xml 7aab158 feed/src/main/resources/config/workflow/retention-workflow.xml 422209e feed/src/test/java/org/apache/falcon/converter/OozieFeedMapperTest.java d3d8e91 feed/src/test/resources/ingest-feed.xml PRE-CREATION feed/src/test/resources/mysql-db.xml PRE-CREATION messaging/src/main/java/org/apache/falcon/messaging/EntityInstanceMessage.java ddd6781 messaging/src/main/java/org/apache/falcon/messaging/MessageProducer.java cf5c2d7 messaging/src/test/java/org/apache/falcon/messaging/FalconTopicProducerTest.java 9912678 messaging/src/test/java/org/apache/falcon/messaging/FeedProducerTest.java 32f731f messaging/src/test/java/org/apache/falcon/messaging/ProcessProducerTest.java 078b9c2 oozie/src/main/java/org/apache/falcon/logging/LogMover.java e8a85c6 oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java 914828b oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java fcb9d80 oozie/src/main/resources/oozie-workflow-0.3.xsd 8cda3d9 oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java c6485cd prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java d94e8c5 prism/src/main/java/org/apache/falcon/resource/proxy/InstanceManagerProxy.java 643f98b process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java 8f75736 process/src/main/resources/config/workflow/process-parent-workflow.xml 4f6dd43 webapp/pom.xml 54d6938 webapp/src/main/java/org/apache/falcon/resource/InstanceManager.java 104dfef webapp/src/test/java/org/apache/falcon/resource/DatabaseIT.java PRE-CREATION webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabase.java PRE-CREATION webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabaseIT.java PRE-CREATION webapp/src/test/java/org/apache/falcon/resource/EmbeddedDatabaseTest.java PRE-CREATION webapp/src/test/java/org/apache/falcon/resource/PigProcessIT.java 1cb5825 webapp/src/test/java/org/apache/falcon/resource/TestContext.java 6fa07c9 webapp/src/test/resources/acquisition-feed-template.xml PRE-CREATION webapp/src/test/resources/apps/pig/data.txt e51d124 webapp/src/test/resources/database-template.xml PRE-CREATION webapp/src/test/resources/embedded-acquisition-feed-template.xml PRE-CREATION webapp/src/test/resources/embedded-database-template.xml PRE-CREATION Diff: https://reviews.apache.org/r/12977/diff/ Testing ------- Have added unit and integration tests that cover import. All existing unit and integration tests pass. Also have tested end to end import and export using mysql db and latest sqoop 1.4.4-rc0. Thanks, Seetharam Venkatesh --===============5045319518381505967==--