From dev-return-21411-archive-asf-public=cust-asf.ponee.io@atlas.apache.org Mon Mar 5 06:54:51 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 116F7180676 for ; Mon, 5 Mar 2018 06:54:50 +0100 (CET) Received: (qmail 57106 invoked by uid 500); 5 Mar 2018 05:48:09 -0000 Mailing-List: contact dev-help@atlas.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.apache.org Delivered-To: mailing list dev@atlas.apache.org Received: (qmail 54704 invoked by uid 99); 5 Mar 2018 05:44:55 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2018 05:44:55 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4B34A180576; Mon, 5 Mar 2018 05:39:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.939 X-Spam-Level: X-Spam-Status: No, score=0.939 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_MED=-2.3, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id qoo7PkPRwX12; Mon, 5 Mar 2018 05:39:42 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 226575F181; Mon, 5 Mar 2018 05:39:42 +0000 (UTC) Received: from reviews.apache.org (unknown [10.41.0.12]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E4792E00C7; Mon, 5 Mar 2018 05:39:40 +0000 (UTC) Received: from reviews-vm2.apache.org (localhost [IPv6:::1]) by reviews.apache.org (ASF Mail Server at reviews-vm2.apache.org) with ESMTP id 9D0E5C4025B; Mon, 5 Mar 2018 05:39:39 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2855873786053592237==" MIME-Version: 1.0 Subject: Re: Review Request 65790: Data Migration Utility: Export From: Ashutosh Mestry To: Ruchi Solani , Apoorv Naik , Madhan Neethiraj Cc: Ashutosh Mestry , atlas Date: Mon, 05 Mar 2018 05:39:39 -0000 Message-ID: <20180305053939.43301.19545@reviews-vm2.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Ashutosh Mestry X-ReviewGroup: atlas X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/65790/ X-Sender: Ashutosh Mestry References: <20180304172523.43301.92634@reviews-vm2.apache.org> In-Reply-To: <20180304172523.43301.92634@reviews-vm2.apache.org> X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/assembly/bin.xml X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/resources/migrationContext.xml X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/resources/atlas_migration.py X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/resources/README X-ReviewBoard-Diff-For: tools/atlas-migration-utility/pom.xml X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/resources/migration-export-request.json X-ReviewBoard-Diff-For: tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java Reply-To: Ashutosh Mestry X-ReviewRequest-Repository: atlas --===============2855873786053592237== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65790/ ----------------------------------------------------------- (Updated March 5, 2018, 5:39 a.m.) Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani. Changes ------- Updates include: - Addressed review comments. - Attached latest migration utility. Bugs: ATLAS-2461 https://issues.apache.org/jira/browse/ATLAS-2461 Repository: atlas Description ------- **Background** The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests. **Approach** The migration is a new utility that will perform export of data using a pre-defined export request file. The approach used by this application: - Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_. - Create _ImportService_ using Spring's framework classes. Here are the pieces: - _MigrationApp_: Contains _main_. It is the entry point for the app. - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use. - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out. - _atlas_migration.py_ Creates environment for the app to execute. It launches the application. - _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_. - _README_: Instructions on usage. **Build** The project maven's assembly building plugin. The build will create a ZIP file with the necessary files. _mvn install package_ generates a ZIP file in _target_ directory. **Usage (from README)** - Use Ambari to turn shutdown Atlas. - Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip' - cd atlas-migration-kit - Run 'python atlas_migration.py'. Use 'python atlas_migration.py ' to export to a specific file. - To watch the progress: 'tail -f /var/log/atlas/application.log'. Diffs (updated) ----- pom.xml 7db1be78 tools/atlas-migration-utility/pom.xml PRE-CREATION tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION tools/atlas-migration-utility/src/main/resources/README PRE-CREATION tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION Diff: https://reviews.apache.org/r/65790/diff/5/ Changes: https://reviews.apache.org/r/65790/diff/4-5/ Testing ------- **Unit tests** None. **Functional tests** Exports from existing clusters. File Attachments (updated) ---------------- Migration Utility https://reviews.apache.org/media/uploaded/files/2018/03/05/4d42eda3-cba8-441d-8505-12271cffc569__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip Thanks, Ashutosh Mestry --===============2855873786053592237==--