Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D0A3A18ECD for ; Wed, 1 Jul 2015 16:28:07 +0000 (UTC) Received: (qmail 98835 invoked by uid 500); 1 Jul 2015 16:28:07 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 98767 invoked by uid 500); 1 Jul 2015 16:28:07 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 98431 invoked by uid 99); 1 Jul 2015 16:28:06 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2015 16:28:06 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 01F7B1DFE65; Wed, 1 Jul 2015 16:28:06 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5859763571831878722==" MIME-Version: 1.0 Subject: Re: Review Request 36025: HIVE-11139 Emit more lineage information From: "Jimmy Xiang" To: "Xuefu Zhang" Cc: "Szehon Ho" , "hive" , "Jimmy Xiang" Date: Wed, 01 Jul 2015 16:28:05 -0000 Message-ID: <20150701162805.13307.94464@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Jimmy Xiang" X-ReviewGroup: hive X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/36025/ X-Sender: "Jimmy Xiang" References: <20150629215605.13308.81564@reviews.apache.org> In-Reply-To: <20150629215605.13308.81564@reviews.apache.org> Reply-To: "Jimmy Xiang" X-ReviewRequest-Repository: hive-git --===============5859763571831878722== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36025/ ----------------------------------------------------------- (Updated July 1, 2015, 4:27 p.m.) Review request for hive and Xuefu Zhang. Changes ------- Addressed Szehon's review comments, added support for create view statements, fixed the gson dependency. Repository: hive-git Description ------- Extended HIVE-1131 to log more lineage info. A post exec hook, LineageLogger, is added to log the lineage in JSON format. Diffs (updated) ----- pom.xml 1c9b02f ql/pom.xml b86b85d ql/src/java/org/apache/hadoop/hive/ql/Driver.java 669e6be ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java a0d61f5 ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 0c6a938 ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java f98b38b ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/NoDeleteRollingFileAppender.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/ExprProcFactory.java c930b80 ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/Generator.java 51bef04 ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/LineageCtx.java cef24e3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java 5957ac0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f41668b ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 3a1a4af ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 3a4ea2f ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java e716ed2 ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java e1cab79 ql/src/test/queries/clientpositive/lineage2.q PRE-CREATION ql/src/test/queries/clientpositive/lineage3.q PRE-CREATION ql/src/test/results/clientpositive/alter_partition_change_col.q.out 0d97b7a ql/src/test/results/clientpositive/alter_table_cascade.q.out 0139466 ql/src/test/results/clientpositive/combine2.q.out 2400c96 ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 34cd1ff ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 0d631ce ql/src/test/results/clientpositive/index_auto_mult_tables.q.out c3c1fc8 ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out e3dfcb7 ql/src/test/results/clientpositive/index_auto_partitioned.q.out f3ae876 ql/src/test/results/clientpositive/index_auto_update.q.out 52509be ql/src/test/results/clientpositive/index_bitmap.q.out 596312d ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 8d4774d ql/src/test/results/clientpositive/index_bitmap_rc.q.out 45fe339 ql/src/test/results/clientpositive/index_compact.q.out a33f82a ql/src/test/results/clientpositive/index_compact_2.q.out fd4cdf9 ql/src/test/results/clientpositive/join34.q.out 48c3b74 ql/src/test/results/clientpositive/join35.q.out c0372a7 ql/src/test/results/clientpositive/lineage1.q.out d9f1ce3 ql/src/test/results/clientpositive/lineage2.q.out PRE-CREATION ql/src/test/results/clientpositive/lineage3.q.out PRE-CREATION ql/src/test/results/clientpositive/load_dyn_part13.q.out 1776e12 ql/src/test/results/clientpositive/multiMapJoin1.q.out 08d2bc1 ql/src/test/results/clientpositive/multi_insert.q.out ea2e554 ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out 0fbad49 ql/src/test/results/clientpositive/orc_dictionary_threshold.q.out 10ed4ac ql/src/test/results/clientpositive/ptf.q.out 6bd1747 ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 6c0f4a5 ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 8248a70 ql/src/test/results/clientpositive/spark/join34.q.out 09b8b6b ql/src/test/results/clientpositive/spark/join35.q.out e84c860 ql/src/test/results/clientpositive/spark/load_dyn_part13.q.out 4480310 ql/src/test/results/clientpositive/spark/multi_insert.q.out c77eb05 ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out e3ef39e ql/src/test/results/clientpositive/spark/ptf.q.out 1ca6951 ql/src/test/results/clientpositive/spark/union22.q.out fdb4d47 ql/src/test/results/clientpositive/spark/union28.q.out 98582df ql/src/test/results/clientpositive/spark/union29.q.out 1776b4d ql/src/test/results/clientpositive/spark/union30.q.out 3409623 ql/src/test/results/clientpositive/spark/union33.q.out 0e6b1aa ql/src/test/results/clientpositive/spark/union_date_trim.q.out 86b96ac ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 ql/src/test/results/clientpositive/spark/union_remove_10.q.out 2718775 ql/src/test/results/clientpositive/spark/union_remove_11.q.out be65741 ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 ql/src/test/results/clientpositive/spark/union_remove_17.q.out 74a5b23 ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 ql/src/test/results/clientpositive/spark/union_remove_2.q.out 26c4eff ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 ql/src/test/results/clientpositive/spark/union_remove_23.q.out d9875f3 ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 ql/src/test/results/clientpositive/spark/union_remove_3.q.out a14a8b4 ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 ql/src/test/results/clientpositive/spark/union_remove_5.q.out 55baa8b ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 ql/src/test/results/clientpositive/spark/union_remove_8.q.out f580bd8 ql/src/test/results/clientpositive/spark/union_remove_9.q.out 0931d11 ql/src/test/results/clientpositive/spark/union_top_level.q.out 9b10a46 ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 8447c04 ql/src/test/results/clientpositive/spark/windowing.q.out c8117d5 ql/src/test/results/clientpositive/temp_table_windowing_expressions.q.out 417f595 ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out 3d04de9 ql/src/test/results/clientpositive/union22.q.out f14c512 ql/src/test/results/clientpositive/union28.q.out ab3186e ql/src/test/results/clientpositive/union29.q.out 3a60d0a ql/src/test/results/clientpositive/union30.q.out 4529074 ql/src/test/results/clientpositive/union33.q.out b44e7b2 ql/src/test/results/clientpositive/unionDistinct_1.q.out 207caf2 ql/src/test/results/clientpositive/union_date_trim.q.out 86b96ac ql/src/test/results/clientpositive/union_remove_1.q.out 5fa8caf ql/src/test/results/clientpositive/union_remove_10.q.out 228ff9e ql/src/test/results/clientpositive/union_remove_11.q.out bf38f0a ql/src/test/results/clientpositive/union_remove_15.q.out 67bdb1e ql/src/test/results/clientpositive/union_remove_16.q.out 2444337 ql/src/test/results/clientpositive/union_remove_17.q.out a1b9474 ql/src/test/results/clientpositive/union_remove_18.q.out 26e52e0 ql/src/test/results/clientpositive/union_remove_19.q.out d8314f8 ql/src/test/results/clientpositive/union_remove_2.q.out d3522f2 ql/src/test/results/clientpositive/union_remove_20.q.out 4503726 ql/src/test/results/clientpositive/union_remove_21.q.out 98e0166 ql/src/test/results/clientpositive/union_remove_22.q.out 4f086f3 ql/src/test/results/clientpositive/union_remove_23.q.out d0e21ba ql/src/test/results/clientpositive/union_remove_24.q.out f2f6bba ql/src/test/results/clientpositive/union_remove_25.q.out 3858bdc ql/src/test/results/clientpositive/union_remove_3.q.out 40e554a ql/src/test/results/clientpositive/union_remove_4.q.out 10a8e89 ql/src/test/results/clientpositive/union_remove_5.q.out b59d6da ql/src/test/results/clientpositive/union_remove_6.q.out 67ba74c ql/src/test/results/clientpositive/union_remove_6_subq.q.out 0d1a436 ql/src/test/results/clientpositive/union_remove_7.q.out 76c678f ql/src/test/results/clientpositive/union_remove_8.q.out fe67448 ql/src/test/results/clientpositive/union_remove_9.q.out 684091b ql/src/test/results/clientpositive/union_top_level.q.out 1db196f ql/src/test/results/clientpositive/vectorized_ptf.q.out 194261f ql/src/test/results/clientpositive/windowing.q.out c8117d5 ql/src/test/results/clientpositive/windowing_expressions.q.out 5c411ef Diff: https://reviews.apache.org/r/36025/diff/ Testing ------- Unit test, Hive on Spark/MR. Thanks, Jimmy Xiang --===============5859763571831878722==--