Return-Path: X-Original-To: apmail-sqoop-dev-archive@www.apache.org Delivered-To: apmail-sqoop-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B90D011D9F for ; Wed, 3 Sep 2014 01:59:53 +0000 (UTC) Received: (qmail 37597 invoked by uid 500); 3 Sep 2014 01:59:53 -0000 Delivered-To: apmail-sqoop-dev-archive@sqoop.apache.org Received: (qmail 37564 invoked by uid 500); 3 Sep 2014 01:59:53 -0000 Mailing-List: contact dev-help@sqoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@sqoop.apache.org Delivered-To: mailing list dev@sqoop.apache.org Received: (qmail 37553 invoked by uid 500); 3 Sep 2014 01:59:53 -0000 Delivered-To: apmail-incubator-sqoop-dev@incubator.apache.org Received: (qmail 37549 invoked by uid 99); 3 Sep 2014 01:59:53 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Sep 2014 01:59:53 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 60CA51DD3C8; Wed, 3 Sep 2014 01:59:52 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5115873151312477001==" MIME-Version: 1.0 Subject: Review Request 25284: SQOOP-1395: Use random generated class name for SqoopRecord From: "Qian Xu" To: "Qian Xu" , "Sqoop" Date: Wed, 03 Sep 2014 01:59:52 -0000 Message-ID: <20140903015952.16960.59634@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Qian Xu" X-ReviewGroup: Sqoop X-ReviewRequest-URL: https://reviews.apache.org/r/25284/ X-Sender: "Qian Xu" Reply-To: "Qian Xu" X-ReviewRequest-Repository: sqoop-trunk --===============5115873151312477001== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25284/ ----------------------------------------------------------- Review request for Sqoop. Bugs: SQOOP-1395 https://issues.apache.org/jira/browse/SQOOP-1395 Repository: sqoop-trunk Description ------- If you import a table "users". Sqoop will generate an entity class named "users.java". The class will be compiled, submitted and used by a mapreduce job. If the target file format is Avro or Parquet, an Avro schema will be generated as well. According to Avro specification, the entity class is described as "record", the name of the "record" is "users". For Parquet file format handling, we use the Kite SDK to manage Parquet file reading and writing with minimal efforts. Kite requires an Avro schema and all data records to be packed into GenericRecord instances. There will be a problem here. Kite will read the schema first and try to instantiate a record regarding its name. In this case, Kite will try to instantiate a "users" class. Unfortunately, there is a "users.java" out there. This will cause mapreduce job fail. In order to solve this problem, I intend to keep the name of the entity class and the Avro record different. The patch will: Change the record name in Avro schema. Remove the SqoopAvroRecord, as it is no longer required. (ClassWriter.java is reverted to previous state) Diffs ----- src/java/org/apache/sqoop/avro/AvroUtil.java 4b37d58 src/java/org/apache/sqoop/lib/SqoopAvroRecord.java 80875d2 src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 6adad79 src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 905ba8c src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java effbadd src/java/org/apache/sqoop/mapreduce/ParquetJob.java a74432a src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 806bace src/java/org/apache/sqoop/orm/ClassWriter.java 4f9dedd src/java/org/apache/sqoop/orm/TableClassName.java 88ab622 Diff: https://reviews.apache.org/r/25284/diff/ Testing ------- Manually verified the unittests of Avro and Parquet file formats Manually tested local Parquet and Avro import export. Thanks, Qian Xu --===============5115873151312477001==--