From commits-return-9729-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Tue Jan 14 07:29:55 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0C07518061A for ; Tue, 14 Jan 2020 08:29:54 +0100 (CET) Received: (qmail 84917 invoked by uid 500); 14 Jan 2020 07:29:54 -0000 Mailing-List: contact commits-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hudi.apache.org Delivered-To: mailing list commits@hudi.apache.org Received: (qmail 84894 invoked by uid 99); 14 Jan 2020 07:29:54 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jan 2020 07:29:54 +0000 From: GitBox To: commits@hudi.apache.org Subject: [GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc Message-ID: <157898699440.7175.18405678001141213779.gitbox@gitbox.apache.org> References: In-Reply-To: Date: Tue, 14 Jan 2020 07:29:54 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit vinothchandar commented on a change in pull request #1200: [HUDI-514] A schema provider to get metadata through Jdbc URL: https://github.com/apache/incubator-hudi/pull/1200#discussion_r366183041 ########## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java ########## @@ -236,4 +250,57 @@ public static TypedProperties readConfig(InputStream in) throws IOException { defaults.load(in); return defaults; } + + /*** + * call spark function get the schema through jdbc. + * @param options + * @return + * @throws Exception + */ + public static Schema getSchema(Map options) throws Exception { + scala.collection.immutable.Map ioptions = toScalaImmutableMap(options); + JDBCOptions jdbcOptions = new JDBCOptions(ioptions); + Connection conn = JdbcUtils.createConnectionFactory(jdbcOptions).apply(); + String url = jdbcOptions.url(); + String table = jdbcOptions.tableOrQuery(); + JdbcOptionsInWrite jdbcOptionsInWrite = new JdbcOptionsInWrite(ioptions); + boolean tableExists = JdbcUtils.tableExists(conn, jdbcOptionsInWrite); + if (tableExists) { + JdbcDialect dialect = JdbcDialects.get(url); + try { + PreparedStatement statement = conn.prepareStatement(dialect.getSchemaQuery(table)); + try { + statement.setQueryTimeout(Integer.parseInt(options.get("timeout"))); + ResultSet rs = statement.executeQuery(); + try { + StructType structType; + if (Boolean.parseBoolean(ioptions.get("nullable").get())) { + structType = JdbcUtils.getSchema(rs, dialect, true); + } else { + structType = JdbcUtils.getSchema(rs, dialect, false); + } + return AvroConversionUtils.convertStructTypeToAvroSchema(structType, table, "hoodie." + table); + } finally { + rs.close(); + } + } finally { + statement.close(); + } + } finally { + conn.close(); + } + } else { + throw new HoodieException(String.format("%s table not exists!", table)); Review comment: change to `table does not exist!`? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services