From issues-return-211854-archive-asf-public=cust-asf.ponee.io@spark.apache.org Sat Dec 15 17:05:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3ADEB180652 for ; Sat, 15 Dec 2018 17:05:04 +0100 (CET) Received: (qmail 32778 invoked by uid 500); 15 Dec 2018 16:05:03 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 32761 invoked by uid 99); 15 Dec 2018 16:05:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Dec 2018 16:05:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E2AC8C7C4E for ; Sat, 15 Dec 2018 16:05:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id uIBOhl0tmWWI for ; Sat, 15 Dec 2018 16:05:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 0CC615F3ED for ; Sat, 15 Dec 2018 16:05:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 54243E00D4 for ; Sat, 15 Dec 2018 16:05:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0C10A23FAD for ; Sat, 15 Dec 2018 16:05:00 +0000 (UTC) Date: Sat, 15 Dec 2018 16:05:00 +0000 (UTC) From: "Maxim Gekk (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (SPARK-26376) Skip inputs without tokens by JSON datasource MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Maxim Gekk created SPARK-26376: ---------------------------------- Summary: Skip inputs without tokens by JSON datasource Key: SPARK-26376 URL: https://issues.apache.org/jira/browse/SPARK-26376 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.0 Reporter: Maxim Gekk The changes https://github.com/apache/spark/commit/38628dd1b8298d2686e5d00de17c461c70db99a8 can potentially break existing application if it doesn't expect a bad record for string without any JSON tokens in the PERMISSIVE mode. This ticket aims to return previous behaviour of JSON datasource and ignore such strings (including empty strings). The from_json function should keep new behaviour and produce bad records for empty strings and strings without any JSON tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org