From issues-return-91254-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Jan 30 21:58:02 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0B86C180667 for ; Thu, 30 Jan 2020 22:58:01 +0100 (CET) Received: (qmail 13582 invoked by uid 500); 30 Jan 2020 21:58:01 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 13540 invoked by uid 99); 30 Jan 2020 21:58:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jan 2020 21:58:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B1327E2E0E for ; Thu, 30 Jan 2020 21:58:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 31D79780402 for ; Thu, 30 Jan 2020 21:58:00 +0000 (UTC) Date: Thu, 30 Jan 2020 21:58:00 +0000 (UTC) From: "Joe Witt (Jira)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (NIFI-6986) ValidateRecord should optionally validate if nullable fields are present MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-6986: --------------------------- Fix Version/s: (was: 1.12.0) > ValidateRecord should optionally validate if nullable fields are present > ------------------------------------------------------------------------ > > Key: NIFI-6986 > URL: https://issues.apache.org/jira/browse/NIFI-6986 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions > Reporter: Mark Payne > Assignee: Shawn Weeks > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently, if a field is nullable according to the schema, ValidateRecord considers the record to be valid, even if the field is missing completely. For some use cases, this is desirable. For example, it is common to drop fields in JSON when the field's value is null, because it can drastically reduce the size of the JSON. > However, in other use cases, this is not desirable. For example, in a CSV file, we may want to require that there are the appropriate number of fields in a Record. It may be acceptable, for instance to have a line like "1234, John Smith, , , ," but not to have a line like "1234, John Smith". > ValidateRecord should be updated with a new Property: "Allow Missing Null Values". If the value is `true` (the default, to avoid changing behavior between versions), the Processor should behave as it does now, where the absence of the field is synonymous with a null value. In this case, a line like "1234, John Smith" would be valid when the CSV is expecting 6 fields, as long as the last 4 fields are nullable. > But if the value of this new property is `false`, the Processor should require that all fields be present in the data, even if the field has a null value. In this case, a line like "1234, John Smith" would be invalid if the CSV were expected to contain 6 fields. > The `WriteJsonResult` class has a method in it: `private boolean isFieldPresent(RecordField field, Record record)`. This method should really exist on `Record` itself with a slightly different signature: `boolean isFieldPresent(RecordField field)`. It should have a default implementation provided, akin to the implementation in `WriteJsonResult` and then `WriteJsonResult` should simply use that method. > `StandardSchemaValidator` should then be updated to use this to validate that records have all required fields, as configured. `SchemaValidationContext` should then be updated also to indicate whether or not the presence of null values should be validated. -- This message was sent by Atlassian Jira (v8.3.4#803005)