From issues-return-64621-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Sat Sep 15 14:51:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8BCDB180629 for ; Sat, 15 Sep 2018 14:51:03 +0200 (CEST) Received: (qmail 82149 invoked by uid 500); 15 Sep 2018 12:51:02 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 82137 invoked by uid 99); 15 Sep 2018 12:51:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Sep 2018 12:51:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 28C8AC0558 for ; Sat, 15 Sep 2018 12:51:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id X87ueHnab-Dr for ; Sat, 15 Sep 2018 12:51:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id EA84A5F3E2 for ; Sat, 15 Sep 2018 12:51:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 62BC2E0A31 for ; Sat, 15 Sep 2018 12:51:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 15FBC23F9C for ; Sat, 15 Sep 2018 12:51:00 +0000 (UTC) Date: Sat, 15 Sep 2018 12:51:00 +0000 (UTC) From: "Pierre Villard (JIRA)" To: issues@nifi.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (NIFI-5592) ConsumeKafkaRecord* processors can stop pulling data if the data doesn't match the configured schema MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/NIFI-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5592: --------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) > ConsumeKafkaRecord* processors can stop pulling data if the data doesn't match the configured schema > ---------------------------------------------------------------------------------------------------- > > Key: NIFI-5592 > URL: https://issues.apache.org/jira/browse/NIFI-5592 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Reporter: Mark Payne > Assignee: Mark Payne > Priority: Major > Fix For: 1.8.0 > > > If the data in the kafka topic does not adhere to the configured schema, the processor should route the data to 'parse.failure' but in some conditions, we may encounter the following SchemaValidationException: > {code} > 2018-09-13 07:37:54,196 ERROR [Timer-Driven Process Thread-1] o.a.n.p.k.pubsub.ConsumeKafkaRecord_1_0 ConsumeKafkaRecord_1_0[id=c258fa20-0165-1000-ffff-ffffb401d2c7] Exception while processing data from kafka so will close the lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@3fa54755 due to org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.serialization.SchemaValidationException: Field designation cannot be null: org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.serialization.SchemaValidationException: Field designation cannot be null > org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.serialization.SchemaValidationException: Field designation cannot be null > at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.writeRecordData(ConsumerLease.java:587) > at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.lambda$processRecords$2(ConsumerLease.java:330) > at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553) > at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.processRecords(ConsumerLease.java:317) > at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.poll(ConsumerLease.java:178) > at org.apache.nifi.processors.kafka.pubsub.ConsumeKafkaRecord_1_0.onTrigger(ConsumeKafkaRecord_1_0.java:378) > at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.nifi.serialization.SchemaValidationException: Field designation cannot be null > at org.apache.nifi.serialization.record.MapRecord.checkTypes(MapRecord.java:81) > at org.apache.nifi.serialization.record.MapRecord.(MapRecord.java:52) > at org.apache.nifi.csv.CSVRecordReader.nextRecord(CSVRecordReader.java:113) > at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50) > at org.apache.nifi.processors.kafka.pubsub.ConsumerLease.writeRecordData(ConsumerLease.java:534) > ... 17 common frames omitted > {code} > In such a case, it will constantly roll back the session and keep trying to pull the data, with the Exception continually occurring. -- This message was sent by Atlassian JIRA (v7.6.3#76005)