From issues-return-87972-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Mon Nov 11 16:40:07 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id DA8BE180629 for ; Mon, 11 Nov 2019 17:40:06 +0100 (CET) Received: (qmail 8429 invoked by uid 500); 11 Nov 2019 16:40:06 -0000 Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list issues@nifi.apache.org Received: (qmail 8420 invoked by uid 99); 11 Nov 2019 16:40:06 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Nov 2019 16:40:06 +0000 From: GitBox To: issues@nifi.apache.org Subject: [GitHub] [nifi-minifi-cpp] bakaid removed a comment on issue #664: MINIFICPP-1062 - Improve FlowFile repository performance by leveraging batching Message-ID: <157349040612.13652.10931018362733099609.gitbox@gitbox.apache.org> Date: Mon, 11 Nov 2019 16:40:06 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit bakaid removed a comment on issue #664: MINIFICPP-1062 - Improve FlowFile repository performance by leveraging batching URL: https://github.com/apache/nifi-minifi-cpp/pull/664#issuecomment-552518391 > Compiled the code and ran the tests, which worked fine, but running an actual GenerateFlowFile (batch of 10) -> LogAttribute flow, while works, generates the following logs: > > ``` > ��! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: ���! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: o��! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state! > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state! > ``` > > Which is disconcerting both because of hitting this condition and the memory garbage as keys. > There might be some eventual consistency or transaction magic which can account for the first, but then we have to revise the logging. > > This has to be investigated. Platform: macOS config.yml: ```MiNiFi Config Version: 3 Flow Controller: name: elclass7 comment: Created by MiNiFi C2 Flow Designer Core Properties: flow controller graceful shutdown period: 10 sec flow service write delay interval: 500 ms administrative yield duration: 30 sec bored yield duration: 10 millis max concurrent threads: 1 variable registry properties: '' FlowFile Repository: partitions: 256 checkpoint interval: 2 mins always sync: false Swap: threshold: 20000 in period: 5 sec in threads: 1 out period: 5 sec out threads: 4 Content Repository: content claim max appendable size: 10 MB content claim max flow files: 100 always sync: false Provenance Repository: provenance rollover time: 1 min implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository Component Status Repository: buffer size: 1440 snapshot frequency: 1 min Security Properties: keystore: '' keystore type: '' keystore password: '' key password: '' truststore: '' truststore type: '' truststore password: '' ssl protocol: '' Sensitive Props: key: algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL provider: BC Processors: - id: a0a92728-54aa-4f22-875e-287244a4422c name: GenerateFlowFile class: org.apache.nifi.minifi.processors.GenerateFlowFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 1000 ms penalization period: 30000 ms yield period: 1000 ms run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: '1' Data Format: Text File Size: 1 kB Unique FlowFiles: 'true' - id: 200b4e82-0edd-4cfa-b8c2-f9f86fa027a6 name: PutFile class: org.apache.nifi.minifi.processors.PutFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 1000 ms penalization period: 30000 ms yield period: 1000 ms run duration nanos: 0 auto-terminated relationships list: - failure - success Properties: Conflict Resolution Strategy: fail Create Missing Directories: 'true' Directory: /tmp/put/${now():format('Y')} Maximum File Count: '-1' Controller Services: [] Process Groups: [] Input Ports: [] Output Ports: [] Funnels: [] Connections: - id: d0fc88e8-f0e4-44fc-b20b-225a8ce7a787 name: GenerateFlowFile/success/PutFile source id: a0a92728-54aa-4f22-875e-287244a4422c source relationship names: - success destination id: 200b4e82-0edd-4cfa-b8c2-f9f86fa027a6 max work queue size: 0 max work queue data size: 10000 B flowfile expiration: 0 seconds queue prioritizer class: '' Remote Process Groups: [] NiFi Properties Overrides: {}``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services