nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [nifi-minifi-cpp] bakaid removed a comment on issue #664: MINIFICPP-1062 - Improve FlowFile repository performance by leveraging batching
Date Mon, 11 Nov 2019 16:40:06 GMT
bakaid removed a comment on issue #664: MINIFICPP-1062 - Improve FlowFile repository performance
by leveraging batching
URL: https://github.com/apache/nifi-minifi-cpp/pull/664#issuecomment-552518391
 
 
   > Compiled the code and ran the tests, which worked fine, but running an actual GenerateFlowFile
(batch of 10) -> LogAttribute flow, while works, generates the following logs:
   > 
   > ```
   >                                                                                  
                                                    ��! DB is most probably in an inconsistent
state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: ���! DB is most probably in an inconsistent
state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: o��! DB is most probably in an inconsistent state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state!
   > [2019-11-11 17:32:08.869] [org::apache::nifi::minifi::core::repository::FlowFileRepository]
[error] Failed to read key from rocksdb: ! DB is most probably in an inconsistent state!
   > ```
   > 
   > Which is disconcerting both because of hitting this condition and the memory garbage
as keys.
   > There might be some eventual consistency or transaction magic which can account for
the first, but then we have to revise the logging.
   > 
   > This has to be investigated.
   
   Platform: macOS
   config.yml:
   ```MiNiFi Config Version: 3
   Flow Controller:
     name: elclass7
     comment: Created by MiNiFi C2 Flow Designer
   Core Properties:
     flow controller graceful shutdown period: 10 sec
     flow service write delay interval: 500 ms
     administrative yield duration: 30 sec
     bored yield duration: 10 millis
     max concurrent threads: 1
     variable registry properties: ''
   FlowFile Repository:
     partitions: 256
     checkpoint interval: 2 mins
     always sync: false
     Swap:
       threshold: 20000
       in period: 5 sec
       in threads: 1
       out period: 5 sec
       out threads: 4
   Content Repository:
     content claim max appendable size: 10 MB
     content claim max flow files: 100
     always sync: false
   Provenance Repository:
     provenance rollover time: 1 min
     implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
   Component Status Repository:
     buffer size: 1440
     snapshot frequency: 1 min
   Security Properties:
     keystore: ''
     keystore type: ''
     keystore password: ''
     key password: ''
     truststore: ''
     truststore type: ''
     truststore password: ''
     ssl protocol: ''
     Sensitive Props:
       key:
       algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
       provider: BC
   Processors:
   - id: a0a92728-54aa-4f22-875e-287244a4422c
     name: GenerateFlowFile
     class: org.apache.nifi.minifi.processors.GenerateFlowFile
     max concurrent tasks: 1
     scheduling strategy: TIMER_DRIVEN
     scheduling period: 1000 ms
     penalization period: 30000 ms
     yield period: 1000 ms
     run duration nanos: 0
     auto-terminated relationships list: []
     Properties:
       Batch Size: '1'
       Data Format: Text
       File Size: 1 kB
       Unique FlowFiles: 'true'
   - id: 200b4e82-0edd-4cfa-b8c2-f9f86fa027a6
     name: PutFile
     class: org.apache.nifi.minifi.processors.PutFile
     max concurrent tasks: 1
     scheduling strategy: TIMER_DRIVEN
     scheduling period: 1000 ms
     penalization period: 30000 ms
     yield period: 1000 ms
     run duration nanos: 0
     auto-terminated relationships list:
     - failure
     - success
     Properties:
       Conflict Resolution Strategy: fail
       Create Missing Directories: 'true'
       Directory: /tmp/put/${now():format('Y')}
       Maximum File Count: '-1'
   Controller Services: []
   Process Groups: []
   Input Ports: []
   Output Ports: []
   Funnels: []
   Connections:
   - id: d0fc88e8-f0e4-44fc-b20b-225a8ce7a787
     name: GenerateFlowFile/success/PutFile
     source id: a0a92728-54aa-4f22-875e-287244a4422c
     source relationship names:
     - success
     destination id: 200b4e82-0edd-4cfa-b8c2-f9f86fa027a6
     max work queue size: 0
     max work queue data size: 10000 B
     flowfile expiration: 0 seconds
     queue prioritizer class: ''
   Remote Process Groups: []
   NiFi Properties Overrides: {}```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message