nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saulo Sobreiro (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MINIFI-425) MiNiFi contents_repository config problem
Date Wed, 17 Jan 2018 18:01:07 GMT

    [ https://issues.apache.org/jira/browse/MINIFI-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329114#comment-16329114
] 

Saulo Sobreiro commented on MINIFI-425:
---------------------------------------

[~aldrin] by the time of the log, I would say that all those restarts happened when I was
trying to apply the configurations through nifi.properties, so yes, I think those are my fault.

 

"_If not, my suspicion is that you were encountering a heap issue from the SplitText.  Typically
this is aided by tiering SplitText processors (split a very large file into 1k chunks and
then do individual splits)._"

Indeed I had that issue but as you can find in the config.yml I implemented the solution you
are suggesting :). Difference is that my first SplitText is configured for 10k instead of
1k. Can this difference create a problem?

I attached a new log file minifi-bootstrap_2018-01-10.log from the day I had, and hopefully
fixed this issue. I hope it can provide furthers insights regarding the problem.

 

Indeed the minifi-app.log is lite, but I guess it is because it just has logs for the current
day... Looking for older logs I found only errors related to disk space like (minifi-app.log
for 15/01):
{code:java}
2018-01-15 21:52:13,560 INFO [pool-31-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating
checkpoint of FlowFile Repo
sitory
2018-01-15 21:52:13,685 ERROR [pool-31-thread-1] org.wali.MinimalLockingWriteAheadLog Failed
to create new journal for [Parti
tion-0, java.io.IOException: No space left on device] due to {}
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:788)
at org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:528)
at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:45
1)
at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:423)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:1
80)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFuture
{code}
Every other log looks like the one I attached.

 

 

> MiNiFi contents_repository config problem
> -----------------------------------------
>
>                 Key: MINIFI-425
>                 URL: https://issues.apache.org/jira/browse/MINIFI-425
>             Project: Apache NiFi MiNiFi
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>            Reporter: Saulo Sobreiro
>            Priority: Major
>         Attachments: config.yml, minifi-app.log, minifi-bootstrap.log, minifi-bootstrap_2018-01-10.log
>
>
> *Problem:*
> I found 2 problems in this my MiNiFi-NiFi setup:
> - The first is related to my MiNiFi ./content_repository. This path is using 
> around 120GB of disk space, which is far more than I can afford for this 
> application; 
> - The second is related to the solution I tried to the previous problem. I 
> tried to configure the MiNiFi nifi.properties with the following values: 
> ´´´ 
> nifi.content.repository.archive.max.retention.period=6 hours 
> nifi.content.repository.archive.max.usage.percentage=5% 
> ´´´ 
> However, when I restart MiNiFi this instance for it to use the new 
> properties, I noticed that it overwrites my nifi.properties file with some 
> default values erasing the new values. 
>  
> *Versions in use:*
> NiFi - version 1.2.0 installed with HDF (3.0.2.0-76) 
> MiNiFi - version 0.3.0 (transfered from 
> [http://mirrors.up.pt/pub/apache/nifi/minifi/0.3.0/minifi-0.3.0-bin.tar.gz]) 
> NiFi OS - RHEL 7.4
> MiNiFi OS - RHEL 7.2 
>  
> *config.yml*
> The config.yml file can be found attached. It is the version used in production after
replacing some information like URLs. 
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message