nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher McDermott (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage
Date Mon, 04 Apr 2016 19:02:25 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224805#comment-15224805
] 

Christopher McDermott edited comment on NIFI-1726 at 4/4/16 7:01 PM:
---------------------------------------------------------------------

On 4/4/16, 11:40 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" <chris.mcdermott@hpe.com>
wrote:

Hi Joe,

I am running fro 16108467c19f59 plus the pull request for NIFI-1660

Will do on the dump.

Thanks,
Chris

From: Joe Witt <joe.witt@gmail.com<mailto:joe.witt@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Sunday, April 3, 2016 at 8:34 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: Question about content repository archiving


Chris

What version of nifi are you on?

If you happen to see such as case again please do take a thread dump of nifi as well.    bin/nifi.sh
dump

Thanks
Joe

Thanks
Joe

On Apr 3, 2016 8:20 PM, "Andre" <andre-lists@fucs.org<mailto:andre-lists@fucs.org>>
wrote:
Chris,

Can you navigate under /opt/mount2/nifi_data/content_repository and run

du -hsc */archive

keen to see if your archives aren't being deleted after expiry...

Seems pretty much like what I experienced as well ( http://markmail.org/message/37a2dxtkfvhwyty7
)


On Mon, Apr 4, 2016 at 6:09 AM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) <chris.mcdermott@hpe.com<mailto:chris.mcdermott@hpe.com>>
wrote:
I’m having a problem where the disk on which I am storing my NiFi data keeps filling up.
 The content repository is taking up the overwhelming majority of that space.  It seems that
the archiver is not working the way I would expect.

$ du -sh /opt/mount2/nifi_data/*
712K /opt/mount2/nifi_data/conf
147G /opt/mount2/nifi_data/content_repository
656K /opt/mount2/nifi_data/database_repository
318M /opt/mount2/nifi_data/flowfile_repository
937M /opt/mount2/nifi_data/provenance_repository

$ df -lh | grep mount2
/dev/mapper/disk2-mount2
                      148G  148G     0 100% /opt/mount2

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/mount2/nifi_data/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

Can anyone point out what I might be missing?

Thanks,
Chris



was (Author: chris@mcdermott.net):
So the problem cropped up again.  On one of the nodes that is out of space I ran a grep to
find the first errors which show up I also have a thread dump.  Is there some place I can
post these too? 



On 4/4/16, 11:40 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" <chris.mcdermott@hpe.com>
wrote:

Hi Joe,

I am running fro 16108467c19f59 plus the pull request for NIFI-1660

Will do on the dump.

Thanks,
Chris

From: Joe Witt <joe.witt@gmail.com<mailto:joe.witt@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Date: Sunday, April 3, 2016 at 8:34 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <users@nifi.apache.org<mailto:users@nifi.apache.org>>
Subject: Re: Question about content repository archiving


Chris

What version of nifi are you on?

If you happen to see such as case again please do take a thread dump of nifi as well.    bin/nifi.sh
dump

Thanks
Joe

Thanks
Joe

On Apr 3, 2016 8:20 PM, "Andre" <andre-lists@fucs.org<mailto:andre-lists@fucs.org>>
wrote:
Chris,

Can you navigate under /opt/mount2/nifi_data/content_repository and run

du -hsc */archive

keen to see if your archives aren't being deleted after expiry...

Seems pretty much like what I experienced as well ( http://markmail.org/message/37a2dxtkfvhwyty7
)


On Mon, Apr 4, 2016 at 6:09 AM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) <chris.mcdermott@hpe.com<mailto:chris.mcdermott@hpe.com>>
wrote:
I’m having a problem where the disk on which I am storing my NiFi data keeps filling up.
 The content repository is taking up the overwhelming majority of that space.  It seems that
the archiver is not working the way I would expect.

$ du -sh /opt/mount2/nifi_data/*
712K /opt/mount2/nifi_data/conf
147G /opt/mount2/nifi_data/content_repository
656K /opt/mount2/nifi_data/database_repository
318M /opt/mount2/nifi_data/flowfile_repository
937M /opt/mount2/nifi_data/provenance_repository

$ df -lh | grep mount2
/dev/mapper/disk2-mount2
                      148G  148G     0 100% /opt/mount2

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/mount2/nifi_data/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

Can anyone point out what I might be missing?

Thanks,
Chris


> Archiver is not respecting nifi.content.repository.archive.max.usage.percentage
> -------------------------------------------------------------------------------
>
>                 Key: NIFI-1726
>                 URL: https://issues.apache.org/jira/browse/NIFI-1726
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Christopher McDermott
>         Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the disk becomes
100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message