nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Clarke (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NIFI-1626) ZooKeeper State Provider should block requests > 1 MB
Date Tue, 15 Mar 2016 13:02:33 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195241#comment-15195241
] 

Matthew Clarke edited comment on NIFI-1626 at 3/15/16 1:01 PM:
---------------------------------------------------------------

So i saw some discussions on line where people suggested breaking messages larger than 1MB
in to multiple requests to zookeeper.  I saw nothing that said how to do this.  Feel like
zookeeper set caps and said we will just fail fast if they are exceeded.  Good news is that
i think we figured out the snapshot/transaction log rolling thing.  We need to set the snapCount
and should probably add this to the zookeeper.properties file. If not set it defaults to
100,000 events.  We set it to 1,000 in our tests with listHDFS using "snapCount=1000". First
couple logs rolled at ~129MB and then each after that got a larger and larger (my guess is
because of the increasing size of the events).  So when a single event gets large, the transaction
log size skyrockets.  Even when zookeepers fails, the transaction log still continues to
record events (even the large ones of 1MB).  It too eventually fails.  Not sure what is
the ultimate trigger there.


was (Author: msclarke):
So i saw some discussions on line where people suggested breaking messages larger than 1MB
in to multiple requests to zookeeper.  I saw nothing that said how to do this.  Feel like
zookeeper set caps and said we will just fail fast if they are exceeded.  Good news is that
i think we figured out the snapshot/transaction log rolling thing.  We need to set the snapCount
and should probably add this to the zookeeper.properties file. If not set it defaults to
10,000 events.  We set it to 1,000 in our tests with listHDFS using "snapCount=1000". First
couple logs rolled at ~129MB and then each after that got a larger and larger (my guess is
because of the increasing size of the events).  So when a single event gets large, the transaction
log size skyrockets.  Even when zookeepers fails, the transaction log still continues to
record events (even the large ones of 1MB).  It too eventually fails.  Not sure what is
the ultimate trigger there.

> ZooKeeper State Provider should block requests > 1 MB
> -----------------------------------------------------
>
>                 Key: NIFI-1626
>                 URL: https://issues.apache.org/jira/browse/NIFI-1626
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.6.0
>
>
> ZooKeeper does not support setting data on a ZNode to a value greater than 1 MB. However,
the client doesn't does not check this - it sends the data to the server and lets the server
reject the request. This can cause problems with ZooKeeper if the requests grow extremely
large. As a result, we should prevent this from ever happening by throwing an Exception ourselves
if attempting to set state to a value > 1 MB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message