nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Conrad Crampton <conrad.cramp...@SecData.com>
Subject Re: NPE in ListenSyslog processor
Date Tue, 25 Apr 2017 17:07:51 GMT
Hi,
Thanks for the swift reply (as usual).
NIFI-3738 created [1].

I have passed over to infrastructure to try and establish cause of the zero length datagrams,
but at least I now know there isn’t anything fundamentally wrong here and can (safely) ignore
the errors.

Thanks
Conrad


[1] https://issues.apache.org/jira/browse/NIFI-3738

On 25/04/2017, 17:46, "Bryan Bende" <bbende@gmail.com> wrote:

    Hi Conrad,
    
    Line 431 of ListenSyslog has the following code:
    
    if (!valid || !event.isValid())
    
    So to get an NPE there means event must be null, and event comes from this code:
    
    boolean valid = true;
    try {
        event = parser.parseEvent(rawSyslogEvent.getData(), sender);
    } catch (final ProcessException pe) {
        getLogger().warn("Failed to parse Syslog event; routing to invalid");
        valid = false;
    }
    
    The parser returns null if the bytes sent in are null or length 0.
    
    We should be checking if (!valid || event == null || !event.isValid())
    to avoid this case, and I think a similar situation exists in the
    ParseSyslog processor. It appears this would only happen if parsing
    messages is enabled in ListenSyslog.
    
    Do you want to create a JIRA for this?
    
    The other question is why you are ending up with these 0 length
    messages, but that one I am not sure about. In the case of UDP, its
    just reading from a datagram channel into a byte buffer and passing
    those bytes a long, so I think it means its receiving a 0 byte
    datagram from the sender.
    
    Thanks,
    
    Bryan
    
    
    On Tue, Apr 25, 2017 at 12:31 PM, Conrad Crampton
    <conrad.crampton@secdata.com> wrote:
    > Hi,
    >
    > Been away for a bit from this community due to other work pressures, but
    > picking up Nifi again and successfully upgraded to 1.1.2 (apart from
    > screwing up one of the nodes temporarily).
    >
    > So, with the renewed interest in log processing our infrastructure team has
    > put in an F5 load balancer to distribute the syslog traffic I am collecting
    > to my 6 node cluster. This is to stop one node being the only workhorse for
    > receiving syslog traffic. I had previously used the ‘standard’ pattern of
    > having the ListenSyslog processor connect to a RPG and then the rest of my
    > data processing flow receive via a local port – to effectively distribute
    > the processing load. I was finding though that the single node was getting
    > too many warnings about buffer, sockets being full etc. – hence the external
    > load balancing.
    >
    >
    >
    > I am no load balancing expert, but what I believe happens is the F5 load
    > balancer receives syslog traffic (over UDP) then distributes this load to
    > all Nifi nodes (gives a bit of syslog traffic to each I believe). All
    > appears fine, but then I start getting NPE in my node logs thus:
    >
    >
    >
    > 2017-04-25 17:16:34,832 ERROR [Timer-Driven Process Thread-7]
    > o.a.n.processors.standard.ListenSyslog
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf]
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process due
    > to java.lang.NullPointerException; rolling back session:
    > java.lang.NullPointerException
    >
    > 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
    > o.a.n.processors.standard.ListenSyslog
    >
    > java.lang.NullPointerException: null
    >
    >         at
    > org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431)
    > ~[nifi-standard-processors-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
    > ~[nifi-api-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    > [na:1.8.0_51]
    >
    >         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    > [na:1.8.0_51]
    >
    >         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
    >
    > 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
    > o.a.n.processors.standard.ListenSyslog
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf]
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] failed to process
    > session due to java.lang.NullPointerException:
    > java.lang.NullPointerException
    >
    > 2017-04-25 17:16:34,833 ERROR [Timer-Driven Process Thread-7]
    > o.a.n.processors.standard.ListenSyslog
    >
    > java.lang.NullPointerException: null
    >
    >         at
    > org.apache.nifi.processors.standard.ListenSyslog.onTrigger(ListenSyslog.java:431)
    > ~[na:na]
    >
    >         at
    > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
    > ~[nifi-api-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
    > ~[nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >        at
    > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
    > [nifi-framework-core-1.1.2.jar:1.1.2]
    >
    >         at
    > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    > [na:1.8.0_51]
    >
    >         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    > [na:1.8.0_51]
    >
    >         at
    > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    > [na:1.8.0_51]
    >
    >         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
    >
    > 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7]
    > o.a.n.processors.standard.ListenSyslog
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] Processor
    > Administratively Yielded for 1 sec due to processing failure
    >
    > 2017-04-25 17:16:34,833 WARN [Timer-Driven Process Thread-7]
    > o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
    > ListenSyslog[id=0a932c37-0158-1000-0000-0000656754bf] due to uncaught
    > Exception: java.lang.NullPointerException
    >
    > 2017-04-25 17:16:34,834 WARN [Timer-Driven Process Thread-7]
    > o.a.n.c.t.ContinuallyRunProcessorTask
    >
    >
    >
    > is what I’m trying to do screwing up the processor? If so, is this a)
    > something that can be done with a load balancer – I’ve seen blog posts about
    > putting an HA proxy in front of Nifi for other ListXX processors but not
    > ListenSyslog, and b) if can’t be done is there suggestions on how I can
    > better distribute load other than the standard pattern using RPG (which only
    > distributes the processing of the data once it is in).
    >
    >
    >
    > Many thanks
    >
    > Conrad
    >
    >
    >
    > SecureData, combating cyber threats
    >
    > ________________________________
    >
    > The information contained in this message or any of its attachments may be
    > privileged and confidential and intended for the exclusive use of the
    > intended recipient. If you are not the intended recipient any disclosure,
    > reproduction, distribution or other dissemination or use of this
    > communications is strictly prohibited. The views expressed in this email are
    > those of the individual and not necessarily of SecureData Europe Ltd. Any
    > prices quoted are only valid if followed up by a formal written quote.
    >
    > SecureData Europe Limited. Registered in England & Wales 04365896.
    > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
    > Maidstone, Kent, ME16 9NT
    
    
     ***This email originated outside SecureData***
    
    Click https://www.mailcontrol.com/sr/Ua65DeG3YiLGX2PQPOmvUqEZpHzWxmiIfINdU0qyFhoA5x+dEtpBxXgn6ouHwVqxHTbTRgxltt8YNBM7q0TQEQ==
 to report this email as spam.
    

Mime
View raw message