Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <BANLkTi=1B16pZ5RTQm8_+-6Gc7AekfXeVg@mail.gmail.com>
References: <BANLkTi=juG=PBFdvnijOfD-OyVRviMxeXg@mail.gmail.com>
	<BANLkTinGe101cNpdDH18e0mXpcWH2ad-0A@mail.gmail.com>
	<BANLkTi=t=5TZgMQX09utQ+bimqramWrt+g@mail.gmail.com>
	<BANLkTi=1B16pZ5RTQm8_+-6Gc7AekfXeVg@mail.gmail.com>
Date: Mon, 9 May 2011 10:12:41 +0100
Message-ID: <BANLkTikKNWjVMWYA=4E3-1Dbv6jsRmZY4g@mail.gmail.com>
Subject: Re: Camel route stops sending messages
From: Tom Howe <tomhowe@artcore.com>
To: users@camel.apache.org
Content-Type: multipart/alternative; boundary=002354447931d69ec404a2d43d36

--002354447931d69ec404a2d43d36
Content-Type: text/plain; charset=ISO-8859-1

What do you mean by propagates to file and point?

We do actually have an error handler - the previous route I posted was just
a summary. Below is the full config for the 2 camel contexts in the app. The
first is for the main message routing, the 2nd is for the heartbeat.

The app sends messages to splunk and a heartbeart to an activeMQ.

The symptons we are finding is that occasionally, the application completely
stops sending messages. The log which normally shows both messages being
sent suddendly quietens down and only shows the heartbeat messages (even
though they are not arriving either). After a restart it continues fine.

The exceptions found in the log that I previously pasted do not coincide
with the time at which it stops so may be a red herring.

This application runs on quite heavily loaded VMs. We wonder if it could be
due to memory issues as other applications on the box have died due to
running out of memory.

   <camelContext id="main" autoStartup="true" xmlns="
http://camel.apache.org/schema/spring">

        <errorHandler id="mr.error" type="DeadLetterChannel"
deadLetterUri="activemq:mr.error">
            <redeliveryPolicy maximumRedeliveries="10" redeliveryDelay="100"
logHandled="true"/>
        </errorHandler>

        <errorHandler id="mr.splunk.error" type="DeadLetterChannel"
deadLetterUri="activemq:mr.splunk.error">
            <redeliveryPolicy maximumRedeliveries="10" redeliveryDelay="100"
logHandled="true"/>
        </errorHandler>

        <!--direct endpoints-->
        <endpoint id="input" uri="${mr.input.endpoint}"/>
        <endpoint id="output" uri="${mr.output.endpoint}"/>

        <endpoint id="splunk" uri="${splunk.endpoint}"/>
        <!--<endpoint id="splunk.in" uri="${splunk.in.endpoint}"/>-->

        <!-- convert ispy messages to xml and place in general in queue -->
        <route autoStartup="true" errorHandlerRef="mr.error" xmlns="
http://camel.apache.org/schema/spring">
            <from ref="input"/>
            <!--<log message="Message passed f=${header.CamelFileName}
#${header.JMSMessageID} from: ${header.JMSDestination} to output"-->
                 <!--loggingLevel="INFO"/>-->
            <convertBodyTo type="java.lang.String"/>
            <bean ref="ispyJsonToXml" method="process"/>
            <choice>
                <when>
                    <xpath>/*[starts-with(@name, 'wfe.')]</xpath>
                    <to uri="direct:splunk.in"/>
                </when>
                <otherwise>
                    <log message="sending message to activmq"
loggingLevel="INFO"/>
                    <to ref="output"/>
                </otherwise>
            </choice>
        </route>


        <!-- filter and convert all messages for splunk to splunk format -->
        <route autoStartup="true" errorHandlerRef="mr.splunk.error" xmlns="
http://camel.apache.org/schema/spring">
            <from uri="direct:splunk.in"/>
            <choice>
                <when>
                    <!-- convert ispy-events -->
                    <xpath>/*[local-name() = "event"][./*/*[local-name() =
"ispy-event"]]</xpath>
                    <bean ref="ispyXmlToSplunk"/>
                </when>
                <otherwise>
                    <!-- convert everything else by grabbing envelope
information -->
                    <bean ref="envelopeXmlToSplunk"/>
                </otherwise>
            </choice>
            <log message="sending message to splunk" loggingLevel="INFO"/>
            <to ref="splunk"/>
       </route>


    </camelContext>

  <bean id="createHeartBeat" class="agentsmith.generators.CreateHeartBeat">
        <!--<property name="eventName" value="${personality}.heartbeat"/>-->
        <property name="eventName" ref="heartBeatName"/>
    </bean>

    <camelContext id="heartbeat"  autoStartup="true" xmlns="
http://camel.apache.org/schema/spring">

        <endpoint id="general2.in" camelContextId="heartbeat"
uri="${general.in.endpoint}"/>

        <endpoint id="heartbeattimer" camelContextId="heartbeat"
uri="quartz://mygroup/heartbeat?cron=0+*+*+?+*+*"/>

        <route autoStartup="true"  inheritErrorHandler="true" xmlns="
http://camel.apache.org/schema/spring">
            <from ref="heartbeattimer"/>
            <bean ref="createHeartBeat"/>
            <to ref="general2.in"/>
        </route>

    </camelContext>


On Mon, May 9, 2011 at 9:49 AM, Filippo Balicchia <fbalicchia@gmail.com>wrote:

> If there is no explicit error handling the default error i triggered,
> and the exception is propagates it back to the caller.
> In your case seems to have network error then then error is propagates
> to file and point.
>
> To resolve your problem i just add 'delay' parameter on your
> expression. With this parameter you instruct camel
> to wait  milliseconds before the next poll of the file/directory
>
> Cheers.
>
> --Filippo
>
>
> 2011/5/8 Tom Howe <tomhowe@artcore.com>:
> > Most of the messages are flowing to splunk fine. A few are failing to get
> > through - possibly we are flooding splunk - hence those errors.
> > But what I dont understand is why the process then stops processing any
> more
> > messages.
> >
> >
> > On Sun, May 8, 2011 at 6:07 AM, Filippo Balicchia <fbalicchia@gmail.com
> >wrote:
> >
> >> Does your splunk  accept you connection ?
> >>
> >> From stacktrace seems that refused your connection
> >>
> >>
> >>  2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException:
> >>  Connection refused2011-05-07 12:41:47.042078500   at
> >>
> >>
> >> --Filippo
> >>
> >>
> >> 2011/5/7 Tom Howe <tomhowe@artcore.com>:
> >> > I have a simple java app that collect files of disk, converts the
> [json]
> >> > contents and then forwards it to a splunk instance over tcp using
> mina.
> >> >
> >> >
> >> >        <endpoint id="input" uri="${input.endpoint}"/>
> >> >        <endpoint id="output" uri="${output.endpoint}"/>
> >> >
> >> >        <route autoStartup="true" xmlns="
> >> > http://camel.apache.org/schema/spring">
> >> >            <from ref="input"/>
> >> >            <convertBodyTo type="java.lang.String"/>
> >> >            <bean ref="jsonToSplunk"/>
> >> >            <to ref="splunk" />
> >> >        </route>
> >> >
> >> > input.endpoint =
> >> > file:/dir/to/files?delete=true&recursive=true&maxMessagesPerPoll=500
> >> > output.endpoint =
> >> > mina:tcp://splunkhost/?textline=true&textlineDelimiter=UNIX&sync=false
> >> >
> >> > It also emits a heartbeat message  every minute over JMS using camel
> >> quartz
> >> > endpoint.
> >> >
> >> > The problem I am having is that after about a day of running, the
> process
> >> > stops picking up files and sending them to splunk and stops sending
> the
> >> > heartbeat JMS messages. However, the log is still showing that it is
> >> > emitting messages.
> >> >
> >> > It's like the camel routes have just stopped working properly but I
> have
> >> no
> >> > idea why.
> >> >
> >> >
> >> > I see a bunch of these exceptions in the log like the following..
> which
> >> is
> >> > suggesting it is failing to connect to splunk sometimes.
> >> >
> >> > 2011-05-07 12:41:50.457659500 2011-05-07 13:41:50,456 [Camel (main)
> >> thread
> >> > #0 - file:///home/wfe/var/ispy/events] ERROR
> >> > org.apache.camel.processor.DeadLetterChannel  - Failed delivery for
> >> > exchangeId: ID-vp-prod-wfe-a-61766-1304769141692-0-95088. Exhausted
> after
> >> > delivery attempt: 11 caught:
> org.apache.mina.common.RuntimeIOException:
> >> > Failed to get the session.. Processed by failure processor:
> >> > sendTo(Endpoint[activemq://splunk.error] InOnly)
> >> > 2011-05-07 12:41:50.457664500
> org.apache.mina.common.RuntimeIOException:
> >> > Failed to get the session.
> >> > 2011-05-07 12:41:50.457952500   at
> >> >
> >>
> org.apache.mina.common.support.DefaultConnectFuture.getSession(DefaultConnectFuture.java:63)
> >> > 2011-05-07 12:41:50.457953500   at
> >> >
> >>
> org.apache.camel.component.mina.MinaProducer.openConnection(MinaProducer.java:211)
> >> > 2011-05-07 12:41:50.457954500   at
> >> >
> >>
> org.apache.camel.component.mina.MinaProducer.process(MinaProducer.java:80)
> >> > 2011-05-07 12:41:50.457955500   at
> >> >
> >>
> org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)
> >> > 2011-05-07 12:41:50.457963500   at
> >> >
> >>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
> >> > 2011-05-07 12:41:50.457964500   at
> >> >
> >>
> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104)
> >> > ..snip..
> >> > 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException:
> >> > Connection refused2011-05-07 12:41:47.042078500   at
> >> > sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> > 2011-05-07 12:41:47.042082500   at
> >> > sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
> >> > 2011-05-07 12:41:47.042083500   at
> >> >
> >>
> org.apache.mina.transport.socket.nio.SocketConnector.processSessions(SocketConnector.java:293)
> >> > 2011-05-07 12:41:47.042084500   at
> >> >
> >>
> org.apache.mina.transport.socket.nio.SocketConnector.access$800(SocketConnector.java:53)
> >> > 2011-05-07 12:41:47.042085500   at
> >> >
> >>
> org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:383)
> >> > 2011-05-07 12:41:47.042089500   at
> >> >
> >>
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> >> > 2011-05-07 12:41:47.042090500   ... 3 more
> >> >
> >> > Any idea if this could be causing the routes to stop?
> >> >
> >> > Let me know if I should provide extra information.
> >> >
> >> > Thanks, Tom
> >> >
> >>
> >
>

--002354447931d69ec404a2d43d36--