flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 15:29:06 GMT
HI Alex,
Anyways i made some progress. I think in the agent properties we need to
give the exact class name of the sink which we are using. I was giving
Esper so it was throwing error like Unable to load the class Esper. So i
tried with the exact class name and it has created sink, and it started
calling the esper engine, but after that no further processing. it is
unable to instantiate the class itself.(Since i put logger.info and
logger.trace in my sink none of them are getting printed on the console.
When i ran the agent, the following information is printed on the console:

2012-03-28 00:12:13,602 (conf-file-poller-0) [INFO -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)]
Creating instance of sink log typeEsperSink
2012-03-28 00:12:13,614 (conf-file-poller-0) [DEBUG -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:77)]
Sink type EsperSink is a custom type
2012-03-28 00:12:13,945 (conf-file-poller-0) [INFO -
com.espertech.esper.core.service.EPServiceProviderImpl.doInitialize(EPServiceProviderImpl.java:393)]
Initializing engine URI 'default' version 4.5.0
2012-03-28 00:12:13,948 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.metric.MetricReportingPath.setMetricsEnabled(MetricReportingPath.java:38)]
Metrics reporting has been disabled, this setting takes affect for all
engine instances at engine initialization time.
2012-03-28 00:12:13,987 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.core.EngineImportServiceImpl.addImport(EngineImportServiceImpl.java:90)]
Adding import java.lang.*
2012-03-28 00:12:13,987 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.core.EngineImportServiceImpl.addImport(EngineImportServiceImpl.java:90)]
Adding import java.math.*
2012-03-28 00:12:13,988 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.core.EngineImportServiceImpl.addImport(EngineImportServiceImpl.java:90)]
Adding import java.text.*
2012-03-28 00:12:13,988 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.core.EngineImportServiceImpl.addImport(EngineImportServiceImpl.java:90)]
Adding import java.util.*
2012-03-28 00:12:13,989 (conf-file-poller-0) [DEBUG -
com.espertech.esper.epl.core.EngineImportServiceImpl.addImport(EngineImportServiceImpl.java:90)]
Adding import com.espertech.esper.client.annotation.*
2012-03-28 00:12:14,127 (conf-file-poller-0) [DEBUG -
com.espertech.esper.timer.TimerServiceImpl.startInternalClock(TimerServiceImpl.java:71)]
.startInternalClock Starting internal clock daemon thread, resolution=100


After this nothing is printed, any idea whats happening?

Regards,
Som Shekhar

On Wed, Mar 28, 2012 at 7:21 PM, alo alt <wget.null@googlemail.com> wrote:

> Oh, now we clear.
> flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm.
> Here you should take a look into the source code (flume-ng-sinks) to write
> own plugins, feel free to submit a jira with a review request. Please note,
> NG isn't ready for production yet.
>
> sorry for confusion, we all talk about flume 0.9.4, which will be shipped
> with cloudera's distribution and support decorators.
>
> - Alex
>
>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote:
>
> > Hello Alexander,
> > I have created a class known as EsperSink.java and EsperListener.java,
> compiled them and created a jar(esper-sink.jar) file.
> >
> > I have put the esper-sink.jar along with all esper related jar into the
> lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib)
> >
> > And my agent file is something like this:
> > # Sources
> > agent1.sources.log.type = exec
> > agent1.sources.log.command=/usr/bin/tail -F
> /home/dev/EsperPOC/SamplePOC/test.txt
> > agent1.sources.log.channels = log
> >
> > # Channgels
> > agent1.channels.log.type = memory
> >
> > # Sinks
> > agent1.sinks.log.type = Esper
> > agent1.sinks.log.hostname = namenode
> > agent1.sinks.log.port = 41414
> > agent1.sinks.log.batch-size = 10
> > agent1.sinks.log.runner.type = polling
> > agent1.sinks.log.runner.polling.interval = 5
> > agent1.sinks.log.channel = log
> >
> > # Load everything
> > agent1.sources = log
> > agent1.sinks = log
> > agent1.channels = log
> >
> >
> > When i am running the agent, it is throwing the following error:
> > 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR -
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)]
> Failed to load configuration data. Exception follows.
> > org.apache.flume.FlumeException: Unable to load sink type: Esper, class:
> Esper
> >
> > There is no file with a name flume-conf.xml only property template and
> this property template is meant for defining sources and sinks.
> > Anyways i will create flume-conf.xml and will add the plugins
> >
> > Regards,
> > Som
> >
> > On Wed, Mar 28, 2012 at 6:59 PM, alo alt <wget.null@googlemail.com>
> wrote:
> > You have to put the them into ./lib (from a flume view) and enable the
> jars in flume-conf.xml:
> >
> > <property>
> >  <name>flume.plugin.classes</name>
> >  <value>here.the.plugins</value>
> >  <description>Comma separated list of plugin classes</description>
> > </property>
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
> > On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote:
> >
> > > Hel,
> > > i have written a custom sink for Esper, can you please how would i
> make FLume to recognize this is my new sink..Do i have to specify in
> configuration file. Can you please help me?
> > >
> > > Regards,
> > > Som
> > >
> > > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com>
> wrote:
> > > Thanks for a quick response..
> > > But the sinks written for cloudera flume (i was going through hello
> world plugin) use cloudera classes and all.. For FLume-728 the sinks
>  written (HDFSSink and IRCSink) is different from cloudera's.
> > >  CLoudera flume requires cloudera's hadoop to be installed, if i am
> not wrong?
> > >
> > > My idea is to collect the events from Flume agent and send to Esper,
> where i will query the streams using EPL.
> > > Can you please tell me what all things are required to write a custom
> sink. i mean what needs to be done so that Flume node recognizes the sink.
> > >
> > > Thanks
> > >
> > > Regards,
> > > Som
> > >
> > >
> > > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com>
> wrote:
> > > Hi,
> > >
> > > Cloudera uses in 3u3 Flume-798
> > >
> > > Apache flume isn't different, you can build it among your installation
> and tweak the configs to match them or use the binary distribution:
> > > https://github.com/cloudera/flume/downloads
> > >
> > > As you read, apache flume 1.1.0 was just released:
> > > http://www.apache.org/dyn/closer.cgi/incubator/flume/
> > >
> > > You have to write plugins for sinks, here you found some created from
> the community:
> > > https://cwiki.apache.org/FLUME/flume-plugins.html
> > >
> > > And the docs:
> > >
> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
> > >
> > > Q3 - I don't know, if you use serialized data whats with avro?
> > >
> > > - Alex
> > >
> > > --
> > > Alexander Lorenz
> > > http://mapredit.blogspot.com
> > >
> > > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> > >
> > > > Hello every one,
> > > > I am trying to integrate FLume and Esper by writing a custom sink
> for Esper. But as per the documentation , i am unable to do so. i have
> tried the hello world plugin, but that one also not working.
> > > > I am using FLume-728.
> > > > I have few question:
> > > >
> > > > Q1. Is Apache Flume is different from Cloudera's Flume?
> > > >        If i am using Cloudera's FLume, then does it have any
> dependency that i have to use CLoudera's distribution of Hadoop.
> > > >
> > > > Q2. What are the necessary steps for writing custom sink ?
> > > >
> > > >
> > > > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old
> java objects), XML or Map as events. Now if i try integrating both, how
> feasible it is?
> > > >
> > > > Regards,
> > > > Som Shekhar Sharma
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

Mime
View raw message