flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: Flume Exception - http source
Date Fri, 13 Nov 2015 19:34:13 GMT
Glad you understand it more now.

How could the documentation be improved to make things more clear?

You are welcome to contribute a change, or suggest one here (
https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute)


-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Fri, Nov 13, 2015 at 8:04 AM, Timothy Garza <
timothy.garza@collinsongroup.com> wrote:

> The documentation unfortunately isn’t very clear on this particular
> Flume-source.
>
>
>
> I’ve reconfigured the Flume Agent to bind to (1.) *IP 0.0.0.0 and port
> 2443*  and in a second attempt to (2) *IP <local IP address> and port
> 2443*. It seems to be happy now.
>
>
>
> What was not evident to me (until now) is that an intermediary process is
> required to POST from the actual source of data (e.g. a web-API) to the
> Flume http Source web-service hosted on the master node of a Hadoop cluster.
>
>
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> *Sent:* 12 November 2015 23:24
> *To:* user@flume.apache.org
> *Subject:* Re: Flume Exception - http source
>
>
>
> I don't think all is in order. HTTP Source is trying to listen on a port
> which has another process already listening on.
>
>
>
>
> Thanks,
>
> Hari
>
>
>
> On Thu, Nov 12, 2015 at 3:01 PM, Timothy Garza <
> timothy.garza@collinsongroup.com> wrote:
>
> So why the exception if all is in order?
>
>
>
>
>
> *From:* iain wright [mailto:iainwrig@gmail.com]
> *Sent:* 12 November 2015 22:53
>
>
> *To:* user@flume.apache.org
> *Subject:* Re: Flume Exception - http source
>
>
>
> "A source which accepts Flume Events by HTTP POST and GET. GET should be
> used for experimentation only." --
> https://flume.apache.org/FlumeUserGuide.html#http-source
>
>
>
> This is a source that *accepts* data, it will not * poll* an api. You
> would need your API to push events to flume, or have some tooling
> inbetween, that polled the API & posted the result to flume.
>
>
>
> bind
>
> 0.0.0.0
>
> The hostname or IP address to listen on
>
>
> --
>
> Iain Wright
>
>
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
>
> On Thu, Nov 12, 2015 at 2:35 PM, Timothy Garza <
> timothy.garza@collinsongroup.com> wrote:
>
> Should there be an exception, whilst Flume is listening?
>
>
>
>
>
> *Kind Regards *
>
>
> *Timothy Garza*
>
> Database and BI Developer
> *Collinson Technology Services*
>
> Skype: timothy.garza.cts
>
> collinsongroup.com <http://www.collinsongroup.com/>
>
>
>
> [image: Collinson Group] <http://www.collinsongroup.com/>
>
>
>
>
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com]
> *Sent:* 12 November 2015 22:02
> *To:* user@flume.apache.org
> *Subject:* Re: Flume Exception - http source
>
>
>
> HTTP Source does not pull anything from an address, instead it binds to
> the address specified in the config. It can listen on that port and then
> accept data from applications.
>
>
> Thanks,
> Hari Shreedharan
>
>
>
> On Nov 12, 2015, at 1:59 PM, Timothy Garza <
> timothy.garza@collinsongroup.com> wrote:
>
>
>
> Hi Flumers
>
>
>
> I’m new to Flume so please go easy on me, but this is a placeholder for
> those that follow.
>
>
>
> I’m seeing the following Exception when configuring a http source for
> Flume v1.6.0, pointing at an externally hosted web API I know I can connect
> to, and GET/POST to otherwise
>
>
>
> 15/11/12 21:23:34 ERROR lifecycle.LifecycleSupervisor: Unable to start
> EventDrivenSourceRunner: {
> source:org.apache.flume.source.http.HTTPSource{name:<source-name>,state:IDLE}
> } - Exception follows.
>
> java.lang.IllegalStateException: Running HTTP Server found in source:
> <source-name> before I started one.Will not attempt to start.  “ßException
> text!!!”
>
>         at
> com.google.common.base.Preconditions.checkState(Preconditions.java:145)
>
>         at
> org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:176)
>
>         at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
>
>         at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> Here are my config lines for the Flume source:
>
> *flume-ng agent *
>
> -Dflume.root.logger=WARN,console
>
>
>
> *--conf-file*
>
> <Flume-Agent>.sources.<source-name>.type = http
>
> <Flume-Agent>.sources.<source-name>.port = 443
>
> <Flume-Agent>.sources.<source-name>.bind = https://<url to a perfectly
> healthy web API> ß serving JSON data
>
> <Flume-Agent>.sources.<source-name>.handler =
> org.apache.flume.source.http.JSONHandler
>
> <Flume-Agent>.sources.<source-name>.selector.type = replicating ß multiple
> sinks
>
>
>
> Now I’ve seen others have had to recompile the Flume Jar to resolve this
> but is that really what’s going on here?
>
>
> https://community.cloudera.com/t5/Data-Ingestion-Integration/Flume-TwitterSource-language-filter/td-p/23519
>
> https://github.com/cloudera/cdh-twitter-example/issues/10
>
>
>
> Apart from the Exceptions, what other logging will help diagnose this?
>
>
>
>
>
> The Collinson Group Limited; Registered number: 2577557, Registered in
> England & Wales; Registered Office: Cutlers Exchange, 123 Houndsditch,
> London, EC3A 7BU.
>
>
> This e-mail may contain privileged and confidential information and/or
> copyright material and is intended for the use of the addressee only. If
> you receive this e-mail by mistake please advise the sender immediately by
> using the reply facility in your e-mail software and delete this e-mail
> from your computer system. You may not deliver, copy or disclose its
> contents to anyone else. Any unauthorised use may be unlawful. Any views
> expressed in this e-mail are those of the individual sender and may not
> necessarily reflect the views of The Collinson Group Ltd and/or its
> subsidiaries or any other associated company (collectively “Collinson
> Group”).
>
> As communications via the Internet are not secure Collinson Group cannot
> accept any liability if this e-mail is accessed by third parties during the
> course of transmission or is modified or amended in any way following
> despatch. Collinson Group cannot guarantee that any attachment to this
> email does not contain a virus, therefore it is strongly recommended that
> you carry out your own virus check before opening any attachment, as we
> cannot accept liability for any damage sustained as a result of software
> virus infection. Senders of messages shall be taken to consent to the
> monitoring and recording of e-mails addressed to members of the Company.
>
>
>
>
>
>
>

Mime
View raw message