flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlotta Hicks <Carlotta.Hi...@sas.com>
Subject RE: Java API for Flume Agent
Date Fri, 23 Jan 2015 04:05:03 GMT
Here is the code change in EmbeddedAgentConfiguration:

  /**
   * Avro sink which can send events to a downstream avro source. This is the
   * only supported sink for Embedded Agents.
   */
  public static final String SINK_TYPE_AVRO = SinkType.AVRO.name();
  public static final String SINK_TYPE_HBASE = SinkType.HBASE.name();
  public static final String SINK_TYPE_HDFS = SinkType.HDFS.name();

  private static final String[] ALLOWED_SINKS = {
   SINK_TYPE_AVRO,
    SINK_TYPE_HBASE,
    SINK_TYPE_HDFS
  };

This is my code with configurations:

        Map<String, String> properties = new HashMap<String, String>();

        properties.put("channel.type", "file");
        properties.put("sinks", "hdfs1");
        properties.put("hdfs1.type", "hdfs");
        properties.put("hdfs1.path", "hdfs://lax07.unx.sas.com:8020/tmp");
        properties.put("processor.type", "default");

         agent = new EmbeddedAgent(“myAgent”);

         agent.configure(properties);             <-- This statement causes the exception
below

org.apache.flume.FlumeException: Expected one source and got 0
                at org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:160)
                at org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:95)



From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Tuesday, January 20, 2015 3:23 PM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: RE: Java API for Flume Agent

Can you send the code you used?

Thanks,
Hari


On Tue, Jan 20, 2015 at 8:48 AM, Carlotta Hicks <Carlotta.Hicks@sas.com<mailto:Carlotta.Hicks@sas.com>>
wrote:

I added in an HDFS sink. But, I am getting this FlumeException: Expected one source and got
0 .

The only source for the embedded agent is 'embedded', correct? Not sure how to proceed from
here.

Need help.

-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Saturday, January 17, 2015 7:58 AM
To: user@flume.apache.org<mailto:user@flume.apache.org>
Subject: Re: Java API for Flume Agent

Try a bit of hacking in
org.apache.flume.agent.embedded.EmbeddedAgentConfiguration class, and add HBase Sink to allowed
sinks in the section

private static final String[] ALLOWED_SINKS = {
SINK_TYPE_AVRO,
// Add Hbase sink here <<<

};

See if it works and let us know.

HTH!

On Sat, Jan 17, 2015 at 10:19 AM, Carlotta Hicks <Carlotta.Hicks@sas.com<mailto:Carlotta.Hicks@sas.com>>
wrote:
> Thanks Joey! I would love to use the embedded agent. It is exactly what I am looking
for. However, I can't use the embedded agent because with the embedded agent, the sink must
be avro. I need the sink to be hbase.
> I am assuming that requirements has not changed. Please correct me if I am wrong.
> ________________________________________
> From: Joey Echeverria <joey@cloudera.com<mailto:joey@cloudera.com>>
> Sent: Friday, January 16, 2015 3:47 PM
> To: user@flume.apache.org<mailto:user@flume.apache.org>
> Subject: Re: Java API for Flume Agent
>
> You could use an embedded flume agent[1]. This will do the same thing
> that the Flume flow does but run flume inside of another Java
> application.
>
> Let me know if that meets your needs.
>
> -Joey
>
> [1] http://flume.apache.org/FlumeDeveloperGuide.html#embedded-agent
>
>
> On Fri, Jan 16, 2015 at 10:30 AM, Carlotta Hicks <Carlotta.Hicks@sas.com<mailto:Carlotta.Hicks@sas.com>>
wrote:
>> I need to create an agent using java API to load an HBase table using
>> a CSV file. I am able to accomplish this using flume-ng and an agent
>> with a spooldir directory source, file channel and HBase sink.
>>
>>
>>
>> I need information on how to accomplish this via a java API.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> -CM
>>
>>
>
>
>
> --
> Joey Echeverria



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message