flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhipcui (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLUME-3323) Kafka channel should supports disable producer/consumer according to the need
Date Fri, 08 Mar 2019 05:10:00 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zhipcui updated FLUME-3323:
---------------------------
    Description: 
As the doc said:
{noformat}
The Kafka channel can be used for multiple scenarios:

1. With Flume source and sink - it provides a reliable and highly available channel for events
2. With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka
topic, for use by other apps
3. With Flume sink, but no source - it is a low-latency, fault tolerant way to send events
from Kafka to Flume sinks such as HDFS, HBase or Solr{noformat}
 

Kafka channel will generates producer and consumer both now.

When using Kafka channel as scenario 2 (3), the producer(consumer) is just useless, and if
our kafka cluster enable SASL the situation is getting worse.

Assume we have a kafka account which only has read permissions, and using kafka channel
without source, producer shouldn't be generated.

I propose that add two properties to disable producer/consumer.

 
{noformat}
agent.channels.c.noProducer = false
agent.channels.c.noConsumer= false{noformat}
 

 

 

 

  was:
As the doc said:
{noformat}
The Kafka channel can be used for multiple scenarios:

1. With Flume source and sink - it provides a reliable and highly available channel for events
2. With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka
topic, for use by other apps
3. With Flume sink, but no source - it is a low-latency, fault tolerant way to send events
from Kafka to Flume sinks such as HDFS, HBase or Solr{noformat}
 

Kafka channel will generates producer and consumer both now.

When using Kafka channel as scenario 2 (3), the producer(consumer) is just useless, and if
our kafka cluster enable SASL the situation is getting worse.

Assume we have a kafka account which only has read permissions, and using kafka channel
without source, producer shouldn't be generated.

I propose that add two properties to disable producer/consumer.

 
{noformat}
agent.channels.c.withSource = false
agent.channels.c.withSink= false{noformat}
 

 

 

 


> Kafka channel should supports disable producer/consumer according to the need
> -----------------------------------------------------------------------------
>
>                 Key: FLUME-3323
>                 URL: https://issues.apache.org/jira/browse/FLUME-3323
>             Project: Flume
>          Issue Type: Improvement
>          Components: Kafka Channel
>    Affects Versions: 1.9.0
>            Reporter: zhipcui
>            Priority: Minor
>
> As the doc said:
> {noformat}
> The Kafka channel can be used for multiple scenarios:
> 1. With Flume source and sink - it provides a reliable and highly available channel for
events
> 2. With Flume source and interceptor but no sink - it allows writing Flume events into
a Kafka topic, for use by other apps
> 3. With Flume sink, but no source - it is a low-latency, fault tolerant way to send events
from Kafka to Flume sinks such as HDFS, HBase or Solr{noformat}
>  
> Kafka channel will generates producer and consumer both now.
> When using Kafka channel as scenario 2 (3), the producer(consumer) is just useless, and
if our kafka cluster enable SASL the situation is getting worse.
> Assume we have a kafka account which only has read permissions, and using kafka channel
without source, producer shouldn't be generated.
> I propose that add two properties to disable producer/consumer.
>  
> {noformat}
> agent.channels.c.noProducer = false
> agent.channels.c.noConsumer= false{noformat}
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message