flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guyle M. Taber" <gu...@gmtech.net>
Subject Flume - Interceptor, regex mapping and two default selectors
Date Thu, 13 Jun 2019 15:42:23 GMT
I've got three channels and three sinks. I'm using a flume regex interceptor and sending only
matching data to a specific sink, but I want all data going to the other two channels/sinks.
I may be doing this wrong, but it seems like I might be able to use multiple channels for
the "selector.default" to send ALL data to the other two sinks. (one in HDFS and the other
in S3).

So what I have below is regex matching data going to the GCS channel. I want EVERYTHING going
to the other two channels (HDFS and S3).

agent.sources.apache.interceptors.ievent.type = regex_extractor
agent.sources.apache.interceptors.ievent.regex = (product_page)
agent.sources.apache.interceptors.ievent.serializers = to_gcs
agent.sources.apache.interceptors.ievent.serializers.to_gcs.name = event_type
agent.sources.apache.selector.type = multiplexing
agent.sources.apache.selector.header = event_type
agent.sources.apache.selector.mapping.product_page = GCSChannel AvroChannel S3Channel
agent.sources.apache.selector.default = AvroChannel S3Channel
Is this an approach that would work? "agent.sources.apache.selector.default = AvroChannel
S3Channel"
Mime
View raw message