Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of gshapira@cloudera.com
 designates 209.85.213.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <543E7E8F.20308@target2sell.com>
References: <543E7E8F.20308@target2sell.com>
Date: Wed, 15 Oct 2014 08:57:01 -0700
Message-ID: 
 <CAHBV8WdurBmFNuhzdByScDyw_jHy4T9X+PFKWE=B8E1KU85+zw@mail.gmail.com>
Subject: Re: HDFS sink: "clever" routing
From: Gwen Shapira <gshapira@cloudera.com>
To: user@flume.apache.org
Content-Type: text/plain; charset=UTF-8

Yes, this is absolutely possible - but you need to make sure the flume
event has the matching keys in the event header (tenant, type, and
timestamp).
Do this either using interceptors or through a custom source.

On Wed, Oct 15, 2014 at 7:02 AM, Jean-Philippe Caruana
<jp@target2sell.com> wrote:
> Hi,
>
> I am new to Flume (and to HDFS), so I hope my question is not stupid.
>
> I have a multi-tenant application (about 100 different customers as for
> now).
> I have 16 different data types.
>
> (In production, we have approx. 15 million messages/day through our
> RabbitMQ)
>
> I want to write to HDFS all my events, separated by tenant, data type,
> and date, like this :
> /data/{tenant}/{data_type}/2014/10/15/file-08.csv
>
> Is it possible with one sink definition ? I don't want to duplicate
> configuration, and new client arrive every week or so
>
> In documentation, I see
> agent1.sinks.hdfs-sink1.hdfs.path = hdfs://server/events/%Y/%m/%d/%H/
>
> Is this possible ?
> agent1.sinks.hdfs-sink1.hdfs.path =
> hdfs://server/events/%tenant/%type/%Y/%m/%d/%H/
>
> I want to write to different folder according to my incoming data.
>
> Thanks
>
> --
> Jean-Philippe Caruana
> http://www.barreverte.fr
>