Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: Dynamic partitioning for stream output
From: Kostas Kloudas <k.kloudas@data-artisans.com>
In-Reply-To: <1464088960939-7122.post@n4.nabble.com>
Date: Tue, 24 May 2016 14:50:32 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <99D798EB-0E63-4D45-B31E-E238DE02877D@data-artisans.com>
References: <1464088960939-7122.post@n4.nabble.com>
To: user@flink.apache.org
archived-at: Tue, 24 May 2016 12:50:41 -0000

Hi Juho,

If I understand correctly, you want a custom RollingSink that caches =
some=20
buckets, one for each topic/date key, and whenever the volume of data =
buffered
exceeds a limit, then it flushes to disk, right?=20

If this is the case, then you are right that this is not currently =
supported
out-of-the-box, but it would be interesting to update the RollingSink=20
to support such scenarios.

One clarification: when you say that you want partition by date,
you mean the date of the event, right? Not the processing time.

Kostas

> On May 24, 2016, at 1:22 PM, Juho Autio <juho.autio@rovio.com> wrote:
>=20
> Could you suggest how to dynamically partition data with Flink =
streaming?
>=20
> We've looked at RollingSink, that takes care of writing batches to S3, =
but
> it doesn't allow defining the partition dynamically based on the tuple
> fields.
>=20
> Our data is coming from Kafka and essentially has the kafka topic and =
a
> date, among other fields.
>=20
> We'd like to consume all topics (also automatically subscribe to new =
ones)
> and write to S3 partitioned by topic and date, for example:
>=20
> s3://bucket/path/topic=3Dtopic2/date=3D20160522/
> s3://bucket/path/topic=3Dtopic2/date=3D20160523/
> s3://bucket/path/topic=3Dtopic1/date=3D20160522/
> s3://bucket/path/topic=3Dtopic1/date=3D20160523/
>=20
> There are two problems with RollingSink as it is now:
> - Only allows partitioning by date
> - Flushes the batch every time the path changes. In our case the =
stream can
> for example have a random mix of different topics and that would mean =
that
> RollingSink isn't able to respect the max flush size but keeps =
flushing the
> files pretty much on every tuple.
>=20
> We've thought that we could implement a sink that internally creates =
and
> handles multiple RollingSink instances as needed for partitions. But =
it
> would be great to first hear any suggestions that you might have.
>=20
> If we have to extend RollingSink, it would be nice to make it take a
> partitioning function as a parameter. The function would be called for =
each
> tuple to create the output path.
>=20
>=20
>=20
> --
> View this message in context: =
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dynami=
c-partitioning-for-stream-output-tp7122.html
> Sent from the Apache Flink User Mailing List archive. mailing list =
archive at Nabble.com.