flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Distinct lines in a file
Date Thu, 30 Apr 2015 11:52:43 GMT
Hi Flavio,

I agree, distinct() is a bit limited right now and in fact, there is no
good reason for that except nobody found time to improve it.
You can use distinct(KeySelector k) to work directly on DataSet<String> but
that's not very convenient either:

DataSet<String> strings = env.fromElements("Hello", "Hello", "World",

strings.distinct(new KeySelector<String, String>() {
   public String getKey(String value) throws Exception {
      return value;

Making distinct more generic should take long.
I'll open a JIRA and might eventually fix it, if nobody picks it up.

Cheers, Fabian

2015-04-30 12:05 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:

> Hi to all,
> I'd like to do a unique lines of a file with Flink. Do I really need to
> make a map from String to Tuple1<String>, call unique() and then another
> map from Tuple1 to String again before output?
> Is there a smarter way to do it?
> Best,
> Flavio

View raw message