Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <1461683544840-6452.post@n4.nabble.com>
References: <1461683544840-6452.post@n4.nabble.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Tue, 26 Apr 2016 18:21:08 +0200
Message-ID: 
 <CAAdrtT0qq4tmEpq6n_4M2g+d4DVwS22yh4o1GrUKWTyzExsyPw@mail.gmail.com>
Subject: Re: Return unique counter using groupReduceFunction
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1148fdf0049f10053165b055

--001a1148fdf0049f10053165b055
Content-Type: text/plain; charset=UTF-8

Hi Biplob,

Flink is a distributed, data parallel system which means that there are
several instances of you ReduceFunction running in parallel, each with its
own timestamp counter.
If you want to have a unique timestamp, you have to set the parallelism of
the reduce operator to 1, but then the program might become inefficient.

Maybe DataSetUtils.zipWithIndex() or DataSetUtils.zipWithUniqueId() are
helpful for you use case.

Best, Fabian


2016-04-26 17:12 GMT+02:00 Biplob Biswas <revolutionisme@gmail.com>:

> Hi,
>
> I am using a groupreduce function to aggregate the content of the objects
> but at the same time i need to return a unique counter from the function
> but
> my attempts are failing and the identifiers are somehow very random and
> getting duplicated.
>
> Following is the part of my code which is supposed to generate a unique
> counter and return it with out.collect.
>
>
>         public static class sumReducer implements
>         GroupReduceFunction<Tuple2&lt;Integer, Point>,
> Tuple5<Integer,Point, Point,
> Long, Long>> {
>
>                 double sum[] = null;
>                 double sumOfSquare[] = null;
>                 long timestamp = 0;
>                 @Override
>                 public void reduce(Iterable<Tuple2&lt;Integer, Point>> in,
> Collector<Tuple5&lt;Integer,Point, Point, Long, Long>> out)
>                                 throws Exception {
>
>                         int id = 0;
>                         long count = 0;
>                         boolean flag = true;
>                         for(Tuple2<Integer, Point> i:in)
>                         {
>                                 if(flag)
>                                 {
>                                         timestamp++;
>                                         System.out.println("uniqueid: " +
> i.f0 + ", t: " + timestamp );
>                                         sum = new double[i.f1.pt.length];
>                                         sumOfSquare = new
> double[sum.length];
>                                         id = i.f0;
>                                         for(int j=0;j<sum.length;j++)
>                                         {
>                                                 sum[j] = i.f1.pt[j];
>                                                 sumOfSquare[j] = i.f1.pt[j]
> * i.f1.pt[j];
>                                         }
>                                         flag = false;
>                                 }
>                                 else
>                                 {
>                                         int len = i.f1.pt.length;
>                                         for(int j=0;j&lt;len;j++)
>                                         {
>                                                 sum[j] += i.f1.pt[j];
>                                                 sumOfSquare[j] += (i.f1.pt[j]
> * i.f1.pt[j]);
>                                         }
>                                 }
>                                 count++;
>                         }
>                         out.collect(new Tuple5&lt;Integer,Point, Point,
> Long, Long>(id,new
> Point(sum), new Point(sumOfSquare),count, timestamp));
>                 }
>
> I want the timestamp to be unique, but even though the code
> "System.out.println("uniqueid: " + i.f0 + ", t: " + timestamp );" executes
> once for each of the identifier (given by i.f0) by which it is grouped and
> then the groupReducce function is called still I get the following output
> for the above println statement.
>
> uniqueid: 2, t: 1
> uniqueid: 1, t: 1
> uniqueid: 7, t: 2
> uniqueid: 9, t: 3
> uniqueid: 6, t: 2
> uniqueid: 3, t: 1
> uniqueid: 5, t: 2
> uniqueid: 8, t: 3
>
> I dont really get why I am getting this discrepancy, probably I am missing
> some Flink concept, I am relatively very new to the flink platform and any
> help is appreciated. Thanks a lot.
>
> Thanks and Regards
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Return-unique-counter-using-groupReduceFunction-tp6452.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

--001a1148fdf0049f10053165b055
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PjxkaXY+PGRpdj5IaSBCaXBsb2IsPGJyPjxicj48L2Rp
dj5GbGluayBpcyBhIGRpc3RyaWJ1dGVkLCBkYXRhIHBhcmFsbGVsIHN5c3RlbSB3aGljaCBtZWFu
cyB0aGF0IHRoZXJlIGFyZSBzZXZlcmFsIGluc3RhbmNlcyBvZiB5b3UgUmVkdWNlRnVuY3Rpb24g
cnVubmluZyBpbiBwYXJhbGxlbCwgZWFjaCB3aXRoIGl0cyBvd24gdGltZXN0YW1wIGNvdW50ZXIu
PGJyPjwvZGl2PklmIHlvdSB3YW50IHRvIGhhdmUgYSB1bmlxdWUgdGltZXN0YW1wLCB5b3UgaGF2
ZSB0byBzZXQgdGhlIHBhcmFsbGVsaXNtIG9mIHRoZSByZWR1Y2Ugb3BlcmF0b3IgdG8gMSwgYnV0
IHRoZW4gdGhlIHByb2dyYW0gbWlnaHQgYmVjb21lIGluZWZmaWNpZW50Ljxicj48YnI+PC9kaXY+
TWF5YmUgRGF0YVNldFV0aWxzLnppcFdpdGhJbmRleCgpIG9yIERhdGFTZXRVdGlscy56aXBXaXRo
VW5pcXVlSWQoKSBhcmUgaGVscGZ1bCBmb3IgeW91IHVzZSBjYXNlLjxicj48YnI+PC9kaXY+QmVz
dCwgRmFiaWFuPGJyPjxkaXY+PGRpdj48ZGl2PjxkaXY+PGRpdj48ZGl2Pjxicj48L2Rpdj48L2Rp
dj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+
PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj4yMDE2LTA0LTI2IDE3OjEyIEdNVCswMjowMCBC
aXBsb2IgQmlzd2FzIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOnJldm9sdXRp
b25pc21lQGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPnJldm9sdXRpb25pc21lQGdtYWlsLmNv
bTwvYT4mZ3Q7PC9zcGFuPjo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHls
ZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1s
ZWZ0OjFleCI+SGksPGJyPg0KPGJyPg0KSSBhbSB1c2luZyBhIGdyb3VwcmVkdWNlIGZ1bmN0aW9u
IHRvIGFnZ3JlZ2F0ZSB0aGUgY29udGVudCBvZiB0aGUgb2JqZWN0czxicj4NCmJ1dCBhdCB0aGUg
c2FtZSB0aW1lIGkgbmVlZCB0byByZXR1cm4gYSB1bmlxdWUgY291bnRlciBmcm9tIHRoZSBmdW5j
dGlvbiBidXQ8YnI+DQpteSBhdHRlbXB0cyBhcmUgZmFpbGluZyBhbmQgdGhlIGlkZW50aWZpZXJz
IGFyZSBzb21laG93IHZlcnkgcmFuZG9tIGFuZDxicj4NCmdldHRpbmcgZHVwbGljYXRlZC48YnI+
DQo8YnI+DQpGb2xsb3dpbmcgaXMgdGhlIHBhcnQgb2YgbXkgY29kZSB3aGljaCBpcyBzdXBwb3Nl
ZCB0byBnZW5lcmF0ZSBhIHVuaXF1ZTxicj4NCmNvdW50ZXIgYW5kIHJldHVybiBpdCB3aXRoIG91
dC5jb2xsZWN0Ljxicj4NCjxicj4NCjxicj4NCsKgIMKgIMKgIMKgIHB1YmxpYyBzdGF0aWMgY2xh
c3Mgc3VtUmVkdWNlciBpbXBsZW1lbnRzPGJyPg0KwqAgwqAgwqAgwqAgR3JvdXBSZWR1Y2VGdW5j
dGlvbiZsdDtUdXBsZTImYW1wO2x0O0ludGVnZXIsIFBvaW50Jmd0OywgVHVwbGU1Jmx0O0ludGVn
ZXIsUG9pbnQsIFBvaW50LDxicj4NCkxvbmcsIExvbmcmZ3Q7Jmd0OyB7PGJyPg0KPGJyPg0KwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgZG91YmxlIHN1bVtdID0gbnVsbDs8YnI+DQrCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCBkb3VibGUgc3VtT2ZTcXVhcmVbXSA9IG51bGw7PGJyPg0KwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgbG9uZyB0aW1lc3RhbXAgPSAwOzxicj4NCsKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIEBPdmVycmlkZTxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIHB1YmxpYyB2b2lk
IHJlZHVjZShJdGVyYWJsZSZsdDtUdXBsZTImYW1wO2x0O0ludGVnZXIsIFBvaW50Jmd0OyZndDsg
aW4sPGJyPg0KQ29sbGVjdG9yJmx0O1R1cGxlNSZhbXA7bHQ7SW50ZWdlcixQb2ludCwgUG9pbnQs
IExvbmcsIExvbmcmZ3Q7Jmd0OyBvdXQpPGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgdGhyb3dzIEV4Y2VwdGlvbiB7PGJyPg0KPGJyPg0KwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgaW50IGlkID0gMDs8YnI+DQrCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCBsb25nIGNvdW50ID0gMDs8YnI+DQrCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCBib29sZWFuIGZsYWcgPSB0cnVlOzxicj4NCsKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIGZvcihUdXBsZTImbHQ7SW50ZWdlciwgUG9p
bnQmZ3Q7IGk6aW4pPGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgezxi
cj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIGlmKGZs
YWcpPGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
ezxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIHRpbWVzdGFtcCsrOzxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIFN5c3RlbS5vdXQucHJpbnRsbigmcXVvdDt1
bmlxdWVpZDogJnF1b3Q7ICsgaS5mMCArICZxdW90OywgdDogJnF1b3Q7ICsgdGltZXN0YW1wICk7
PGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgc3VtID0gbmV3IGRvdWJsZVtpLmYxLnB0Lmxlbmd0aF07PGJyPg0KwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgc3VtT2ZT
cXVhcmUgPSBuZXcgZG91YmxlW3N1bS5sZW5ndGhdOzxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIGlkID0gaS5mMDs8YnI+DQrC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCBmb3IoaW50IGo9MDtqJmx0O3N1bS5sZW5ndGg7aisrKTxicj4NCsKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIHs8YnI+DQrCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCBzdW1bal0gPSA8YSBocmVmPSJodHRwOi8vaS5mMS5wdCIgcmVsPSJub3JlZmVycmVy
IiB0YXJnZXQ9Il9ibGFuayI+aS5mMS5wdDwvYT5bal07PGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgc3Vt
T2ZTcXVhcmVbal0gPSA8YSBocmVmPSJodHRwOi8vaS5mMS5wdCIgcmVsPSJub3JlZmVycmVyIiB0
YXJnZXQ9Il9ibGFuayI+aS5mMS5wdDwvYT5bal0gKiA8YSBocmVmPSJodHRwOi8vaS5mMS5wdCIg
cmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aS5mMS5wdDwvYT5bal07PGJyPg0KwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
fTxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIGZsYWcgPSBmYWxzZTs8YnI+DQrCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCB9PGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgZWxzZTxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIHs8YnI+DQrCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCBpbnQgbGVuID0gaS5mMS5wdC5sZW5ndGg7PGJy
Pg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgZm9yKGludCBqPTA7aiZhbXA7bHQ7bGVuO2orKyk8YnI+DQrCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCB7PGJyPg0KwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgc3VtW2pdICs9IDxhIGhyZWY9Imh0dHA6Ly9pLmYxLnB0IiByZWw9Im5vcmVmZXJy
ZXIiIHRhcmdldD0iX2JsYW5rIj5pLmYxLnB0PC9hPltqXTs8YnI+DQrCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCBz
dW1PZlNxdWFyZVtqXSArPSAoPGEgaHJlZj0iaHR0cDovL2kuZjEucHQiIHJlbD0ibm9yZWZlcnJl
ciIgdGFyZ2V0PSJfYmxhbmsiPmkuZjEucHQ8L2E+W2pdICogPGEgaHJlZj0iaHR0cDovL2kuZjEu
cHQiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmkuZjEucHQ8L2E+W2pdKTs8YnI+
DQrCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCB9PGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgfTxicj4NCsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IGNvdW50Kys7PGJyPg0KwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgfTxicj4N
CsKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIG91dC5jb2xsZWN0KG5ldyBUdXBs
ZTUmYW1wO2x0O0ludGVnZXIsUG9pbnQsIFBvaW50LCBMb25nLCBMb25nJmd0OyhpZCxuZXc8YnI+
DQpQb2ludChzdW0pLCBuZXcgUG9pbnQoc3VtT2ZTcXVhcmUpLGNvdW50LCB0aW1lc3RhbXApKTs8
YnI+DQrCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCB9PGJyPg0KPGJyPg0KSSB3YW50IHRoZSB0aW1l
c3RhbXAgdG8gYmUgdW5pcXVlLCBidXQgZXZlbiB0aG91Z2ggdGhlIGNvZGU8YnI+DQomcXVvdDtT
eXN0ZW0ub3V0LnByaW50bG4oJnF1b3Q7dW5pcXVlaWQ6ICZxdW90OyArIGkuZjAgKyAmcXVvdDss
IHQ6ICZxdW90OyArIHRpbWVzdGFtcCApOyZxdW90OyBleGVjdXRlczxicj4NCm9uY2UgZm9yIGVh
Y2ggb2YgdGhlIGlkZW50aWZpZXIgKGdpdmVuIGJ5IGkuZjApIGJ5IHdoaWNoIGl0IGlzIGdyb3Vw
ZWQgYW5kPGJyPg0KdGhlbiB0aGUgZ3JvdXBSZWR1Y2NlIGZ1bmN0aW9uIGlzIGNhbGxlZCBzdGls
bCBJIGdldCB0aGUgZm9sbG93aW5nIG91dHB1dDxicj4NCmZvciB0aGUgYWJvdmUgcHJpbnRsbiBz
dGF0ZW1lbnQuPGJyPg0KPGJyPg0KdW5pcXVlaWQ6IDIsIHQ6IDE8YnI+DQp1bmlxdWVpZDogMSwg
dDogMTxicj4NCnVuaXF1ZWlkOiA3LCB0OiAyPGJyPg0KdW5pcXVlaWQ6IDksIHQ6IDM8YnI+DQp1
bmlxdWVpZDogNiwgdDogMjxicj4NCnVuaXF1ZWlkOiAzLCB0OiAxPGJyPg0KdW5pcXVlaWQ6IDUs
IHQ6IDI8YnI+DQp1bmlxdWVpZDogOCwgdDogMzxicj4NCjxicj4NCkkgZG9udCByZWFsbHkgZ2V0
IHdoeSBJIGFtIGdldHRpbmcgdGhpcyBkaXNjcmVwYW5jeSwgcHJvYmFibHkgSSBhbSBtaXNzaW5n
PGJyPg0Kc29tZSBGbGluayBjb25jZXB0LCBJIGFtIHJlbGF0aXZlbHkgdmVyeSBuZXcgdG8gdGhl
IGZsaW5rIHBsYXRmb3JtIGFuZCBhbnk8YnI+DQpoZWxwIGlzIGFwcHJlY2lhdGVkLiBUaGFua3Mg
YSBsb3QuPGJyPg0KPGJyPg0KVGhhbmtzIGFuZCBSZWdhcmRzPGJyPg0KPGJyPg0KPGJyPg0KPGJy
Pg0KLS08YnI+DQpWaWV3IHRoaXMgbWVzc2FnZSBpbiBjb250ZXh0OiA8YSBocmVmPSJodHRwOi8v
YXBhY2hlLWZsaW5rLXVzZXItbWFpbGluZy1saXN0LWFyY2hpdmUuMjMzNjA1MC5uNC5uYWJibGUu
Y29tL1JldHVybi11bmlxdWUtY291bnRlci11c2luZy1ncm91cFJlZHVjZUZ1bmN0aW9uLXRwNjQ1
Mi5odG1sIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vYXBhY2hlLWZs
aW5rLXVzZXItbWFpbGluZy1saXN0LWFyY2hpdmUuMjMzNjA1MC5uNC5uYWJibGUuY29tL1JldHVy
bi11bmlxdWUtY291bnRlci11c2luZy1ncm91cFJlZHVjZUZ1bmN0aW9uLXRwNjQ1Mi5odG1sPC9h
Pjxicj4NClNlbnQgZnJvbSB0aGUgQXBhY2hlIEZsaW5rIFVzZXIgTWFpbGluZyBMaXN0IGFyY2hp
dmUuIG1haWxpbmcgbGlzdCBhcmNoaXZlIGF0IE5hYmJsZS5jb20uPGJyPg0KPC9ibG9ja3F1b3Rl
PjwvZGl2Pjxicj48L2Rpdj4NCg==
--001a1148fdf0049f10053165b055--