flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: keyBy on a collection of Pojos
Date Mon, 23 May 2016 17:30:31 GMT
Actually, the program works correctly (according to the DataStream API)
Let me explain what happens:

1) You do not initialize the count variable, so it will be 0 (summing 0s
results in 0)
2) DataStreams are considered to be unbound (have an infinite size). KeyBy
does not group the records because it would have to wait forever to close
the group due to the infinite input. Instead keyBy basically partitions the
data.
3) By calling sum() on a KeyedStream you compute a running aggregate which
emits one record for each incoming record summing the declared field (this
stays 0 because 0 + 0 = 0).

You will need to
1) initialize count to 1
2) define window to discretize the stream into finite sets (windows) of
records.

Cheers, Fabian

2016-05-23 17:16 GMT+02:00 Deepak Sharma <deepakmca05@gmail.com>:

> Can you try serializing your POJO ?
>
> Thanks
> Deepak
>
> On Mon, May 23, 2016 at 8:42 PM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
>> Sorry Rami, you're right :)
>> Unfortunattely I've never used Flink streaming so I cannot be helpful
>> there..
>> Myabe is it something related to the default triggering policy of the
>> streaming environment?
>>
>>
>> On Mon, May 23, 2016 at 5:06 PM, Al-Isawi Rami <Rami.Al-Isawi@comptel.com
>> > wrote:
>>
>>> Thanks, setters and getters for public fields have no purpose. Also per
>>> the conditions you have mentioned:
>>> "All fields *either* have to be public *or* there must be getters and
>>> setters for all non-public fields.”
>>> Since my fields are declared public there are no impact on adding
>>> getters and setters. ( I have also testing after adding the setters and
>>> getters and as expected that has no effect).
>>>
>>> Could you spot anything else? this should be really easy basic case. I
>>> am really wondering why it is not working.
>>>
>>> For the people who are lazy to open the gist code snippet, this is what
>>> I am trying to do:
>>>
>>> pojoExampleDataStream.
>>> keyBy("productId").
>>> sum("count").
>>> print();
>>>
>>>
>>>
>>> Regards,
>>> -Rami
>>>
>>>
>>> On 23 May 2016, at 17:11, Flavio Pompermaier <pompermaier@okkam.it>
>>> wrote:
>>>
>>> You don't have getters and setters for count and productId.
>>>
>>> Your class should be
>>>
>>> public class PojoExample {
>>> public int count;
>>> public String productId;
>>>
>>> public PojoExample() {}
>>>
>>> public int getCount() {
>>> return count;
>>> }
>>>
>>> public void setCount(int count) {
>>> this.count = count;
>>> }
>>>
>>> public String getProductId() {
>>> return productId;
>>> }
>>>
>>> public void setProductId(String productId) {
>>> this.productId = productId;
>>> }
>>> }
>>>
>>>
>>>
>>> On Mon, May 23, 2016 at 3:40 PM, Al-Isawi Rami <
>>> Rami.Al-Isawi@comptel.com> wrote:
>>>
>>>> Thanks Flavio, but as you can see in my code I have already declared my
>>>> pojo to achieve those conditions:
>>>> public class PojoExample {
>>>> public int count;
>>>> public String productId;
>>>> public PojoExample() {
>>>> }
>>>> }
>>>>
>>>> So it cannot be that.
>>>>
>>>> -Rami
>>>>
>>>> On 23 May 2016, at 16:30, Flavio Pompermaier <pompermaier@okkam.it>
>>>> wrote:
>>>>
>>>> *Conditions* for a class to be treated as a POJO by Flink:
>>>>
>>>>    - The class must be public
>>>>    - It must have a public constructor without arguments
>>>>    - All fields either have to be public or there must be getters and
>>>>    setters for all non-public fields. If the field name is foo the
>>>>    getter and setters must be called getFoo() and setFoo().
>>>>
>>>> I don't know whether you need to implement also hashCode() and equals()
>>>> actually
>>>> Best,
>>>> Flavio
>>>>
>>>> On Mon, May 23, 2016 at 3:24 PM, Al-Isawi Rami <
>>>> Rami.Al-Isawi@comptel.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I was trying to test some specific issue, but now I cannot seem to get
>>>>> the very basic case working. It is most likely that I am blind to
>>>>> something, would anyone have quick look at it?
>>>>> https://gist.github.com/rami-alisawi/d6ff33ae2d4d6e7bb1f8b329e3e5fa77
>>>>>
>>>>> It is just a collection of pojos where I am just trying to keyBy one
>>>>> field and sum into the other, but I am getting:
>>>>> 5> PojoExample{count=0, productId='productA'}
>>>>> 8> PojoExample{count=0, productId='productB'}
>>>>> 5> PojoExample{count=0, productId='productA'}
>>>>> 8> PojoExample{count=0, productId='productB'}
>>>>> 5> PojoExample{count=0, productId='productA'}
>>>>> 5> PojoExample{count=0, productId='productA'}
>>>>> 5> PojoExample{count=0, productId='productA’}
>>>>>
>>>>> Regards,
>>>>> -Rami
>>>>>
>>>>> Disclaimer: This message and any attachments thereto are intended
>>>>> solely for the addressed recipient(s) and may contain confidential
>>>>> information. If you are not the intended recipient, please notify the
>>>>> sender by reply e-mail and delete the e-mail (including any attachments
>>>>> thereto) without producing, distributing or retaining any copies thereof.
>>>>> Any review, dissemination or other use of, or taking of any action in
>>>>> reliance upon, this information by persons or entities other than the
>>>>> intended recipient(s) is prohibited. Thank you.
>>>>>
>>>>
>>>>
>>>> Disclaimer: This message and any attachments thereto are intended
>>>> solely for the addressed recipient(s) and may contain confidential
>>>> information. If you are not the intended recipient, please notify the
>>>> sender by reply e-mail and delete the e-mail (including any attachments
>>>> thereto) without producing, distributing or retaining any copies thereof.
>>>> Any review, dissemination or other use of, or taking of any action in
>>>> reliance upon, this information by persons or entities other than the
>>>> intended recipient(s) is prohibited. Thank you.
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Disclaimer: This message and any attachments thereto are intended solely
>>> for the addressed recipient(s) and may contain confidential information. If
>>> you are not the intended recipient, please notify the sender by reply
>>> e-mail and delete the e-mail (including any attachments thereto) without
>>> producing, distributing or retaining any copies thereof. Any review,
>>> dissemination or other use of, or taking of any action in reliance upon,
>>> this information by persons or entities other than the intended
>>> recipient(s) is prohibited. Thank you.
>>>
>>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>

Mime
View raw message