Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A1C5185EC for ; Wed, 16 Sep 2015 14:05:30 +0000 (UTC) Received: (qmail 7666 invoked by uid 500); 16 Sep 2015 14:05:30 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 7590 invoked by uid 500); 16 Sep 2015 14:05:30 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 7568 invoked by uid 99); 16 Sep 2015 14:05:30 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2015 14:05:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id BB9D718098E for ; Wed, 16 Sep 2015 14:05:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GRRgh1M5-1Ub for ; Wed, 16 Sep 2015 14:05:23 +0000 (UTC) Received: from mail-qg0-f48.google.com (mail-qg0-f48.google.com [209.85.192.48]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 1AA7C20FF3 for ; Wed, 16 Sep 2015 14:05:23 +0000 (UTC) Received: by qgez77 with SMTP id z77so172588914qge.1 for ; Wed, 16 Sep 2015 07:05:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=GythIOcQxnOyA4NHr9wZCjhGIdumFX8A82kBm+dvHXU=; b=oP3hNjHsaX9exLIr3Y5aXJr2dSmn8nddwFRkgBrvYqYi1klKefVfWq+QVx/Kx5m1hn +pjSIC0Rp61JVwjD3a64tbgmPHONx9cEkcNvQaLZJKS4luPNOsQA1a6Bh6l6rivN5Ok9 CzaKJpQX2BmCotcU93EmFqKq+oOHq/jiRGNVmSJmGYMN2OwcqClmSEZydQA7u+N3kwu8 q/kdEBcF8Cb0mgSAZ42XeZgVGPeqHu9w/taZJ8dHwOpEQ1Vej/BUecrpOL04yPgDIOoN vM3bkw+M63NSHGEA8pyXwwtRnqQq+dPdJbhTG++a4oPTmN8sFFoek8Z/7028QrAwgpI/ Y87g== MIME-Version: 1.0 X-Received: by 10.140.237.212 with SMTP id i203mr43786202qhc.97.1442412322137; Wed, 16 Sep 2015 07:05:22 -0700 (PDT) Sender: ewenstephan@gmail.com Received: by 10.55.37.5 with HTTP; Wed, 16 Sep 2015 07:05:22 -0700 (PDT) In-Reply-To: References: <55F952AD.6000807@mailbox.org> Date: Wed, 16 Sep 2015 16:05:22 +0200 X-Google-Sender-Auth: P0JhhpvQ3vteOj8chAomi2FFBrU Message-ID: Subject: Re: Inheritance and FlatMap with custom POJO From: Stephan Ewen To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a11359b3214af8a051fddcafa --001a11359b3214af8a051fddcafa Content-Type: text/plain; charset=UTF-8 Could you also try the other variant (disabeling the closure cleaner)? I would be curious if this behavior is expected Java Serialization behavior, or whether our pre-processing code is causing it. Greetings, Stephan On Wed, Sep 16, 2015 at 3:38 PM, Giacomo Licari wrote: > Thank you Martin and Stephan for your help. > I tried directly to implement java.io.Serializable in Base class and it > worked perfectly! > > Now I can develop more flexible and maintainable code. Thank you a lot > guys. > > Greetings, > Giacomo > > On Wed, Sep 16, 2015 at 1:46 PM, Stephan Ewen wrote: > >> Hi! >> >> Interesting case. We use plain Java Serialization to distribute UDFs, and >> perform additional "cleaning" of scopes, which may be causing the issue. >> >> Can you try the following to see if any of those resolves the problem? >> >> 1) On the environment, disable the closure cleaner (in the execution >> config). >> >> 2) Let the CullTimeBase class implement java.io.Serializable. >> >> Please let us know how it turns out! >> >> Greetings, >> Stephan >> >> >> On Wed, Sep 16, 2015 at 1:29 PM, Martin Junghanns < >> m.junghanns@mailbox.org> wrote: >> >>> Hi Giacomo, >>> >>> I ran into the same issue. Seems to be coupled to the serialization >>> mechanism of UDFs. I solved it by letting the base class implement the UDF >>> interface (e.g. FlatMapFunction) and in addition make it generic (which >>> should be necessary in your example). >>> >>> public [abstract] class CullTimeBase implements >>> FlatMapFunction { >>> // ... >>> } >>> >>> public class CullTimeRainFall extends CullTimeBase>> RainFallPOJO> { >>> // ... >>> } >>> >>> This should work. >>> >>> Best, >>> Martin >>> >>> >>> On 16.09.2015 10:41, Giacomo Licari wrote: >>> >>> Hi guys, >>> I'm trying to create a base class which is inherited from classes >>> implementing FlatMap method on specific POJO types. >>> >>> It seems inheritance doesn't work, I can access this.PropertyName or >>> super.PropertyName from flatMap method but values are always null. >>> >>> Here the derived class, using RainfallPOJO: >>> >>> public class CullTimeRainfall extends CullTimeBase implements >>> FlatMapFunction { >>> >>> public CullTimeRainfall(int num, int den, String time_data_name, String >>> start_time, String end_time, int interval, String time_unit){ >>> super(num, den, time_data_name, start_time, end_time, interval, >>> time_unit); >>> } >>> >>> public void flatMap(RainfallPOJO obj, Collector coll) >>> throws Exception { >>> DateFormat formatter = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ss.SSS"); >>> try { >>> Date time = formatter.parse( obj.getTime().replaceAll( >>> "([0-9\\-T]+:[0-9]{2}:[0-9.+]+):([0-9]{2})", "$1$2" ) ); >>> if(time.after(this.startTime) && time.before(this.endTime)){ >>> coll.collect(obj); >>> } >>> } catch(Exception e){ >>> e.printStackTrace(); >>> } >>> } >>> } >>> >>> My Base class is: >>> >>> public class CullTimeBase { >>> >>> protected int numerator; >>> protected int denominator; >>> protected String timeDataName; >>> protected Date startTime; >>> protected Date endTime; >>> protected int interval; >>> protected String timeUnit; >>> public CullTimeBase(int num, int den, String time_data_name, String >>> start_time, String end_time, int interv, String time_unit){ >>> numerator = num; >>> denominator = den; >>> timeDataName = time_data_name; >>> interval = interv; >>> timeUnit = time_unit; >>> DateFormat formatter = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS"); >>> try { >>> startTime = formatter.parse(start_time); >>> endTime = formatter.parse(end_time); >>> } catch (ParseException e) { >>> e.printStackTrace(); >>> } >>> } >>> >>> It works only if I declare all variables and methods in only one class, >>> but so I should repeat the same properties in more classes. I would only >>> specialize each derived class with a custom flatMap method. which uses a >>> custom POJO type. >>> >>> Thanks a lot, >>> Giacomo >>> >>> >>> >> > --001a11359b3214af8a051fddcafa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Could you also try the other variant (disabeling the closu= re cleaner)? I would be curious if this behavior is expected Java Serializa= tion behavior, or whether our pre-processing code is causing it.

Greetings,
Stephan


On Wed, Sep 16, 2015 at 3:3= 8 PM, Giacomo Licari <giacomo.licari@gmail.com> wrote= :
Thank you Martin and S= tephan for your help.
I tried directly to implement java.io.Serializabl= e in Base class and it worked perfectly!

Now I can= develop more flexible and maintainable code. Thank you a lot guys.

Greetings,
Giacomo

On Wed, Sep 16, 2015 at 1:46 PM, Stephan Ewen <<= a href=3D"mailto:sewen@apache.org" target=3D"_blank">sewen@apache.org&g= t; wrote:
Hi!
Interesting case. We use plain Java Serialization to distri= bute UDFs, and perform additional "cleaning" of scopes, which may= be causing the issue.

Can you try the following t= o see if any of those resolves the problem?

1) On = the environment, disable the closure cleaner (in the execution config).

2) Let the=C2=A0Cull= TimeBase=C2=A0class implement java.= io.Serializable.

Please let us know how it tu= rns out!

Greetings,
Stephan


On Wed, Sep 16, 2015 at 1:29 PM, Martin Junghanns <= span dir=3D"ltr"><m.junghanns@mailbox.org> wrote:
=20 =20 =20
Hi Giacomo,

I ran into the same issue. Seems to be coupled to the serialization mechanism of UDFs. I solved it by letting the base class implement the UDF interface (e.g. FlatMapFunction) and in addition make it generic (which should be necessary in your example).

public [abstract] class CullTimeBase<IN, OUT> implements FlatMapFunction<IN, OUT> {
// ...
}

public class CullTimeRainFall extends CullTimeBase<RainFallPOJO, RainFallPOJO> {
// ...
}

This should work.

Best,
Martin


On 16.09.2015 10:41, Giacomo Licari wrote:
Hi guys,
I'm trying to create a base class which is inherited from classes implementing FlatMap method on specific POJO types.

It seems inheritance doesn't work, I can access this.PropertyName or super.PropertyName from flatMap method but values are always null.=C2=A0

Here the derived class, using RainfallPOJO:

public class CullTimeRainfall extends CullTimeBase implements FlatMapFunction<RainfallPOJO, RainfallPOJO> {

public CullTimeRainfall(int num, int den, String time_data_name, String start_time, String end_time, int interval, String time_unit){
super(num, den, time_data_name, start_time, end_time, interval, time_unit);
}

public void flatMap(RainfallPOJO obj, Collector<RainfallPOJO> coll) throws Exception { <= /span>
DateFormat formatter =3D new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ss.SSS")= ;
try {
=C2=A0 =C2=A0= Date time =3D formatter.parse( obj.getTime().replaceAll( "([0-9\\-T]+:[0-9]{2}:[0-9.+]+):([0-9]{2})", "$1= $2" ) );
=C2=A0 =C2=A0 = =C2=A0 =C2=A0if(time.after(this.startTime) && time.before(this.endTime)){
coll.collect= (obj);
} =C2=A0 =C2=A0
} catch(Exception e){
e.printStackT= race();
}
}
}

My Base class is:

public class CullTimeBase {

=C2=A0 protect= ed int numerator;
=C2=A0 =C2=A0 protected int denominator;
=C2=A0 =C2=A0 protected String timeDataName;
=C2=A0 =C2=A0 protected Date startTime;
=C2=A0 =C2=A0 protected Date endTime;
=C2=A0 =C2=A0 protected int interval;
=C2=A0 =C2=A0 protected String timeUnit;
public CullTimeBase(int num, int den, String time_data_name, String start_time, String end_time, int interv, String time_unit){
numerator =3D num;
denominator =3D den;
timeDataName =3D time_data_name;
interval =3D interv;
timeUnit =3D time_unit;
DateFormat formatter =3D new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS")= ;
try {
startTime =3D formatter.parse(start_time);
endTime =3D formatter.parse(end_time);
} catch (ParseException e) {
e.printStackT= race();
}
}

It works only if I declare all variables and methods in only one class, but so I should repeat the same properties in more classes. I would only specialize each derived class with a custom flatMap method. which uses a custom POJO type.

Thanks a lot,
Giacomo




--001a11359b3214af8a051fddcafa--