flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea Sella <andrea.se...@radicalbit.io>
Subject Re: Unexpected behavior with Scala App trait.
Date Tue, 19 Jan 2016 09:33:25 GMT
Hi Chiwan,

I’m not expert of Scala but It seems about closure cleaning problem. Scala
> App trait extends DelayedInit trait to initialize object. But Flink
> serialization stack doesn’t handle this special initialization. (It is just
> my opinion, not verified.)
>

I arrived at the same conclusion with the DelayedInit trait.

>
> To run TFIDFNPE safely, you need to just change tokenize and uniqueWords
> to method like following:
>
> ```
> def tokenize = …
> def uniqueWords = ...
> ```
>
> With this change, I tested that TFIDFNPE works safely in Flink 0.10.1
> cluster.
>

Yeah, it works and i knew it. My aim it was to use tricky (functional)
Scala mechanisms to test how much the Scala APIs are robust and
idomitic-friendly for Scala users.

>
> About TFIDF object, you should avoid overriding main method if the object
> is derived by App trait. It is also related DelaytedInit mechanism [1].
>

Yeah, App trait is into a comment 'cause I am overriding the main method.
TFIDF is equal to TFIDFNPE with App trait and not overriding main method.


Thanks,
Andrea

>
> [1]:
> https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31
>
> > On Jan 19, 2016, at 12:08 AM, Andrea Sella <andrea.sella@radicalbit.io>
> wrote:
> >
> > Hi,
> >
> > I was implementing TF-IDF example of flink-training when I faced a
> problem with NPE during the deploy of my Job.
> >
> > Source code: https://github.com/alkagin/flink-tfidf-example
> >
> > I used 0.10.1 version and started in local mode.
> > During the deploy of TFIDFNPE Job, which it extends App, Flink throws
> NullPointerException on both flatMap functions.
> > If I include the tokenize function into the closures of flatMap
> functions, the Job works fine; see example TFIDFApp.
> > To avoid this unexpected behavior I don't have use Scala App trait, see
> TFIDF, but why?
> >
> >
> > Thanks,
> > Andrea
> >
>
>
> Regards,
> Chiwan Park
>
>

Mime
View raw message