flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chiwan Park <chiwanp...@apache.org>
Subject Re: Unexpected behavior with Scala App trait.
Date Tue, 19 Jan 2016 02:50:44 GMT
Hi Andrea,

I’m not expert of Scala but It seems about closure cleaning problem. Scala App trait extends
DelayedInit trait to initialize object. But Flink serialization stack doesn’t handle this
special initialization. (It is just my opinion, not verified.)

To run TFIDFNPE safely, you need to just change tokenize and uniqueWords to method like following:

def tokenize = …
def uniqueWords = ...

With this change, I tested that TFIDFNPE works safely in Flink 0.10.1 cluster.

About TFIDF object, you should avoid overriding main method if the object is derived by App
trait. It is also related DelaytedInit mechanism [1].

[1]: https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31

> On Jan 19, 2016, at 12:08 AM, Andrea Sella <andrea.sella@radicalbit.io> wrote:
> Hi,
> I was implementing TF-IDF example of flink-training when I faced a problem with NPE during
the deploy of my Job.
> Source code: https://github.com/alkagin/flink-tfidf-example
> I used 0.10.1 version and started in local mode.
> During the deploy of TFIDFNPE Job, which it extends App, Flink throws NullPointerException
on both flatMap functions. 
> If I include the tokenize function into the closures of flatMap functions, the Job works
fine; see example TFIDFApp.
> To avoid this unexpected behavior I don't have use Scala App trait, see TFIDF, but why?
> Thanks,
> Andrea

Chiwan Park

View raw message