flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject Re: Scala API rewrite almost complete
Date Sun, 28 Sep 2014 19:25:03 GMT
Hi Robert,  I didn't have problem with IntelliJ idea last week, will try it
again with latest master.

User can deactivate unwanted maven profile via idea's maven project option
(not remember exact name of that feature)

On Sunday, September 28, 2014, Robert Metzger <rmetzger@apache.org> wrote:

> I've worked on a quite outdated version of Flink for a while now and
> rebased my code to the latest master on Friday.
> Back at home, I wanted to continue my work and found that it is very
> difficult to properly set up the latest eclipse for Flink.
>
> What I've done so far:
> - Downloaded Eclipse Luna SR1 (for linux)
> - Installed Scala IDE 4.0 nightly
> - add the -clean flag to the eclipse.ini, because eclipse is not starting
> anymore due to a bug
> - installed some missing m2e connectors
> - Set the Scala compiler to 2.10 (thats a new feature in Scala IDE) (See:
> http://scala-ide.org/blog/scala-installations.html)
>
> Now, all the tests in flink-scala/src/test are mostly about this:
> could not find implicit value for evidence parameter of type
> org.apache.flink.api.common.typeinfo.TypeInformation[(Array[String],)];
> /flink-scala/src/test/scala/org/apache/flink/api/scala/runtime
>
> It seems that the flink-tests project is calling Scala examples for testing
> purposes. So if the flink-scala project is not building, its blocking
> flink-tests from building.
>
> Any ideas?
>
> If you are wondering why I'm not switching to IntelliJ: It is currently not
> possible to deactivate a maven build profile that is activated by default,
> so I'm not able to properly work on the "flink-yarn" project with IntelliJ
> (the respective issue has been closed at IntelliJ, so I guess the next
> release will contain the fix).
> In addition, I think its good if at least a few core developers make sure
> that Flink is also "developable" using Eclipse.
>
>
>
>
> On Sun, Sep 14, 2014 at 12:48 PM, Márton Balassi <balassi.marton@gmail.com
> <javascript:;>>
> wrote:
>
> > Answer posted to "Example packages naming convention" thread as the issue
> > diverged from this topic.
> >
> > On Sun, Sep 14, 2014 at 11:14 AM, Kostas Tzoumas <ktzoumas@apache.org>
> > wrote:
> >
> > > Good catch, I suggest to use examples
> > >
> > > On Sat, Sep 13, 2014 at 3:27 PM, Márton Balassi <
> > balassi.marton@gmail.com>
> > > wrote:
> > >
> > > > Pull request issued. One minor naming concern:
> > > >
> > > > As of today the scala examples are located at
> > > > the org.apache.flink.examples.scala package, while the java ones in
> > > > the org.apache.flink.example.java. I suggest using only one
> convention
> > > for
> > > > this either example or examples.
> > > >
> > > > Cheers,
> > > >
> > > > Marton
> > > >
> > > > On Fri, Sep 12, 2014 at 9:17 PM, Márton Balassi <
> > > balassi.marton@gmail.com>
> > > > wrote:
> > > >
> > > > > Sorry for being a bit silent after already bidding on LR. The pull
> > > > request
> > > > > is coming soon.
> > > > >
> > > > > On Fri, Sep 12, 2014 at 6:25 PM, Stephan Ewen <sewen@apache.org>
> > > wrote:
> > > > >
> > > > >> I suppose that having the option between simple return type, and a
> > > > >> collector is the easiest to understand.
> > > > >>  Am 12.09.2014 16:50 schrieb "Aljoscha Krettek" <
> > aljoscha@apache.org
> > > >:
> > > > >>
> > > > >> > So, should I change join and coGroup to have a simple return
> > value,
> > > no
> > > > >> > Option or Collection? Also what's happening with the relational
> > > > >> > examples and the LinearRegression examples? I'd like to make a
> > pull
> > > > >> > request before this weekend.
> > > > >> >
> > > > >> > I also added a test that checks whether the Scala API has the
> same
> > > > >> > methods as the Java API (ScalaAPICompletenessTest).
> > > > >> >
> > > > >> > On Fri, Sep 12, 2014 at 4:00 PM, Aljoscha Krettek <
> > > > aljoscha@apache.org>
> > > > >> > wrote:
> > > > >> > > Yes, there is already a Collector version, you can do:
> > > > >> > >
> > > > >> > > left.join(right).where("foo").equalTo("bar") {
> > > > >> > >   (left, right, out: Collector[Page]) =>
> > > > >> > >     if (...) out.collect(...)
> > > > >> > > }
> > > > >> > >
> > > > >> > > I wasn't sure on what our Function2 variant should be. That's
> > why
> > > I
> > > > >> > > asked. There are some cases where you want to have the option
> > (pun
> > > > >> > > intended) of either returning something from a join or not.
> But
> > > you
> > > > >> > > could also use a filter after the join. It's a preference
> > thing, I
> > > > >> > > suppose.
> > > > >> > >
> > > > >> > > On Fri, Sep 12, 2014 at 2:43 PM, Stephan Ewen <
> sewen@apache.org
> > >
> > > > >> wrote:
> > > > >> > >> I think it seems weird that normal joins need to go through
> > > option.
> > > > >> > >>
> > > > >> > >> The option variant is to allow filters in the join function.
> > > > >> Wouldn't a
> > > > >> > >> collector variant allow you to do the same, and would be
> > > function3
> > > > ?
> > > > >> I
> > > > >> > know
> > > > >> > >> that option reads more functionally...
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Am 12.09.2014 14:24 schrieb "Aljoscha Krettek" <
> > > > aljoscha@apache.org
> > > > >> >:
> > > > >> > >>>
> > > > >> > >>> As already mentioned this is not possible because of type
> > > erasure.
> > > > >> We
> > > > >> > >>> can only have one join variant that takes a Function2.
> > > > >> > >>>
> > > > >> > >>> On Fri, Sep 12, 2014 at 12:34 PM, Stephan Ewen <
> > > sewen@apache.org>
> > > > >> > wrote:
> > > > >> > >>> > It would be nice to have a join variant that directly
> > returns
> > > > the
> > > > >> > value
> > > > >> > >>> > rathern than an option. Why not have both (they are
> wrapped
> > as
> > > > >> > flatJoins
> > > > >> > >>> > anyway below, right?)
> > > > >> > >>> >
> > > > >> > >>> > On Fri, Sep 12, 2014 at 11:50 AM, Fabian Hueske <
> > > > >> fhueske@apache.org>
> > > > >> > >> wrote:
> > > > >> > >>> >
> > > > >> > >>> >> Sweet! I'm lovin' this :-)
> > > > >> > >>> >>
> > > > >> > >>> >> 2014-09-12 11:46 GMT+02:00 Aljoscha Krettek <
> > > > aljoscha@apache.org
> > > > >> >:
> > > > >> > >>> >>
> > > > >> > >>> >> > Also, you can use CaseClasses directly as the type for
> > CSV
> > > > >> input.
> > > > >> > So
> > > > >> > >>> >> > instead of reading it as tuples and then having a
> mapper
> > > that
> > > > >> > maps to
> > > > >> > >>> >> > your case classes you can use:
> > > > >> > >>> >> >
> > > > >> > >>> >> > env.readCsv[Edge](...)
> > > > >> > >>> >> >
> > > > >> > >>> >> > On Fri, Sep 12, 2014 at 11:43 AM, Aljoscha Krettek <
> > > > >> > >> aljoscha@apache.org>
> > > > >> > >>> >> > wrote:
> > > > >> > >>> >> > > I added support for specifying keys by name for
> > > > CaseClasses.
> > > > >> > Check
> > > > >> > >> out
> > > > >> > >>> >> > > the PageRank and TriangleEnumeration examples to see
> it
> > > in
> > > > >> > action.
> > > > >> > >>> >> > >
> > > > >> > >>> >> > > @Kostas: I think you could use them for the TPC-H
> > > examples.
> > > > >> > >>> >> > >
> > > > >> > >>> >> > > On Fri, Sep 12, 2014 at 7:23 AM, Aljoscha Krettek <
> > > > >> > >> aljoscha@apache.org
> > > > >> > >>> >> >
> > > > >> > >>> >> > wrote:
> > > > >> > >>> >> > >> Yes, that would allow list comprehensions. It would
> be
> > > > >> > possible to
> > > > >> > >>> >> > >> have the Collection signature for join (and
> coGroup),
> > > > i.e.:
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> apply[R]((T, O) => TraversableOnce[O]): DataSet[O]
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> (T and O are the left and right input type, R is
> > result
> > > > >> type)
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> Then you can return collections and still return an
> > > > option,
> > > > >> as
> > > > >> > in:
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> a.join(b).where(0).equalTo(0) { (l, r) => if (r >
> ...)
> > > > >> Some(l)
> > > > >> > >> else
> > > > >> > >>> >> > None }
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> Because there is an implicit conversion from Options
> > to
> > > a
> > > > >> > >> Collection.
> > > > >> > >>> >> > >> This will always wrap the return value in a List
> with
> > > only
> > > > >> one
> > > > >> > >> value.
> > > > >> > >>> >> > >> I'm not sure we want the overhead here. I'm also not
> > > sure
> > > > >> > whether
> > > > >> > >> we
> > > > >> > >>> >> > >> want the overhead of always having to use an Option
> > even
> > > > >> though
> > > > >> > >> the
> > > > >> > >>> >> > >> join always returns a value.
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> What do you think?
> > > > >> > >>> >> > >>
> > > > >> > >>> >> > >> On Thu, Sep 11, 2014 at 11:22 PM, Fabian Hueske <
> > > > >> > >> fhueske@apache.org>
> > > > >> > >>> >> > wrote:
> > > > >> > >>> >> > >>> Hmmm, tricky question...
> > > > >> > >>> >> > >>> How about the Option for Join as this is a
> tuple-wise
> > > > >> > operation
> > > > >> > >> and
> > > > >> > >>> >> the
> > > > >> > >>> >> > >>> Collection for Cogroup which is group-wise?
> > > > >> > >>> >> > >>> Could we in that case use list comprehensions in
> > > Cogroup
> > > > >> > >> functions?
> > > > >> > >>> >> > >>>
> > > > >> > >>> >> > >>> Or is that too much mixing?
> > > > >> > >>> >> > >>>
> > > > >> > >>> >> > >>> 2014-09-11 23:00 GMT+02:00 Aljoscha Krettek <
> > > > >> > aljoscha@apache.org
> > > > >> > >>>:
> > > > >> > >>> >> > >>>
> > > > >> > >>> >> > >>>> I didn't look at the example either.
> > > > >> > >>> >> > >>>>
> > > > >> > >>> >> > >>>> Addings collections is easy, it's just that we can
> > > > either
> > > > >> > have
> > > > >> > >>> >> > >>>> Collections or the Option, not both.
> > > > >> > >>> >> > >>>>
> > > > >> > >>> >> > >>>> For the coding style I followed this:
> > > > >> > >>> >> > >>>>
> > > > >> > >>> >> >
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
> > > > >> > >>> >> ,
> > > > >> > >>> >> > >>>> which itself is based on this:
> > > > >> > http://docs.scala-lang.org/style/
> > > > >> > >> .
> > > > >> > >>> >> It
> > > > >> > >>> >> > >>>> is different from the Java Code Guidelines we have
> > in
> > > > >> place,
> > > > >> > >> yes.
> > > > >> > >>> >> > >>>>
> > > > >> > >>> >> > >>>> On Thu, Sep 11, 2014 at 10:10 PM, Fabian Hueske <
> > > > >> > >> fhueske@apache.org
> > > > >> > >>> >> >
> > > > >> > >>> >> > >>>> wrote:
> > > > >> > >>> >> > >>>> > I haven't looked at the LineRank example in
> > detail,
> > > > but
> > > > >> if
> > > > >> > you
> > > > >> > >>> >> > think that
> > > > >> > >>> >> > >>>> > it adds something new to the examples
> collection,
> > we
> > > > can
> > > > >> > >> certainly
> > > > >> > >>> >> > port
> > > > >> > >>> >> > >>>> it
> > > > >> > >>> >> > >>>> > also to Java.
> > > > >> > >>> >> > >>>> > I think the Option and Collector return types
> are
> > > > >> > sufficient
> > > > >> > >> right
> > > > >> > >>> >> > now
> > > > >> > >>> >> > >>>> but
> > > > >> > >>> >> > >>>> > if Collections are easy to add, go for it. ;-)
> > > > >> > >>> >> > >>>> >
> > > > >> > >>> >> > >>>> > Great that the Scala primitives are working!
> Also
> > > > thanks
> > > > >> > for
> > > > >> > >>> >> adding
> > > > >> > >>> >> > >>>> > genSequence and adapting my examples.
> > > > >> > >>> >> > >>>> > Btw. does the codestyle not apply for Scala
> files
> > or
> > > > do
> > > > >> we
> > > > >> > >> have a
> > > > >> > >>> >> > >>>> different
> > > > >> > >>> >> > >>>> > there?
> > > > >> > >>> >> > >>>> >
> > > > >> > >>> >> > >>>> > 2014-09-11 17:55 GMT+02:00 Aljoscha Krettek <
> > > > >> > >> aljoscha@apache.org
> > > > >> > >>> >> >:
> > > > >> > >>> >> > >>>> >
> > > > >> > >>> >> > >>>> >> What about the LineRank example? We had that in
> > > Scala
> > > > >> but
> > > > >> > >> never
> > > > >> > >>> >> > had a
> > > > >> > >>> >> > >>>> >> Java Example.
> > > > >> > >>> >> > >>>> >>
> > > > >> > >>> >> > >>>> >> On Thu, Sep 11, 2014 at 5:51 PM, Aljoscha
> > Krettek <
> > > > >> > >>> >> > aljoscha@apache.org>
> > > > >> > >>> >> > >>>> >> wrote:
> > > > >> > >>> >> > >>>> >> > Yes, I like that. For the ITCases I always
> just
> > > > >> copied
> > > > >> > the
> > > > >> > >> Java
> > > > >> > >>> >> > >>>> ITCase.
> > > > >> > >>> >> > >>>> >> >
> > > > >> > >>> >> > >>>> >> > The only examples that are missing now are
> > > > >> > >> LinearRegression and
> > > > >> > >>> >> > the
> > > > >> > >>> >> > >>>> >> > relational stuff.
> > > > >> > >>> >> > >>>> >> >
> > > > >> > >>> >> > >>>> >> > On Thu, Sep 11, 2014 at 5:48 PM, Fabian
> Hueske
> > <
> > > > >> > >>> >> > fhueske@apache.org>
> > > > >> > >>> >> > >>>> >> wrote:
> > > > >> > >>> >> > >>>> >> >> I just removed the old CountEdgeDegrees
> > example.
> > > > >> > >>> >> > >>>> >> >> That was a preprocessing step for the
> > > > >> > >> TriangleEnumeration, and
> > > > >> > >>> >> > is now
> > > > >> > >>> >> > >>>> >> part
> > > > >> > >>> >> > >>>> >> >> of the new TriangleEnumerationOpt example.
> > > > >> > >>> >> > >>>> >> >> So I guess, we don't need to port that one.
> > As I
> > > > >> said
> > > > >> > >> before,
> > > > >> > >>> >> > I'd
> > > > >> > >>> >> > >>>> >> prefer to
> > > > >> > >>> >> > >>>> >> >> keep Java and Scala examples in sync.
> > > > >> > >>> >> > >>>> >> >>
> > > > >> > >>> >> > >>>> >> >> Cheers, Fabian
> > > > >> > >>> >> > >>>> >> >>
> > > > >> > >>> >> > >>>> >> >> 2014-09-11 17:40 GMT+02:00 Aljoscha Krettek
> <
> > > > >> > >>> >> > aljoscha@apache.org>:
> > > > >> > >>> >> > >>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> I added the PageRank example, thanks again
> > > > fabian.
> > > > >> :D
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> Regarding the other stuff:
> > > > >> > >>> >> > >>>> >> >>>  - There is a comment in DataSet.scala
> about
> > > > >> including
> > > > >> > >>> >> > >>>> >> >>> org.apache.flink.api.scala._ because of the
> > > > >> > >> TypeInformation.
> > > > >> > >>> >> > >>>> >> >>>  - I added generateSequence to
> > > > >> ExecutionEnvironment.
> > > > >> > >>> >> > >>>> >> >>>  - It is possible to use Scala Primitives
> in
> > > > >> Array, I
> > > > >> > >> noticed
> > > > >> > >>> >> > it
> > > > >> > >>> >> > >>>> while
> > > > >> > >>> >> > >>>> >> >>> writing the tests, you probably had an
> older
> > > > >> version
> > > > >> > of
> > > > >> > >> the
> > > > >> > >>> >> > code.
> > > > >> > >>> >> > >>>> >> >>>  - Yes, using List and other Interfaces is
> > not
> > > > >> > possible,
> > > > >> > >> this
> > > > >> > >>> >> > is
> > > > >> > >>> >> > >>>> also
> > > > >> > >>> >> > >>>> >> >>> a restriction in the Java API.
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> What do you think about the interface of
> join
> > > and
> > > > >> > >> coGroup?
> > > > >> > >>> >> > Right
> > > > >> > >>> >> > >>>> now,
> > > > >> > >>> >> > >>>> >> >>> you can either use a lambda that returns an
> > > > Option
> > > > >> or
> > > > >> > the
> > > > >> > >>> >> > lambda
> > > > >> > >>> >> > >>>> with
> > > > >> > >>> >> > >>>> >> >>> the Collector. Originally I wanted to have
> > also
> > > > >> have a
> > > > >> > >> lambda
> > > > >> > >>> >> > that
> > > > >> > >>> >> > >>>> >> >>> returns a Collection, but due to type
> erasure
> > > > this
> > > > >> has
> > > > >> > >> the
> > > > >> > >>> >> > same type
> > > > >> > >>> >> > >>>> >> >>> as the lambda with the Option so I couldn't
> > use
> > > > it.
> > > > >> > >> There is
> > > > >> > >>> >> an
> > > > >> > >>> >> > >>>> >> >>> implicit conversion from Option to a
> > > Collection,
> > > > >> so I
> > > > >> > >> could
> > > > >> > >>> >> > change
> > > > >> > >>> >> > >>>> it
> > > > >> > >>> >> > >>>> >> >>> without breaking the examples we have now.
> > What
> > > > do
> > > > >> you
> > > > >> > >> think?
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> So far we have ported: WordCount, KMeans,
> > > > >> > >>> >> ConnectedComponents,
> > > > >> > >>> >> > >>>> >> >>> WebLogAnalysis, TransitiveClosureNaive,
> > > > >> > >>> >> > >>>> TriangleEnumerationNaive/Opt,
> > > > >> > >>> >> > >>>> >> >>> PageRank
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> These are the examples people called dibs
> on:
> > > > >> > >>> >> > >>>> >> >>>  - BatchGradientDescent (Márton) (Should
> be a
> > > > port
> > > > >> of
> > > > >> > >>> >> > >>>> LinearRegression
> > > > >> > >>> >> > >>>> >> >>> Example from Java)
> > > > >> > >>> >> > >>>> >> >>>  - ComputeEdgeDegrees (Hermann)
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> Those are unclaimed (if I'm not mistaken):
> > > > >> > >>> >> > >>>> >> >>>  - The relational Stuff
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> On Thu, Sep 11, 2014 at 3:06 PM, Stephan
> > Ewen <
> > > > >> > >>> >> > sewen@apache.org>
> > > > >> > >>> >> > >>>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> > +1 for removing RelationQuery
> > > > >> > >>> >> > >>>> >> >>> >
> > > > >> > >>> >> > >>>> >> >>> > On Thu, Sep 11, 2014 at 3:04 PM, Aljoscha
> > > > >> Krettek <
> > > > >> > >>> >> > >>>> >> aljoscha@apache.org>
> > > > >> > >>> >> > >>>> >> >>> > wrote:
> > > > >> > >>> >> > >>>> >> >>> >
> > > > >> > >>> >> > >>>> >> >>> >> By the way, what was called
> > > > >> BatchGradientDescent in
> > > > >> > >> the
> > > > >> > >>> >> > Scala
> > > > >> > >>> >> > >>>> >> examples
> > > > >> > >>> >> > >>>> >> >>> >> should be replaced by a port of the
> > > > >> > LinearRegression
> > > > >> > >>> >> > Example from
> > > > >> > >>> >> > >>>> >> >>> >> Java. I had them as two separate
> examples
> > > > >> earlier.
> > > > >> > >>> >> > >>>> >> >>> >>
> > > > >> > >>> >> > >>>> >> >>> >> What about RelationalQuery and TPC-H-Q3.
> > Any
> > > > >> > thoughts
> > > > >> > >>> >> about
> > > > >> > >>> >> > >>>> removing
> > > > >> > >>> >> > >>>> >> >>> >> RelationalQuery?
> > > > >> > >>> >> > >>>> >> >>> >>
> > > > >> > >>> >> > >>>> >> >>> >> On Thu, Sep 11, 2014 at 11:43 AM,
> Aljoscha
> > > > >> Krettek
> > > > >> > <
> > > > >> > >>> >> > >>>> >> aljoscha@apache.org
> > > > >> > >>> >> > >>>> >> >>> >
> > > > >> > >>> >> > >>>> >> >>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> > I added the Triangle Enumeration
> > Examples,
> > > > >> thanks
> > > > >> > >>> >> Fabian.
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> > So far we have ported: WordCount,
> > KMeans,
> > > > >> > >>> >> > ConnectedComponents,
> > > > >> > >>> >> > >>>> >> >>> >> > WebLogAnalysis,
> TransitiveClosureNaive,
> > > > >> > >>> >> > >>>> >> TriangleEnumerationNaive/Opt
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> > These are the examples people called
> > dibs
> > > > on:
> > > > >> > >>> >> > >>>> >> >>> >> >  - PageRank (Fabian)
> > > > >> > >>> >> > >>>> >> >>> >> >  - BatchGradientDescent (Márton)
> > > > >> > >>> >> > >>>> >> >>> >> >  - ComputeEdgeDegrees (Hermann)
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> > Those are unclaimed (if I'm not
> > mistaken):
> > > > >> > >>> >> > >>>> >> >>> >> >  - The relational Stuff
> > > > >> > >>> >> > >>>> >> >>> >> >  - LinearRegression
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> > On Wed, Sep 10, 2014 at 6:04 PM,
> > Aljoscha
> > > > >> > Krettek <
> > > > >> > >>> >> > >>>> >> >>> aljoscha@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >> Thanks, I added it. I'll keep a
> running
> > > > list
> > > > >> of
> > > > >> > >>> >> > >>>> ported/unported
> > > > >> > >>> >> > >>>> >> >>> >> >> examples in my mails. I'll rename the
> > > java
> > > > >> > example
> > > > >> > >>> >> > package to
> > > > >> > >>> >> > >>>> >> >>> examples
> > > > >> > >>> >> > >>>> >> >>> >> >> once the Scala API merge is done.
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> I think the termination criterion is
> > fine
> > > > as
> > > > >> it
> > > > >> > is.
> > > > >> > >>> >> Just
> > > > >> > >>> >> > >>>> because
> > > > >> > >>> >> > >>>> >> >>> Scala
> > > > >> > >>> >> > >>>> >> >>> >> >> enables functional programming
> doesn't
> > > mean
> > > > >> it's
> > > > >> > >> always
> > > > >> > >>> >> > the
> > > > >> > >>> >> > >>>> best
> > > > >> > >>> >> > >>>> >> >>> >> >> choice. :D
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> So far we have ported: WordCount,
> > KMeans,
> > > > >> > >>> >> > ConnectedComponents,
> > > > >> > >>> >> > >>>> >> >>> >> >> WebLogAnalysis,
> TransitiveClosureNaive
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> These are the examples people called
> > dibs
> > > > on:
> > > > >> > >>> >> > >>>> >> >>> >> >>  - TriangleEnumration and PageRank
> > > (Fabian)
> > > > >> > >>> >> > >>>> >> >>> >> >>  - BatchGradientDescent (Márton)
> > > > >> > >>> >> > >>>> >> >>> >> >>  - ComputeEdgeDegrees (Hermann)
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> Those are unclaimed (if I'm not
> > > mistaken):
> > > > >> > >>> >> > >>>> >> >>> >> >>  - The relational Stuff
> > > > >> > >>> >> > >>>> >> >>> >> >>  - LinearRegression
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> Cheers,
> > > > >> > >>> >> > >>>> >> >>> >> >> Aljoscha
> > > > >> > >>> >> > >>>> >> >>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >> On Wed, Sep 10, 2014 at 4:23 PM,
> Kostas
> > > > >> Tzoumas
> > > > >> > <
> > > > >> > >>> >> > >>>> >> ktzoumas@apache.org
> > > > >> > >>> >> > >>>> >> >>> >
> > > > >> > >>> >> > >>>> >> >>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>> Transitive closure here, I also
> added
> > a
> > > > >> > >> termination
> > > > >> > >>> >> > criterion
> > > > >> > >>> >> > >>>> >> in the
> > > > >> > >>> >> > >>>> >> >>> >> Java
> > > > >> > >>> >> > >>>> >> >>> >> >>> version:
> > > > >> > >>> >> > >>>> >> >>> >>
> > > > >> > >>> >> > >>>>
> > > > >> > >>
> > > https://github.com/ktzoumas/incubator-flink/tree/tc-scala-example
> > > > >> > >>> >> > >>>> >> >>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>> Perhaps you can make the termination
> > > > >> criterion
> > > > >> > in
> > > > >> > >>> >> Scala
> > > > >> > >>> >> > more
> > > > >> > >>> >> > >>>> >> >>> >> functional?
> > > > >> > >>> >> > >>>> >> >>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>> I noticed that the examples package
> > name
> > > > is
> > > > >> > >>> >> > example.java but
> > > > >> > >>> >> > >>>> >> >>> >> examples.scala
> > > > >> > >>> >> > >>>> >> >>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>> Kostas
> > > > >> > >>> >> > >>>> >> >>> >> >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>> On Tue, Sep 9, 2014 at 6:12 PM,
> Kostas
> > > > >> Tzoumas
> > > > >> > <
> > > > >> > >>> >> > >>>> >> ktzoumas@apache.org
> > > > >> > >>> >> > >>>> >> >>> >
> > > > >> > >>> >> > >>>> >> >>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>> I'll take TransitiveClosure and
> > > > >> PiEstimation
> > > > >> > >> (was not
> > > > >> > >>> >> > on
> > > > >> > >>> >> > >>>> your
> > > > >> > >>> >> > >>>> >> >>> list).
> > > > >> > >>> >> > >>>> >> >>> >> >>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>> If nobody volunteers for the
> > relational
> > > > >> stuff
> > > > >> > I
> > > > >> > >> can
> > > > >> > >>> >> > take
> > > > >> > >>> >> > >>>> those
> > > > >> > >>> >> > >>>> >> as
> > > > >> > >>> >> > >>>> >> >>> >> well.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>> How about removing the
> > > "RelationalQuery"
> > > > >> from
> > > > >> > >> both
> > > > >> > >>> >> > Scala and
> > > > >> > >>> >> > >>>> >> Java?
> > > > >> > >>> >> > >>>> >> >>> It
> > > > >> > >>> >> > >>>> >> >>> >> >>>> seems to be a proper subset of
> TPC-H
> > > Q3.
> > > > >> Does
> > > > >> > it
> > > > >> > >> add
> > > > >> > >>> >> > some
> > > > >> > >>> >> > >>>> >> teaching
> > > > >> > >>> >> > >>>> >> >>> >> value on
> > > > >> > >>> >> > >>>> >> >>> >> >>>> top of TPC-H Q3?
> > > > >> > >>> >> > >>>> >> >>> >> >>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>> Kostas
> > > > >> > >>> >> > >>>> >> >>> >> >>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>> On Tue, Sep 9, 2014 at 5:57 PM,
> > > Aljoscha
> > > > >> > Krettek
> > > > >> > >> <
> > > > >> > >>> >> > >>>> >> >>> aljoscha@apache.org
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Thanks, I added it, along with an
> > > > ITCase.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> So far we have ported: WordCount,
> > > > KMeans,
> > > > >> > >>> >> > >>>> ConnectedComponents,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> WebLogAnalysis
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> These are the examples people
> called
> > > > dibs
> > > > >> on:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - TriangleEnumration and PageRank
> > > > >> (Fabian)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - BatchGradientDescent (Márton)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - ComputeEdgeDegrees (Hermann)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Those are unclaimed (if I'm not
> > > > mistaken):
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - TransitiveClosure
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - The relational Stuff
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - LinearRegression
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Cheers,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Aljoscha
> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> On Tue, Sep 9, 2014 at 5:21 PM,
> > Kostas
> > > > >> > Tzoumas <
> > > > >> > >>> >> > >>>> >> >>> ktzoumas@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > WebLog here:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
> > > > >> > >>> >> > >>>> >> >>> >>
> > > > >> > >>> >> > >>>> >> >>>
> > > > >> > >>> >> > >>>> >>
> > > > >> > >>> >> > >>>>
> > > > >> > >>> >> >
> > > > >> > >>> >>
> > > > >> > >>
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/ktzoumas/incubator-flink/tree/webloganalysis-example-scala
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > Do you need any more done?
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > On Tue, Sep 9, 2014 at 3:08 PM,
> > > > Aljoscha
> > > > >> > >> Krettek <
> > > > >> > >>> >> > >>>> >> >>> >> aljoscha@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> I added the ConnectedComponents
> > > > Example
> > > > >> > from
> > > > >> > >>> >> Vasia.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Keep 'em coming, people. :D
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> On Mon, Sep 8, 2014 at 6:07 PM,
> > > > Fabian
> > > > >> > >> Hueske <
> > > > >> > >>> >> > >>>> >> >>> fhueske@apache.org
> > > > >> > >>> >> > >>>> >> >>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > Alright, will do.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > Thanks!
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > 2014-09-08 17:48 GMT+02:00
> > > Aljoscha
> > > > >> > >> Krettek <
> > > > >> > >>> >> > >>>> >> >>> >> aljoscha@apache.org>:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Ok people, executive
> decision.
> > > :D
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Please look at
> KMeansData.java
> > > and
> > > > >> > >>> >> > KMeans.scala. I'm
> > > > >> > >>> >> > >>>> >> storing
> > > > >> > >>> >> > >>>> >> >>> >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> data
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> in multi-dimensional object
> > > arrays
> > > > >> and
> > > > >> > >> then
> > > > >> > >>> >> > >>>> converting
> > > > >> > >>> >> > >>>> >> it to
> > > > >> > >>> >> > >>>> >> >>> >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> required Java or Scala
> > objects.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Also, I changed isEqualTo to
> > > > >> equalTo to
> > > > >> > >> make
> > > > >> > >>> >> it
> > > > >> > >>> >> > >>>> >> consistent
> > > > >> > >>> >> > >>>> >> >>> >> with the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Java
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> API.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Regarding Join (and
> coGroup).
> > > > There
> > > > >> is
> > > > >> > no
> > > > >> > >> need
> > > > >> > >>> >> > for a
> > > > >> > >>> >> > >>>> >> >>> keyword,
> > > > >> > >>> >> > >>>> >> >>> >> you
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> can
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> just write:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> left.join(right).where(0).equalTo(1) {
> > > > >> > >> (le,
> > > > >> > >>> >> re)
> > > > >> > >>> >> > =>
> > > > >> > >>> >> > >>>> new
> > > > >> > >>> >> > >>>> >> >>> >> MyResult(le,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> re)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> }
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> On Mon, Sep 8, 2014 at 2:07
> > PM,
> > > > >> Fabian
> > > > >> > >> Hueske
> > > > >> > >>> >> <
> > > > >> > >>> >> > >>>> >> >>> >> fhueske@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > Aside from the DataSet
> > issue,
> > > I
> > > > >> also
> > > > >> > >> found
> > > > >> > >>> >> an
> > > > >> > >>> >> > >>>> >> >>> inconsistency
> > > > >> > >>> >> > >>>> >> >>> >> with
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Java
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > API. In Java join is done
> > as:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> ds1.join(ds2).where(...).equalTo(...)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > where in the current Scala
> > > this
> > > > >> is:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > ds1.join(d2).where(...).isEqualTo(...)
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > isEqualTo() should be
> > renamed
> > > to
> > > > >> > >> equalTo(),
> > > > >> > >>> >> > IMO.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > Also, join (+cross and
> > > coGroup?)
> > > > >> > lacks
> > > > >> > >> the
> > > > >> > >>> >> > with()
> > > > >> > >>> >> > >>>> >> method
> > > > >> > >>> >> > >>>> >> >>> >> because
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> "with"
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> is
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > a keyword in Scala. Should
> > be
> > > > >> offer
> > > > >> > >>> >> something
> > > > >> > >>> >> > >>>> similar
> > > > >> > >>> >> > >>>> >> for
> > > > >> > >>> >> > >>>> >> >>> >> Scala
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > or go
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> with
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > map() on Tuple2(left,
> > right)?
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > 2014-09-08 13:51 GMT+02:00
> > > > Stephan
> > > > >> > Ewen
> > > > >> > >> <
> > > > >> > >>> >> > >>>> >> sewen@apache.org
> > > > >> > >>> >> > >>>> >> >>> >:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> Instead of Strings,
> > > Object[][]
> > > > >> would
> > > > >> > >> work
> > > > >> > >>> >> as
> > > > >> > >>> >> > well.
> > > > >> > >>> >> > >>>> >> That
> > > > >> > >>> >> > >>>> >> >>> is a
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> generic
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> representation of a
> Tuple.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> Alternatively, they could
> > be
> > > > >> stored
> > > > >> > as
> > > > >> > >> Java
> > > > >> > >>> >> > or
> > > > >> > >>> >> > >>>> Scala
> > > > >> > >>> >> > >>>> >> >>> Tuples,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> with a
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> generic
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> utility method to convert
> > > > between
> > > > >> > the
> > > > >> > >> two.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> On Mon, Sep 8, 2014 at
> > 10:55
> > > > AM,
> > > > >> > Fabian
> > > > >> > >>> >> > Hueske
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> <fhueske@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > Yeah, I ran into the
> same
> > > > >> > problem...
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > +1 for using Strings
> and
> > > > >> parsing
> > > > >> > >> them,
> > > > >> > >>> >> but
> > > > >> > >>> >> > >>>> using
> > > > >> > >>> >> > >>>> >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > CSVFormat
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> won't
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> work
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > because this is based
> on
> > a
> > > > >> > >>> >> FileInputFormat.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > So we would need to
> parse
> > > the
> > > > >> > Strings
> > > > >> > >>> >> > >>>> manually...
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > 2014-09-08 10:35
> > GMT+02:00
> > > > >> > Aljoscha
> > > > >> > >>> >> Krettek
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > <aljoscha@apache.org>:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Hi,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > on second thought.
> > Maybe
> > > we
> > > > >> > should
> > > > >> > >> just
> > > > >> > >>> >> > change
> > > > >> > >>> >> > >>>> >> all
> > > > >> > >>> >> > >>>> >> >>> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > example
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> input
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > data to strings and
> use
> > > CSV
> > > > >> > input
> > > > >> > >>> >> > formats in
> > > > >> > >>> >> > >>>> all
> > > > >> > >>> >> > >>>> >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > examples.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> What
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> do
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > you think?
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Cheers,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Aljoscha
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > On Mon, Sep 8, 2014
> at
> > > 7:46
> > > > >> AM,
> > > > >> > >>> >> Aljoscha
> > > > >> > >>> >> > >>>> Krettek
> > > > >> > >>> >> > >>>> >> <
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> aljoscha@apache.org>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Hi,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > yes it's
> unfortunate
> > > that
> > > > >> the
> > > > >> > >> data
> > > > >> > >>> >> > types are
> > > > >> > >>> >> > >>>> >> >>> >> incompatible.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > I'm
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> afraid
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > you have to to what
> > you
> > > > >> > proposed:
> > > > >> > >>> >> move
> > > > >> > >>> >> > the
> > > > >> > >>> >> > >>>> >> data to
> > > > >> > >>> >> > >>>> >> >>> a
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > static
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> field
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> and
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > convert it in the
> > > > >> > >>> >> > getDefaultEdgeDataSet()
> > > > >> > >>> >> > >>>> >> method in
> > > > >> > >>> >> > >>>> >> >>> >> Scala.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > It's
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> not
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > nice, but copying
> > would
> > > > >> > >> duplicate the
> > > > >> > >>> >> > data
> > > > >> > >>> >> > >>>> and
> > > > >> > >>> >> > >>>> >> >>> make it
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > easier
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> for
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> it
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > to go out of sync
> in
> > > the
> > > > >> Java
> > > > >> > and
> > > > >> > >>> >> Scala
> > > > >> > >>> >> > >>>> >> versions.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > What do the others
> > > think?
> > > > >> This
> > > > >> > >> will
> > > > >> > >>> >> > probably
> > > > >> > >>> >> > >>>> >> occur
> > > > >> > >>> >> > >>>> >> >>> in
> > > > >> > >>> >> > >>>> >> >>> >> all
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> examples.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Cheers,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Aljoscha
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > On Sun, Sep 7, 2014
> > at
> > > > >> 10:04
> > > > >> > PM,
> > > > >> > >>> >> > Vasiliki
> > > > >> > >>> >> > >>>> >> Kalavri
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > <
> > > > vasilikikalavri@gmail.com
> > > > >> >
> > > > >> > >> wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Hey,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> I have ported the
> > > > >> Connected
> > > > >> > >>> >> Components
> > > > >> > >>> >> > >>>> >> example,
> > > > >> > >>> >> > >>>> >> >>> but
> > > > >> > >>> >> > >>>> >> >>> >> I am
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> not
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> sure
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> how
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > to
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> reuse the example
> > > input
> > > > >> data
> > > > >> > >> from
> > > > >> > >>> >> > >>>> >> java-examples.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> In the
> > > > >> > ConnectedComponentsData
> > > > >> > >>> >> class,
> > > > >> > >>> >> > the
> > > > >> > >>> >> > >>>> >> vertices
> > > > >> > >>> >> > >>>> >> >>> >> and
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> edges
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> data
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> are
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> produced by the
> > > methods
> > > > >> > >>> >> > >>>> >> getDefaultVertexDataSet()
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> and
> > > > >> getDefaultEdgeDataSet(),
> > > > >> > >> which
> > > > >> > >>> >> > take
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> an
> > > > >> > >>> >> > >>>> >> org.apache.flink.api.java.ExecutionEnvironment
> > > > >> > >>> >> > >>>> >> >>> as
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> parameter.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> One way is to
> > provide
> > > > >> public
> > > > >> > >> static
> > > > >> > >>> >> > fields
> > > > >> > >>> >> > >>>> >> (like
> > > > >> > >>> >> > >>>> >> >>> in
> > > > >> > >>> >> > >>>> >> >>> >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> WordCountData
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> class), but this
> > > > >> introduces a
> > > > >> > >>> >> > conversion
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> from
> > > > >> > >>> >> > >>>> org.apache.flink.api.java.tuple.Tuple2 to
> > > > >> > >>> >> > >>>> >> >>> Scala
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> tuple and
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> from
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> java.lang.Long to
> > > > >> scala.Long
> > > > >> > >> and I
> > > > >> > >>> >> > guess
> > > > >> > >>> >> > >>>> this
> > > > >> > >>> >> > >>>> >> is
> > > > >> > >>> >> > >>>> >> >>> an
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> unnecessary
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > complexity
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> for an example
> (?).
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Another way is, of
> > > > >> course, to
> > > > >> > >> copy
> > > > >> > >>> >> the
> > > > >> > >>> >> > >>>> example
> > > > >> > >>> >> > >>>> >> >>> data
> > > > >> > >>> >> > >>>> >> >>> >> in
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> the
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Scala
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > example.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Am I missing
> > something
> > > > >> here?
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Thanks!
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Cheers,
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> V.
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> On 5 September
> 2014
> > > > 15:52,
> > > > >> > >> Aljoscha
> > > > >> > >>> >> > >>>> Krettek <
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> aljoscha@apache.org
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > wrote:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> Alright, I
> updated
> > my
> > > > >> repo:
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
> > > > >> > >>> >> > >>>> >> >>> >>
> > > > >> > >>> >> >
> > > > >> https://github.com/aljoscha/incubator-flink/commits/scala-rework
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>>
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> This now has a
> > > working
> > > > >> > >> WordCount
> > > > >> > >>> >> > example.
> > > > >> > >>> >> > >>>> >> It's
> > > > >> > >>> >> > >>>> >> >>> >> pretty
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> much a
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> copy
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> of
> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> the Java example
> > with
> > > > >> some
> > > > >> > >> fixups
> > > > >> > >>> >> > for the
> > > > >> > >>> >> > >>>> >> syntax
> > > > >> > >>> >> > >>>> >> >>> and
> > > > >> > >>> >> > >>>> >> >>> >> >>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message