flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Scala API rewrite almost complete
Date Sun, 28 Sep 2014 20:09:47 GMT
Hi Robert,
this might be a problem with Eclipse not having a strict separation
between compiling the src/main and src/test code. The code that
generates the TypeInformation is a macro. Macros are only usable if
the code that uses them is compiled in a separate compilation step
from the compilation step that copiles the macro itself.

If Eclipse cannot deal with this we might have to add all the Scala
tests to the flink-tests package. There it would be ensured that they
are compiled in a separate step. Hopefully?

Cheers,
Aljoscha

On Sun, Sep 28, 2014 at 9:25 PM, Henry Saputra <henry.saputra@gmail.com> wrote:
> Hi Robert,  I didn't have problem with IntelliJ idea last week, will try it
> again with latest master.
>
> User can deactivate unwanted maven profile via idea's maven project option
> (not remember exact name of that feature)
>
> On Sunday, September 28, 2014, Robert Metzger <rmetzger@apache.org> wrote:
>
>> I've worked on a quite outdated version of Flink for a while now and
>> rebased my code to the latest master on Friday.
>> Back at home, I wanted to continue my work and found that it is very
>> difficult to properly set up the latest eclipse for Flink.
>>
>> What I've done so far:
>> - Downloaded Eclipse Luna SR1 (for linux)
>> - Installed Scala IDE 4.0 nightly
>> - add the -clean flag to the eclipse.ini, because eclipse is not starting
>> anymore due to a bug
>> - installed some missing m2e connectors
>> - Set the Scala compiler to 2.10 (thats a new feature in Scala IDE) (See:
>> http://scala-ide.org/blog/scala-installations.html)
>>
>> Now, all the tests in flink-scala/src/test are mostly about this:
>> could not find implicit value for evidence parameter of type
>> org.apache.flink.api.common.typeinfo.TypeInformation[(Array[String],)];
>> /flink-scala/src/test/scala/org/apache/flink/api/scala/runtime
>>
>> It seems that the flink-tests project is calling Scala examples for testing
>> purposes. So if the flink-scala project is not building, its blocking
>> flink-tests from building.
>>
>> Any ideas?
>>
>> If you are wondering why I'm not switching to IntelliJ: It is currently not
>> possible to deactivate a maven build profile that is activated by default,
>> so I'm not able to properly work on the "flink-yarn" project with IntelliJ
>> (the respective issue has been closed at IntelliJ, so I guess the next
>> release will contain the fix).
>> In addition, I think its good if at least a few core developers make sure
>> that Flink is also "developable" using Eclipse.
>>
>>
>>
>>
>> On Sun, Sep 14, 2014 at 12:48 PM, Márton Balassi <balassi.marton@gmail.com
>> <javascript:;>>
>> wrote:
>>
>> > Answer posted to "Example packages naming convention" thread as the issue
>> > diverged from this topic.
>> >
>> > On Sun, Sep 14, 2014 at 11:14 AM, Kostas Tzoumas <ktzoumas@apache.org>
>> > wrote:
>> >
>> > > Good catch, I suggest to use examples
>> > >
>> > > On Sat, Sep 13, 2014 at 3:27 PM, Márton Balassi <
>> > balassi.marton@gmail.com>
>> > > wrote:
>> > >
>> > > > Pull request issued. One minor naming concern:
>> > > >
>> > > > As of today the scala examples are located at
>> > > > the org.apache.flink.examples.scala package, while the java ones in
>> > > > the org.apache.flink.example.java. I suggest using only one
>> convention
>> > > for
>> > > > this either example or examples.
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Marton
>> > > >
>> > > > On Fri, Sep 12, 2014 at 9:17 PM, Márton Balassi <
>> > > balassi.marton@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Sorry for being a bit silent after already bidding on LR. The pull
>> > > > request
>> > > > > is coming soon.
>> > > > >
>> > > > > On Fri, Sep 12, 2014 at 6:25 PM, Stephan Ewen <sewen@apache.org>
>> > > wrote:
>> > > > >
>> > > > >> I suppose that having the option between simple return type, and a
>> > > > >> collector is the easiest to understand.
>> > > > >>  Am 12.09.2014 16:50 schrieb "Aljoscha Krettek" <
>> > aljoscha@apache.org
>> > > >:
>> > > > >>
>> > > > >> > So, should I change join and coGroup to have a simple return
>> > value,
>> > > no
>> > > > >> > Option or Collection? Also what's happening with the relational
>> > > > >> > examples and the LinearRegression examples? I'd like to make a
>> > pull
>> > > > >> > request before this weekend.
>> > > > >> >
>> > > > >> > I also added a test that checks whether the Scala API has the
>> same
>> > > > >> > methods as the Java API (ScalaAPICompletenessTest).
>> > > > >> >
>> > > > >> > On Fri, Sep 12, 2014 at 4:00 PM, Aljoscha Krettek <
>> > > > aljoscha@apache.org>
>> > > > >> > wrote:
>> > > > >> > > Yes, there is already a Collector version, you can do:
>> > > > >> > >
>> > > > >> > > left.join(right).where("foo").equalTo("bar") {
>> > > > >> > >   (left, right, out: Collector[Page]) =>
>> > > > >> > >     if (...) out.collect(...)
>> > > > >> > > }
>> > > > >> > >
>> > > > >> > > I wasn't sure on what our Function2 variant should be. That's
>> > why
>> > > I
>> > > > >> > > asked. There are some cases where you want to have the option
>> > (pun
>> > > > >> > > intended) of either returning something from a join or not.
>> But
>> > > you
>> > > > >> > > could also use a filter after the join. It's a preference
>> > thing, I
>> > > > >> > > suppose.
>> > > > >> > >
>> > > > >> > > On Fri, Sep 12, 2014 at 2:43 PM, Stephan Ewen <
>> sewen@apache.org
>> > >
>> > > > >> wrote:
>> > > > >> > >> I think it seems weird that normal joins need to go through
>> > > option.
>> > > > >> > >>
>> > > > >> > >> The option variant is to allow filters in the join function.
>> > > > >> Wouldn't a
>> > > > >> > >> collector variant allow you to do the same, and would be
>> > > function3
>> > > > ?
>> > > > >> I
>> > > > >> > know
>> > > > >> > >> that option reads more functionally...
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >> Am 12.09.2014 14:24 schrieb "Aljoscha Krettek" <
>> > > > aljoscha@apache.org
>> > > > >> >:
>> > > > >> > >>>
>> > > > >> > >>> As already mentioned this is not possible because of type
>> > > erasure.
>> > > > >> We
>> > > > >> > >>> can only have one join variant that takes a Function2.
>> > > > >> > >>>
>> > > > >> > >>> On Fri, Sep 12, 2014 at 12:34 PM, Stephan Ewen <
>> > > sewen@apache.org>
>> > > > >> > wrote:
>> > > > >> > >>> > It would be nice to have a join variant that directly
>> > returns
>> > > > the
>> > > > >> > value
>> > > > >> > >>> > rathern than an option. Why not have both (they are
>> wrapped
>> > as
>> > > > >> > flatJoins
>> > > > >> > >>> > anyway below, right?)
>> > > > >> > >>> >
>> > > > >> > >>> > On Fri, Sep 12, 2014 at 11:50 AM, Fabian Hueske <
>> > > > >> fhueske@apache.org>
>> > > > >> > >> wrote:
>> > > > >> > >>> >
>> > > > >> > >>> >> Sweet! I'm lovin' this :-)
>> > > > >> > >>> >>
>> > > > >> > >>> >> 2014-09-12 11:46 GMT+02:00 Aljoscha Krettek <
>> > > > aljoscha@apache.org
>> > > > >> >:
>> > > > >> > >>> >>
>> > > > >> > >>> >> > Also, you can use CaseClasses directly as the type for
>> > CSV
>> > > > >> input.
>> > > > >> > So
>> > > > >> > >>> >> > instead of reading it as tuples and then having a
>> mapper
>> > > that
>> > > > >> > maps to
>> > > > >> > >>> >> > your case classes you can use:
>> > > > >> > >>> >> >
>> > > > >> > >>> >> > env.readCsv[Edge](...)
>> > > > >> > >>> >> >
>> > > > >> > >>> >> > On Fri, Sep 12, 2014 at 11:43 AM, Aljoscha Krettek <
>> > > > >> > >> aljoscha@apache.org>
>> > > > >> > >>> >> > wrote:
>> > > > >> > >>> >> > > I added support for specifying keys by name for
>> > > > CaseClasses.
>> > > > >> > Check
>> > > > >> > >> out
>> > > > >> > >>> >> > > the PageRank and TriangleEnumeration examples to see
>> it
>> > > in
>> > > > >> > action.
>> > > > >> > >>> >> > >
>> > > > >> > >>> >> > > @Kostas: I think you could use them for the TPC-H
>> > > examples.
>> > > > >> > >>> >> > >
>> > > > >> > >>> >> > > On Fri, Sep 12, 2014 at 7:23 AM, Aljoscha Krettek <
>> > > > >> > >> aljoscha@apache.org
>> > > > >> > >>> >> >
>> > > > >> > >>> >> > wrote:
>> > > > >> > >>> >> > >> Yes, that would allow list comprehensions. It would
>> be
>> > > > >> > possible to
>> > > > >> > >>> >> > >> have the Collection signature for join (and
>> coGroup),
>> > > > i.e.:
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> apply[R]((T, O) => TraversableOnce[O]): DataSet[O]
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> (T and O are the left and right input type, R is
>> > result
>> > > > >> type)
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> Then you can return collections and still return an
>> > > > option,
>> > > > >> as
>> > > > >> > in:
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> a.join(b).where(0).equalTo(0) { (l, r) => if (r >
>> ...)
>> > > > >> Some(l)
>> > > > >> > >> else
>> > > > >> > >>> >> > None }
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> Because there is an implicit conversion from Options
>> > to
>> > > a
>> > > > >> > >> Collection.
>> > > > >> > >>> >> > >> This will always wrap the return value in a List
>> with
>> > > only
>> > > > >> one
>> > > > >> > >> value.
>> > > > >> > >>> >> > >> I'm not sure we want the overhead here. I'm also not
>> > > sure
>> > > > >> > whether
>> > > > >> > >> we
>> > > > >> > >>> >> > >> want the overhead of always having to use an Option
>> > even
>> > > > >> though
>> > > > >> > >> the
>> > > > >> > >>> >> > >> join always returns a value.
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> What do you think?
>> > > > >> > >>> >> > >>
>> > > > >> > >>> >> > >> On Thu, Sep 11, 2014 at 11:22 PM, Fabian Hueske <
>> > > > >> > >> fhueske@apache.org>
>> > > > >> > >>> >> > wrote:
>> > > > >> > >>> >> > >>> Hmmm, tricky question...
>> > > > >> > >>> >> > >>> How about the Option for Join as this is a
>> tuple-wise
>> > > > >> > operation
>> > > > >> > >> and
>> > > > >> > >>> >> the
>> > > > >> > >>> >> > >>> Collection for Cogroup which is group-wise?
>> > > > >> > >>> >> > >>> Could we in that case use list comprehensions in
>> > > Cogroup
>> > > > >> > >> functions?
>> > > > >> > >>> >> > >>>
>> > > > >> > >>> >> > >>> Or is that too much mixing?
>> > > > >> > >>> >> > >>>
>> > > > >> > >>> >> > >>> 2014-09-11 23:00 GMT+02:00 Aljoscha Krettek <
>> > > > >> > aljoscha@apache.org
>> > > > >> > >>>:
>> > > > >> > >>> >> > >>>
>> > > > >> > >>> >> > >>>> I didn't look at the example either.
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>> >> > >>>> Addings collections is easy, it's just that we can
>> > > > either
>> > > > >> > have
>> > > > >> > >>> >> > >>>> Collections or the Option, not both.
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>> >> > >>>> For the coding style I followed this:
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>> >> >
>> > > > >> > >>
>> > > > >> >
>> > > > >>
>> > > >
>> > https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>> > > > >> > >>> >> ,
>> > > > >> > >>> >> > >>>> which itself is based on this:
>> > > > >> > http://docs.scala-lang.org/style/
>> > > > >> > >> .
>> > > > >> > >>> >> It
>> > > > >> > >>> >> > >>>> is different from the Java Code Guidelines we have
>> > in
>> > > > >> place,
>> > > > >> > >> yes.
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>> >> > >>>> On Thu, Sep 11, 2014 at 10:10 PM, Fabian Hueske <
>> > > > >> > >> fhueske@apache.org
>> > > > >> > >>> >> >
>> > > > >> > >>> >> > >>>> wrote:
>> > > > >> > >>> >> > >>>> > I haven't looked at the LineRank example in
>> > detail,
>> > > > but
>> > > > >> if
>> > > > >> > you
>> > > > >> > >>> >> > think that
>> > > > >> > >>> >> > >>>> > it adds something new to the examples
>> collection,
>> > we
>> > > > can
>> > > > >> > >> certainly
>> > > > >> > >>> >> > port
>> > > > >> > >>> >> > >>>> it
>> > > > >> > >>> >> > >>>> > also to Java.
>> > > > >> > >>> >> > >>>> > I think the Option and Collector return types
>> are
>> > > > >> > sufficient
>> > > > >> > >> right
>> > > > >> > >>> >> > now
>> > > > >> > >>> >> > >>>> but
>> > > > >> > >>> >> > >>>> > if Collections are easy to add, go for it. ;-)
>> > > > >> > >>> >> > >>>> >
>> > > > >> > >>> >> > >>>> > Great that the Scala primitives are working!
>> Also
>> > > > thanks
>> > > > >> > for
>> > > > >> > >>> >> adding
>> > > > >> > >>> >> > >>>> > genSequence and adapting my examples.
>> > > > >> > >>> >> > >>>> > Btw. does the codestyle not apply for Scala
>> files
>> > or
>> > > > do
>> > > > >> we
>> > > > >> > >> have a
>> > > > >> > >>> >> > >>>> different
>> > > > >> > >>> >> > >>>> > there?
>> > > > >> > >>> >> > >>>> >
>> > > > >> > >>> >> > >>>> > 2014-09-11 17:55 GMT+02:00 Aljoscha Krettek <
>> > > > >> > >> aljoscha@apache.org
>> > > > >> > >>> >> >:
>> > > > >> > >>> >> > >>>> >
>> > > > >> > >>> >> > >>>> >> What about the LineRank example? We had that in
>> > > Scala
>> > > > >> but
>> > > > >> > >> never
>> > > > >> > >>> >> > had a
>> > > > >> > >>> >> > >>>> >> Java Example.
>> > > > >> > >>> >> > >>>> >>
>> > > > >> > >>> >> > >>>> >> On Thu, Sep 11, 2014 at 5:51 PM, Aljoscha
>> > Krettek <
>> > > > >> > >>> >> > aljoscha@apache.org>
>> > > > >> > >>> >> > >>>> >> wrote:
>> > > > >> > >>> >> > >>>> >> > Yes, I like that. For the ITCases I always
>> just
>> > > > >> copied
>> > > > >> > the
>> > > > >> > >> Java
>> > > > >> > >>> >> > >>>> ITCase.
>> > > > >> > >>> >> > >>>> >> >
>> > > > >> > >>> >> > >>>> >> > The only examples that are missing now are
>> > > > >> > >> LinearRegression and
>> > > > >> > >>> >> > the
>> > > > >> > >>> >> > >>>> >> > relational stuff.
>> > > > >> > >>> >> > >>>> >> >
>> > > > >> > >>> >> > >>>> >> > On Thu, Sep 11, 2014 at 5:48 PM, Fabian
>> Hueske
>> > <
>> > > > >> > >>> >> > fhueske@apache.org>
>> > > > >> > >>> >> > >>>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >> I just removed the old CountEdgeDegrees
>> > example.
>> > > > >> > >>> >> > >>>> >> >> That was a preprocessing step for the
>> > > > >> > >> TriangleEnumeration, and
>> > > > >> > >>> >> > is now
>> > > > >> > >>> >> > >>>> >> part
>> > > > >> > >>> >> > >>>> >> >> of the new TriangleEnumerationOpt example.
>> > > > >> > >>> >> > >>>> >> >> So I guess, we don't need to port that one.
>> > As I
>> > > > >> said
>> > > > >> > >> before,
>> > > > >> > >>> >> > I'd
>> > > > >> > >>> >> > >>>> >> prefer to
>> > > > >> > >>> >> > >>>> >> >> keep Java and Scala examples in sync.
>> > > > >> > >>> >> > >>>> >> >>
>> > > > >> > >>> >> > >>>> >> >> Cheers, Fabian
>> > > > >> > >>> >> > >>>> >> >>
>> > > > >> > >>> >> > >>>> >> >> 2014-09-11 17:40 GMT+02:00 Aljoscha Krettek
>> <
>> > > > >> > >>> >> > aljoscha@apache.org>:
>> > > > >> > >>> >> > >>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> I added the PageRank example, thanks again
>> > > > fabian.
>> > > > >> :D
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> Regarding the other stuff:
>> > > > >> > >>> >> > >>>> >> >>>  - There is a comment in DataSet.scala
>> about
>> > > > >> including
>> > > > >> > >>> >> > >>>> >> >>> org.apache.flink.api.scala._ because of the
>> > > > >> > >> TypeInformation.
>> > > > >> > >>> >> > >>>> >> >>>  - I added generateSequence to
>> > > > >> ExecutionEnvironment.
>> > > > >> > >>> >> > >>>> >> >>>  - It is possible to use Scala Primitives
>> in
>> > > > >> Array, I
>> > > > >> > >> noticed
>> > > > >> > >>> >> > it
>> > > > >> > >>> >> > >>>> while
>> > > > >> > >>> >> > >>>> >> >>> writing the tests, you probably had an
>> older
>> > > > >> version
>> > > > >> > of
>> > > > >> > >> the
>> > > > >> > >>> >> > code.
>> > > > >> > >>> >> > >>>> >> >>>  - Yes, using List and other Interfaces is
>> > not
>> > > > >> > possible,
>> > > > >> > >> this
>> > > > >> > >>> >> > is
>> > > > >> > >>> >> > >>>> also
>> > > > >> > >>> >> > >>>> >> >>> a restriction in the Java API.
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> What do you think about the interface of
>> join
>> > > and
>> > > > >> > >> coGroup?
>> > > > >> > >>> >> > Right
>> > > > >> > >>> >> > >>>> now,
>> > > > >> > >>> >> > >>>> >> >>> you can either use a lambda that returns an
>> > > > Option
>> > > > >> or
>> > > > >> > the
>> > > > >> > >>> >> > lambda
>> > > > >> > >>> >> > >>>> with
>> > > > >> > >>> >> > >>>> >> >>> the Collector. Originally I wanted to have
>> > also
>> > > > >> have a
>> > > > >> > >> lambda
>> > > > >> > >>> >> > that
>> > > > >> > >>> >> > >>>> >> >>> returns a Collection, but due to type
>> erasure
>> > > > this
>> > > > >> has
>> > > > >> > >> the
>> > > > >> > >>> >> > same type
>> > > > >> > >>> >> > >>>> >> >>> as the lambda with the Option so I couldn't
>> > use
>> > > > it.
>> > > > >> > >> There is
>> > > > >> > >>> >> an
>> > > > >> > >>> >> > >>>> >> >>> implicit conversion from Option to a
>> > > Collection,
>> > > > >> so I
>> > > > >> > >> could
>> > > > >> > >>> >> > change
>> > > > >> > >>> >> > >>>> it
>> > > > >> > >>> >> > >>>> >> >>> without breaking the examples we have now.
>> > What
>> > > > do
>> > > > >> you
>> > > > >> > >> think?
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> So far we have ported: WordCount, KMeans,
>> > > > >> > >>> >> ConnectedComponents,
>> > > > >> > >>> >> > >>>> >> >>> WebLogAnalysis, TransitiveClosureNaive,
>> > > > >> > >>> >> > >>>> TriangleEnumerationNaive/Opt,
>> > > > >> > >>> >> > >>>> >> >>> PageRank
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> These are the examples people called dibs
>> on:
>> > > > >> > >>> >> > >>>> >> >>>  - BatchGradientDescent (Márton) (Should
>> be a
>> > > > port
>> > > > >> of
>> > > > >> > >>> >> > >>>> LinearRegression
>> > > > >> > >>> >> > >>>> >> >>> Example from Java)
>> > > > >> > >>> >> > >>>> >> >>>  - ComputeEdgeDegrees (Hermann)
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> Those are unclaimed (if I'm not mistaken):
>> > > > >> > >>> >> > >>>> >> >>>  - The relational Stuff
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> On Thu, Sep 11, 2014 at 3:06 PM, Stephan
>> > Ewen <
>> > > > >> > >>> >> > sewen@apache.org>
>> > > > >> > >>> >> > >>>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> > +1 for removing RelationQuery
>> > > > >> > >>> >> > >>>> >> >>> >
>> > > > >> > >>> >> > >>>> >> >>> > On Thu, Sep 11, 2014 at 3:04 PM, Aljoscha
>> > > > >> Krettek <
>> > > > >> > >>> >> > >>>> >> aljoscha@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> > wrote:
>> > > > >> > >>> >> > >>>> >> >>> >
>> > > > >> > >>> >> > >>>> >> >>> >> By the way, what was called
>> > > > >> BatchGradientDescent in
>> > > > >> > >> the
>> > > > >> > >>> >> > Scala
>> > > > >> > >>> >> > >>>> >> examples
>> > > > >> > >>> >> > >>>> >> >>> >> should be replaced by a port of the
>> > > > >> > LinearRegression
>> > > > >> > >>> >> > Example from
>> > > > >> > >>> >> > >>>> >> >>> >> Java. I had them as two separate
>> examples
>> > > > >> earlier.
>> > > > >> > >>> >> > >>>> >> >>> >>
>> > > > >> > >>> >> > >>>> >> >>> >> What about RelationalQuery and TPC-H-Q3.
>> > Any
>> > > > >> > thoughts
>> > > > >> > >>> >> about
>> > > > >> > >>> >> > >>>> removing
>> > > > >> > >>> >> > >>>> >> >>> >> RelationalQuery?
>> > > > >> > >>> >> > >>>> >> >>> >>
>> > > > >> > >>> >> > >>>> >> >>> >> On Thu, Sep 11, 2014 at 11:43 AM,
>> Aljoscha
>> > > > >> Krettek
>> > > > >> > <
>> > > > >> > >>> >> > >>>> >> aljoscha@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >
>> > > > >> > >>> >> > >>>> >> >>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> > I added the Triangle Enumeration
>> > Examples,
>> > > > >> thanks
>> > > > >> > >>> >> Fabian.
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> > So far we have ported: WordCount,
>> > KMeans,
>> > > > >> > >>> >> > ConnectedComponents,
>> > > > >> > >>> >> > >>>> >> >>> >> > WebLogAnalysis,
>> TransitiveClosureNaive,
>> > > > >> > >>> >> > >>>> >> TriangleEnumerationNaive/Opt
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> > These are the examples people called
>> > dibs
>> > > > on:
>> > > > >> > >>> >> > >>>> >> >>> >> >  - PageRank (Fabian)
>> > > > >> > >>> >> > >>>> >> >>> >> >  - BatchGradientDescent (Márton)
>> > > > >> > >>> >> > >>>> >> >>> >> >  - ComputeEdgeDegrees (Hermann)
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> > Those are unclaimed (if I'm not
>> > mistaken):
>> > > > >> > >>> >> > >>>> >> >>> >> >  - The relational Stuff
>> > > > >> > >>> >> > >>>> >> >>> >> >  - LinearRegression
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> > On Wed, Sep 10, 2014 at 6:04 PM,
>> > Aljoscha
>> > > > >> > Krettek <
>> > > > >> > >>> >> > >>>> >> >>> aljoscha@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >> Thanks, I added it. I'll keep a
>> running
>> > > > list
>> > > > >> of
>> > > > >> > >>> >> > >>>> ported/unported
>> > > > >> > >>> >> > >>>> >> >>> >> >> examples in my mails. I'll rename the
>> > > java
>> > > > >> > example
>> > > > >> > >>> >> > package to
>> > > > >> > >>> >> > >>>> >> >>> examples
>> > > > >> > >>> >> > >>>> >> >>> >> >> once the Scala API merge is done.
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> I think the termination criterion is
>> > fine
>> > > > as
>> > > > >> it
>> > > > >> > is.
>> > > > >> > >>> >> Just
>> > > > >> > >>> >> > >>>> because
>> > > > >> > >>> >> > >>>> >> >>> Scala
>> > > > >> > >>> >> > >>>> >> >>> >> >> enables functional programming
>> doesn't
>> > > mean
>> > > > >> it's
>> > > > >> > >> always
>> > > > >> > >>> >> > the
>> > > > >> > >>> >> > >>>> best
>> > > > >> > >>> >> > >>>> >> >>> >> >> choice. :D
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> So far we have ported: WordCount,
>> > KMeans,
>> > > > >> > >>> >> > ConnectedComponents,
>> > > > >> > >>> >> > >>>> >> >>> >> >> WebLogAnalysis,
>> TransitiveClosureNaive
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> These are the examples people called
>> > dibs
>> > > > on:
>> > > > >> > >>> >> > >>>> >> >>> >> >>  - TriangleEnumration and PageRank
>> > > (Fabian)
>> > > > >> > >>> >> > >>>> >> >>> >> >>  - BatchGradientDescent (Márton)
>> > > > >> > >>> >> > >>>> >> >>> >> >>  - ComputeEdgeDegrees (Hermann)
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> Those are unclaimed (if I'm not
>> > > mistaken):
>> > > > >> > >>> >> > >>>> >> >>> >> >>  - The relational Stuff
>> > > > >> > >>> >> > >>>> >> >>> >> >>  - LinearRegression
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> Cheers,
>> > > > >> > >>> >> > >>>> >> >>> >> >> Aljoscha
>> > > > >> > >>> >> > >>>> >> >>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >> On Wed, Sep 10, 2014 at 4:23 PM,
>> Kostas
>> > > > >> Tzoumas
>> > > > >> > <
>> > > > >> > >>> >> > >>>> >> ktzoumas@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >
>> > > > >> > >>> >> > >>>> >> >>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>> Transitive closure here, I also
>> added
>> > a
>> > > > >> > >> termination
>> > > > >> > >>> >> > criterion
>> > > > >> > >>> >> > >>>> >> in the
>> > > > >> > >>> >> > >>>> >> >>> >> Java
>> > > > >> > >>> >> > >>>> >> >>> >> >>> version:
>> > > > >> > >>> >> > >>>> >> >>> >>
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>
>> > > https://github.com/ktzoumas/incubator-flink/tree/tc-scala-example
>> > > > >> > >>> >> > >>>> >> >>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>> Perhaps you can make the termination
>> > > > >> criterion
>> > > > >> > in
>> > > > >> > >>> >> Scala
>> > > > >> > >>> >> > more
>> > > > >> > >>> >> > >>>> >> >>> >> functional?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>> I noticed that the examples package
>> > name
>> > > > is
>> > > > >> > >>> >> > example.java but
>> > > > >> > >>> >> > >>>> >> >>> >> examples.scala
>> > > > >> > >>> >> > >>>> >> >>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>> Kostas
>> > > > >> > >>> >> > >>>> >> >>> >> >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>> On Tue, Sep 9, 2014 at 6:12 PM,
>> Kostas
>> > > > >> Tzoumas
>> > > > >> > <
>> > > > >> > >>> >> > >>>> >> ktzoumas@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >
>> > > > >> > >>> >> > >>>> >> >>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> I'll take TransitiveClosure and
>> > > > >> PiEstimation
>> > > > >> > >> (was not
>> > > > >> > >>> >> > on
>> > > > >> > >>> >> > >>>> your
>> > > > >> > >>> >> > >>>> >> >>> list).
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> If nobody volunteers for the
>> > relational
>> > > > >> stuff
>> > > > >> > I
>> > > > >> > >> can
>> > > > >> > >>> >> > take
>> > > > >> > >>> >> > >>>> those
>> > > > >> > >>> >> > >>>> >> as
>> > > > >> > >>> >> > >>>> >> >>> >> well.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> How about removing the
>> > > "RelationalQuery"
>> > > > >> from
>> > > > >> > >> both
>> > > > >> > >>> >> > Scala and
>> > > > >> > >>> >> > >>>> >> Java?
>> > > > >> > >>> >> > >>>> >> >>> It
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> seems to be a proper subset of
>> TPC-H
>> > > Q3.
>> > > > >> Does
>> > > > >> > it
>> > > > >> > >> add
>> > > > >> > >>> >> > some
>> > > > >> > >>> >> > >>>> >> teaching
>> > > > >> > >>> >> > >>>> >> >>> >> value on
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> top of TPC-H Q3?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> Kostas
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> On Tue, Sep 9, 2014 at 5:57 PM,
>> > > Aljoscha
>> > > > >> > Krettek
>> > > > >> > >> <
>> > > > >> > >>> >> > >>>> >> >>> aljoscha@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Thanks, I added it, along with an
>> > > > ITCase.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> So far we have ported: WordCount,
>> > > > KMeans,
>> > > > >> > >>> >> > >>>> ConnectedComponents,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> WebLogAnalysis
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> These are the examples people
>> called
>> > > > dibs
>> > > > >> on:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - TriangleEnumration and PageRank
>> > > > >> (Fabian)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - BatchGradientDescent (Márton)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - ComputeEdgeDegrees (Hermann)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Those are unclaimed (if I'm not
>> > > > mistaken):
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - TransitiveClosure
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - The relational Stuff
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>  - LinearRegression
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Cheers,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> Aljoscha
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> On Tue, Sep 9, 2014 at 5:21 PM,
>> > Kostas
>> > > > >> > Tzoumas <
>> > > > >> > >>> >> > >>>> >> >>> ktzoumas@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > WebLog here:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
>> > > > >> > >>> >> > >>>> >> >>> >>
>> > > > >> > >>> >> > >>>> >> >>>
>> > > > >> > >>> >> > >>>> >>
>> > > > >> > >>> >> > >>>>
>> > > > >> > >>> >> >
>> > > > >> > >>> >>
>> > > > >> > >>
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/ktzoumas/incubator-flink/tree/webloganalysis-example-scala
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > Do you need any more done?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > On Tue, Sep 9, 2014 at 3:08 PM,
>> > > > Aljoscha
>> > > > >> > >> Krettek <
>> > > > >> > >>> >> > >>>> >> >>> >> aljoscha@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> > wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> I added the ConnectedComponents
>> > > > Example
>> > > > >> > from
>> > > > >> > >>> >> Vasia.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Keep 'em coming, people. :D
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> On Mon, Sep 8, 2014 at 6:07 PM,
>> > > > Fabian
>> > > > >> > >> Hueske <
>> > > > >> > >>> >> > >>>> >> >>> fhueske@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > Alright, will do.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > Thanks!
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> > 2014-09-08 17:48 GMT+02:00
>> > > Aljoscha
>> > > > >> > >> Krettek <
>> > > > >> > >>> >> > >>>> >> >>> >> aljoscha@apache.org>:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Ok people, executive
>> decision.
>> > > :D
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Please look at
>> KMeansData.java
>> > > and
>> > > > >> > >>> >> > KMeans.scala. I'm
>> > > > >> > >>> >> > >>>> >> storing
>> > > > >> > >>> >> > >>>> >> >>> >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> data
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> in multi-dimensional object
>> > > arrays
>> > > > >> and
>> > > > >> > >> then
>> > > > >> > >>> >> > >>>> converting
>> > > > >> > >>> >> > >>>> >> it to
>> > > > >> > >>> >> > >>>> >> >>> >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> required Java or Scala
>> > objects.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Also, I changed isEqualTo to
>> > > > >> equalTo to
>> > > > >> > >> make
>> > > > >> > >>> >> it
>> > > > >> > >>> >> > >>>> >> consistent
>> > > > >> > >>> >> > >>>> >> >>> >> with the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Java
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> API.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> Regarding Join (and
>> coGroup).
>> > > > There
>> > > > >> is
>> > > > >> > no
>> > > > >> > >> need
>> > > > >> > >>> >> > for a
>> > > > >> > >>> >> > >>>> >> >>> keyword,
>> > > > >> > >>> >> > >>>> >> >>> >> you
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> can
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> just write:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> left.join(right).where(0).equalTo(1) {
>> > > > >> > >> (le,
>> > > > >> > >>> >> re)
>> > > > >> > >>> >> > =>
>> > > > >> > >>> >> > >>>> new
>> > > > >> > >>> >> > >>>> >> >>> >> MyResult(le,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> re)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> }
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> On Mon, Sep 8, 2014 at 2:07
>> > PM,
>> > > > >> Fabian
>> > > > >> > >> Hueske
>> > > > >> > >>> >> <
>> > > > >> > >>> >> > >>>> >> >>> >> fhueske@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > Aside from the DataSet
>> > issue,
>> > > I
>> > > > >> also
>> > > > >> > >> found
>> > > > >> > >>> >> an
>> > > > >> > >>> >> > >>>> >> >>> inconsistency
>> > > > >> > >>> >> > >>>> >> >>> >> with
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Java
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > API. In Java join is done
>> > as:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> ds1.join(ds2).where(...).equalTo(...)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > where in the current Scala
>> > > this
>> > > > >> is:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > ds1.join(d2).where(...).isEqualTo(...)
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > isEqualTo() should be
>> > renamed
>> > > to
>> > > > >> > >> equalTo(),
>> > > > >> > >>> >> > IMO.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > Also, join (+cross and
>> > > coGroup?)
>> > > > >> > lacks
>> > > > >> > >> the
>> > > > >> > >>> >> > with()
>> > > > >> > >>> >> > >>>> >> method
>> > > > >> > >>> >> > >>>> >> >>> >> because
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> "with"
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> is
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > a keyword in Scala. Should
>> > be
>> > > > >> offer
>> > > > >> > >>> >> something
>> > > > >> > >>> >> > >>>> similar
>> > > > >> > >>> >> > >>>> >> for
>> > > > >> > >>> >> > >>>> >> >>> >> Scala
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > or go
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> with
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > map() on Tuple2(left,
>> > right)?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> > 2014-09-08 13:51 GMT+02:00
>> > > > Stephan
>> > > > >> > Ewen
>> > > > >> > >> <
>> > > > >> > >>> >> > >>>> >> sewen@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> Instead of Strings,
>> > > Object[][]
>> > > > >> would
>> > > > >> > >> work
>> > > > >> > >>> >> as
>> > > > >> > >>> >> > well.
>> > > > >> > >>> >> > >>>> >> That
>> > > > >> > >>> >> > >>>> >> >>> is a
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> generic
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> representation of a
>> Tuple.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> Alternatively, they could
>> > be
>> > > > >> stored
>> > > > >> > as
>> > > > >> > >> Java
>> > > > >> > >>> >> > or
>> > > > >> > >>> >> > >>>> Scala
>> > > > >> > >>> >> > >>>> >> >>> Tuples,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> with a
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> generic
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> utility method to convert
>> > > > between
>> > > > >> > the
>> > > > >> > >> two.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> On Mon, Sep 8, 2014 at
>> > 10:55
>> > > > AM,
>> > > > >> > Fabian
>> > > > >> > >>> >> > Hueske
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> <fhueske@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > Yeah, I ran into the
>> same
>> > > > >> > problem...
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > +1 for using Strings
>> and
>> > > > >> parsing
>> > > > >> > >> them,
>> > > > >> > >>> >> but
>> > > > >> > >>> >> > >>>> using
>> > > > >> > >>> >> > >>>> >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > CSVFormat
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> won't
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> work
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > because this is based
>> on
>> > a
>> > > > >> > >>> >> FileInputFormat.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > So we would need to
>> parse
>> > > the
>> > > > >> > Strings
>> > > > >> > >>> >> > >>>> manually...
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > 2014-09-08 10:35
>> > GMT+02:00
>> > > > >> > Aljoscha
>> > > > >> > >>> >> Krettek
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > <aljoscha@apache.org>:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Hi,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > on second thought.
>> > Maybe
>> > > we
>> > > > >> > should
>> > > > >> > >> just
>> > > > >> > >>> >> > change
>> > > > >> > >>> >> > >>>> >> all
>> > > > >> > >>> >> > >>>> >> >>> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > example
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> input
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > data to strings and
>> use
>> > > CSV
>> > > > >> > input
>> > > > >> > >>> >> > formats in
>> > > > >> > >>> >> > >>>> all
>> > > > >> > >>> >> > >>>> >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > examples.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> What
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> do
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > you think?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Cheers,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > Aljoscha
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > On Mon, Sep 8, 2014
>> at
>> > > 7:46
>> > > > >> AM,
>> > > > >> > >>> >> Aljoscha
>> > > > >> > >>> >> > >>>> Krettek
>> > > > >> > >>> >> > >>>> >> <
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> aljoscha@apache.org>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Hi,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > yes it's
>> unfortunate
>> > > that
>> > > > >> the
>> > > > >> > >> data
>> > > > >> > >>> >> > types are
>> > > > >> > >>> >> > >>>> >> >>> >> incompatible.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > I'm
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> afraid
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > you have to to what
>> > you
>> > > > >> > proposed:
>> > > > >> > >>> >> move
>> > > > >> > >>> >> > the
>> > > > >> > >>> >> > >>>> >> data to
>> > > > >> > >>> >> > >>>> >> >>> a
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > static
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> field
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> and
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > convert it in the
>> > > > >> > >>> >> > getDefaultEdgeDataSet()
>> > > > >> > >>> >> > >>>> >> method in
>> > > > >> > >>> >> > >>>> >> >>> >> Scala.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > It's
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> not
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > nice, but copying
>> > would
>> > > > >> > >> duplicate the
>> > > > >> > >>> >> > data
>> > > > >> > >>> >> > >>>> and
>> > > > >> > >>> >> > >>>> >> >>> make it
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > easier
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> for
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> it
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > to go out of sync
>> in
>> > > the
>> > > > >> Java
>> > > > >> > and
>> > > > >> > >>> >> Scala
>> > > > >> > >>> >> > >>>> >> versions.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > What do the others
>> > > think?
>> > > > >> This
>> > > > >> > >> will
>> > > > >> > >>> >> > probably
>> > > > >> > >>> >> > >>>> >> occur
>> > > > >> > >>> >> > >>>> >> >>> in
>> > > > >> > >>> >> > >>>> >> >>> >> all
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> examples.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Cheers,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > Aljoscha
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > On Sun, Sep 7, 2014
>> > at
>> > > > >> 10:04
>> > > > >> > PM,
>> > > > >> > >>> >> > Vasiliki
>> > > > >> > >>> >> > >>>> >> Kalavri
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > > <
>> > > > vasilikikalavri@gmail.com
>> > > > >> >
>> > > > >> > >> wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Hey,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> I have ported the
>> > > > >> Connected
>> > > > >> > >>> >> Components
>> > > > >> > >>> >> > >>>> >> example,
>> > > > >> > >>> >> > >>>> >> >>> but
>> > > > >> > >>> >> > >>>> >> >>> >> I am
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> not
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> sure
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> how
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > to
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> reuse the example
>> > > input
>> > > > >> data
>> > > > >> > >> from
>> > > > >> > >>> >> > >>>> >> java-examples.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> In the
>> > > > >> > ConnectedComponentsData
>> > > > >> > >>> >> class,
>> > > > >> > >>> >> > the
>> > > > >> > >>> >> > >>>> >> vertices
>> > > > >> > >>> >> > >>>> >> >>> >> and
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> edges
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> data
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> are
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> produced by the
>> > > methods
>> > > > >> > >>> >> > >>>> >> getDefaultVertexDataSet()
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> and
>> > > > >> getDefaultEdgeDataSet(),
>> > > > >> > >> which
>> > > > >> > >>> >> > take
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> an
>> > > > >> > >>> >> > >>>> >> org.apache.flink.api.java.ExecutionEnvironment
>> > > > >> > >>> >> > >>>> >> >>> as
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> parameter.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> One way is to
>> > provide
>> > > > >> public
>> > > > >> > >> static
>> > > > >> > >>> >> > fields
>> > > > >> > >>> >> > >>>> >> (like
>> > > > >> > >>> >> > >>>> >> >>> in
>> > > > >> > >>> >> > >>>> >> >>> >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> WordCountData
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> class), but this
>> > > > >> introduces a
>> > > > >> > >>> >> > conversion
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> from
>> > > > >> > >>> >> > >>>> org.apache.flink.api.java.tuple.Tuple2 to
>> > > > >> > >>> >> > >>>> >> >>> Scala
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> tuple and
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> from
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> java.lang.Long to
>> > > > >> scala.Long
>> > > > >> > >> and I
>> > > > >> > >>> >> > guess
>> > > > >> > >>> >> > >>>> this
>> > > > >> > >>> >> > >>>> >> is
>> > > > >> > >>> >> > >>>> >> >>> an
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> unnecessary
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > complexity
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> for an example
>> (?).
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Another way is, of
>> > > > >> course, to
>> > > > >> > >> copy
>> > > > >> > >>> >> the
>> > > > >> > >>> >> > >>>> example
>> > > > >> > >>> >> > >>>> >> >>> data
>> > > > >> > >>> >> > >>>> >> >>> >> in
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> the
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> Scala
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > example.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Am I missing
>> > something
>> > > > >> here?
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Thanks!
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> Cheers,
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> V.
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >> On 5 September
>> 2014
>> > > > 15:52,
>> > > > >> > >> Aljoscha
>> > > > >> > >>> >> > >>>> Krettek <
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> aljoscha@apache.org
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > wrote:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> Alright, I
>> updated
>> > my
>> > > > >> repo:
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >>
>> > > > >> > >>> >> > >>>> >> >>> >>
>> > > > >> > >>> >> >
>> > > > >> https://github.com/aljoscha/incubator-flink/commits/scala-rework
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>>
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> This now has a
>> > > working
>> > > > >> > >> WordCount
>> > > > >> > >>> >> > example.
>> > > > >> > >>> >> > >>>> >> It's
>> > > > >> > >>> >> > >>>> >> >>> >> pretty
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> much a
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> copy
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> of
>> > > > >> > >>> >> > >>>> >> >>> >> >>>>> >> >> >> > > >>> the Java example
>> > with
>> > > > >> some
>> > > > >> > >> fixups
>> > > > >> > >>> >> > for the
>> > > > >> > >>> >> > >>>> >> syntax
>> > > > >> > >>> >> > >>>> >> >>> and
>> > > > >> > >>> >> > >>>> >> >>> >> >>

Mime
View raw message