flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Gévay <gga...@gmail.com>
Subject Re: Adding flink-scala as a dependency to flink-streaming-core
Date Wed, 10 Jun 2015 13:55:58 GMT
> "it does not feel right to add an API package to a core package

Yes, that makes sense. I just tried removing the flink-java dependency
from flink-streaming to see what needs it, and it builds fine without
it :)

What do you think about the second option? (to move the Scala
typeutils (or just CaseClassTypeInfo and CaseClassComparator) to
flink-core, to where the Java typeutils are) This would be mixing
scala code to an otherwise java module, but I don't know whether that
is a problem.

Cheers,
Gabor

2015-06-10 15:46 GMT+02:00 Till Rohrmann <trohrmann@apache.org>:
> Btw: I noticed that all streaming modules depend on flink-core,
> flink-runtime, flink-clients and flink-java. Is there a particular reason
> why the streaming connectors depend on flink-clients and flink-java?
>
> On Wed, Jun 10, 2015 at 3:41 PM Till Rohrmann <trohrmann@apache.org> wrote:
>
>> I see the reason why you want to add flink-scala as a dependency to
>> flink-streaming-core. However, it does not feel right to add an API package
>> to a core package IMHO.
>>
>> But I noticed that flink-streaming-core also depends on flink-java. Which
>> seems odd to me as well. I'm not a streaming expert and thus cannot tell
>> much about the reasons why a core package has a dependency on an API
>> package but for me this looks more like an indicator for a necessary
>> restructuring of our packages. Maybe someone working on the streaming parts
>> can chime in and shed some light on the required dependencies.
>>
>> Cheers,
>> Till
>>
>>
>> On Wed, Jun 10, 2015 at 2:13 PM Gábor Gévay <ggab90@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I would like to ask if it would be OK if I added flink-scala as a
>>> dependency to flink-streaming-core. An alternative would be to move
>>> the Scala typeutils to flink-core (to where the Java typeutils are).
>>> Why I need this:
>>>
>>> While I am implementing the fast median calculation for windows as
>>> part of my Google Summer of Code project, I am refactoring the way
>>> sum, min, max, etc. are accessing the user-specified field
>>> (https://github.com/apache/flink/pull/684). Currently both the logic
>>> of their aggregators are duplicated for the different kinds of types
>>> (tuple, pojo, array, Scala case class, simple), and also the field
>>> access logic is duplicated across the different aggregators. In my
>>> GSoC project I will implement some further methods (avg, variance,
>>> etc.) that take the same kind of parameters as sum, min, etc., so it
>>> will be neccassary to have the field access logic centralized (this is
>>> the FieldAccessor class in the PR). It would be convenient if this
>>> could also handle Scala case classes, for which CaseClassTypeInfo is
>>> needed which is currently in flink-scala.
>>>
>>> Best regards,
>>> Gabor
>>>
>>

Mime
View raw message