flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Restructuring the maven projects
Date Sat, 21 Mar 2015 18:40:28 GMT
Optimizer and Compiler can stay separate, of course.

I was just thinking that we have a lot of projects and this would be a
simple way to get rid of one. But it actually is intuitive to me as well to
keep them separate.

On Wed, Mar 18, 2015 at 1:23 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> I agree that it's a good idea to move the APIs into one module.
>
> But why should we merge client and compiler (optimizer) and the examples
> into one module?
> I think modules with clearly separated responsibilities can also help new
> contributors to navigate the code.
>
> 2015-03-17 16:16 GMT+01:00 Stephan Ewen <sewen@apache.org>:
>
> > The good thing about the API projects is that there is no dependency from
> > Java code to Scala code. I think that caused most of the issues.
> >
> > We may still want to keep it separate. I am not fully decided on this
> > yet...
> >
> > Stephan
> >
> > On Tue, Mar 17, 2015 at 3:52 PM, Ufuk Celebi <uce@apache.org> wrote:
> >
> > > Thanks for bringing this up, Till. You are right, but I think the main
> > > issue was that tight interaction between Java and Scala was
> problematic.
> > I
> > > am not sure whether this is such a big problem for the the APIs.
> > >
> > > On Tue, Mar 17, 2015 at 3:21 PM, Till Rohrmann <trohrmann@apache.org>
> > > wrote:
> > >
> > > > Putting the Scala and Java API into the same module means that we'll
> > have
> > > > more mixed Java/Scala projects, right? I just want to check if
> everyone
> > > is
> > > > aware of it considering our latest experiences with these kind of
> > > modules.
> > > >
> > > > On Tue, Mar 17, 2015 at 2:21 PM, Ufuk Celebi <uce@apache.org> wrote:
> > > >
> > > > > +1 I like the proposed structure.
> > > > >
> > > > > The only thing I was wondering about is whether to name "core" =>
> > > > "batch"?
> > > > >
> > > > > On Tue, Mar 17, 2015 at 11:37 AM, Márton Balassi <
> > > > balassi.marton@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1 for the proposed structure.
> > > > > >
> > > > > > I have no explicit preference for having batch and streaming
> scala
> > > > > together
> > > > > > or separated. That said streaming scala is considerably thin,
it
> > does
> > > > not
> > > > > > really require an own maven submodule.
> > > > > >
> > > > > > Marked an older JIRA for the same issue as duplicate. [1]
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/FLINK-1340
> > > > > >
> > > > > > On Tue, Mar 17, 2015 at 10:46 AM, Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > To not let this discussion die, here is a concrete JIRA
and a
> > > > proposed
> > > > > > > layout to restructure to.
> > > > > > >
> > > > > > > What remains to be discusses is whether we want to keep
the
> > > > Scala/Java
> > > > > > APIs
> > > > > > > for batch/streaming in separate projects or in one project.
> > > > > > >
> > > > > > > Also, we need to find a good time to do this, when we are
low
> on
> > > pull
> > > > > > > requests...
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/FLINK-1712
> > > > > > >
> > > > > > > Proposed Layout:
> > > > > > >
> > > > > > >  - flink-hadoop (shaded fat jar)
> > > > > > >
> > > > > > >  - Core (Core and Java and Scala)
> > > > > > >  - Streaming (core + java + scala)
> > > > > > >  - Runtime
> > > > > > >  - Client (Client + Optimizer)
> > > > > > >
> > > > > > >  - Examples (Java + Scala + Streaming Java + Streaming
Scala)
> > > > > > >  - Tests (test-utils (compile) and tests (test))
> > > > > > >
> > > > > > >  - Quickstarts
> > > > > > >    - Quickstart Java
> > > > > > >    - Quickstart Scala
> > > > > > >
> > > > > > >  - connectors / Input/Output Formats
> > > > > > >    - Avro
> > > > > > >    - HBase
> > > > > > >    - HadoopCompartibility
> > > > > > >    - HCatalogue
> > > > > > >    - JDBC
> > > > > > >    - kafka
> > > > > > >    - rabbit
> > > > > > >    - ...
> > > > > > >
> > > > > > >  - staging
> > > > > > >    - Gelly
> > > > > > >    - Gilbert (ML)
> > > > > > >    - spargel (deprecated)
> > > > > > >    - expression API
> > > > > > >
> > > > > > >  - contrib
> > > > > > >
> > > > > > >  - yarn
> > > > > > >
> > > > > > >  - dist
> > > > > > >
> > > > > > >  - yarn tests
> > > > > > >
> > > > > > >  - java 8
> > > > > > >
> > > > > > > On Mon, Jan 5, 2015 at 7:45 PM, Henry Saputra <
> > > > henry.saputra@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Marton, having 2 threads discussing same thing
can be
> > > > > confusing.
> > > > > > > >
> > > > > > > > - Henry
> > > > > > > >
> > > > > > > > On Mon, Jan 5, 2015 at 3:52 AM, Márton Balassi <
> > > > mbalassi@apache.org>
> > > > > > > > wrote:
> > > > > > > > > Let us consider this thread the standard for
the
> restructure,
> > > it
> > > > is
> > > > > > > > > perfectly in line with the wishes I have posted.
> > > > > > > > >
> > > > > > > > > +1 for keeping the 'flink-' prefix.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 5, 2015 at 10:24 AM, Henry Saputra
<
> > > > > > > henry.saputra@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> @Robert, and @Stephan, sure I am ok with
it, thanks for
> the
> > > > > > responses.
> > > > > > > > >>
> > > > > > > > >> - Henry
> > > > > > > > >>
> > > > > > > > >> On Mon, Jan 5, 2015 at 1:18 AM, Stephan Ewen
<
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > > >> > I think this works well together with
Marton's
> > > restructuring.
> > > > > > > > >> >
> > > > > > > > >> > I would vote to keep the "flink-" prefix,
because it
> > > > guarantees
> > > > > > that
> > > > > > > > the
> > > > > > > > >> > produced jars are prefixed with "flink-".
Otherwise, we
> > will
> > > > > have
> > > > > > to
> > > > > > > > >> start
> > > > > > > > >> > configuring a lot...
> > > > > > > > >> >
> > > > > > > > >> > Greetings,
> > > > > > > > >> > Stephan
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Jan 5, 2015 at 9:32 AM, Henry
Saputra <
> > > > > > > > henry.saputra@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Will this conflict with Marton's
restructuring proposal
> > > which
> > > > > > > happens
> > > > > > > > >> >> in another thread (see "Project
restructure" thread in
> > the
> > > > dev@
> > > > > > > > list).
> > > > > > > > >> >>
> > > > > > > > >> >> Since we are doing refactoring,
may I suggest that we
> > also
> > > > > remove
> > > > > > > > >> >> "flink-" prefix since maven group
name will indicate it
> > is
> > > > part
> > > > > > of
> > > > > > > > >> >> Flink.
> > > > > > > > >> >>
> > > > > > > > >> >> - Henry
> > > > > > > > >> >>
> > > > > > > > >> >> On Fri, Jan 2, 2015 at 6:52 AM,
Stephan Ewen <
> > > > sewen@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >> > Hi everyone!
> > > > > > > > >> >> >
> > > > > > > > >> >> > I think that by now, quite
a bit of the maven project
> > > > > structure
> > > > > > > > can be
> > > > > > > > >> >> > improved to get rid of some
legacy artifacts.
> > Especially
> > > > the
> > > > > > > > >> >> "flink-addons"
> > > > > > > > >> >> > project seems to be a catch-all
place for various
> > > projects.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Here is a suggestion what we
could do:
> > > > > > > > >> >> >
> > > > > > > > >> >> > 1) Move "flink-yarn" to the
root.
> > > > > > > > >> >> >
> > > > > > > > >> >> > 2) Move "flink-streaming" to
the root (this is
> planned
> > > > > anyways
> > > > > > > for
> > > > > > > > the
> > > > > > > > >> >> next
> > > > > > > > >> >> > release)
> > > > > > > > >> >> >
> > > > > > > > >> >> > 3) Create a project "flink-connectors",
which will
> > > contain
> > > > > > > "avro",
> > > > > > > > >> >> "jdbc",
> > > > > > > > >> >> > and "hbase". Should we have
them as separate
> > > sub-projects,
> > > > or
> > > > > > as
> > > > > > > > one
> > > > > > > > >> >> > project?
> > > > > > > > >> >> >
> > > > > > > > >> >> > 4) Consolidate the examples
into a single project
> > > > > > > "flink-examples",
> > > > > > > > >> where
> > > > > > > > >> >> > Java, Scala, Streaming examples
exist in different
> > > > packages.
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > Greetings,
> > > > > > > > >> >> > Stephan
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message