flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: 答复: [discuss] merge module flink-yarn and flink-yarn-test
Date Mon, 26 Sep 2016 08:33:14 GMT
Hello Jinkui Shi,

Due to the nature of most of the Yarn tests, we need them to be in a
separate module. More concretely, these tests have a dependency on
'flink-dist' because they need to deploy the Flink fat jar to the Yarn
tests cluster. The fat jar also contains the 'flink-yarn' code. Thus,
'flink-yarn' needs to be a separate module and built before
'flink-yarn-tests'.

That being said, some of the tests don't need the fat jar, so we could
move some of the tests to 'flink-yarn'. However, that is mostly a
cosmetic change and not important for the testing coverage.

Best,
Max

On Thu, Sep 22, 2016 at 12:26 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> "flink-test-utils" contains, as the name says, utils for testing. Intended
> to be used by users in writing their own tests.
> "flink-tests" contains cross module tests, no user should ever need to have
> a dependency on that.
>
> They are different because users explicitly asked for test utils to be
> factored into a separate project.
>
> As an honest reply here: Setting up a project as huge as Flink need to take
> many things into account
>
>   - Multiple languages (Java / Scala), with limitations of IDEs in mind
>   - Dependency conflicts and much shading magic
>   - Dependency matrices (multiple hadoop and scala versions)
>   - Supporting earlier Java versions
>   - clean scope differentiation, so users can reuse utils and testing code
>
>
> That simply requires some extra modules once in a while. Flink people have
> worked hard on coming up with a structure that serves the need of the
> production users and automated build/testing systems. These production user
> requests are most important to us, and sometimes, we need to take cuts in
> "beauty of directory structure" to help them.
>
> Constantly accusing the community of creating bad structures before even
> trying to understand the reasoning behind that does not come across as very
> friendly. Constantly accusing the community of sloppy work just because
> your laptop settings are incompatible with the default configuration
> likewise.
>
> I hope you understand that.
>
>
> On Thu, Sep 22, 2016 at 2:58 AM, shijinkui <shijinkui@huawei.com> wrote:
>
> > Hi, Stephan
> >
> > Thanks for your reply.
> >
> > In my mind, Maven-shade-plugin and sbt-assembly both default exclude test
> > code for the fat jar.
> >
> > In fact, unit tests are use to test the main code, ensure our code logic
> > fit our expect . This is general convention. I think. Flink has be a top
> > apache project. We shouldn't be special. We're programmer, should be
> > professional.
> >
> > Even more, there are `flink-tes-utils-parent` and `flink-tests` module,
> > what's the relation between them.
> >
> > I have to ask why they are exist? Where is the start of such confusion
> > modules?
> >
> > I think we shouldn't do nothing for this. Code and design should be
> > comfortable.
> >
> > Thanks
> >
> > From Jinkui Shi
> >
> > -----邮件原件-----
> > 发件人: Stephan Ewen [mailto:sewen@apache.org]
> > 发送时间: 2016年9月21日 22:19
> > 收件人: dev@flink.apache.org
> > 主题: Re: [discuss] merge module flink-yarn and flink-yarn-test
> >
> > I would like Robert to comment on this.
> >
> > I think there was a reason to have different modules, which had again
> > something to do with the Maven Shade Plugin Dependencies and shading really
> > seem the trickiest thing in bigger Java/Scala projects ;-)
> >
> > On Wed, Sep 21, 2016 at 11:04 AM, shijinkui <shijinkui@huawei.com> wrote:
> >
> > > Hi, All
> > >
> > > There too much module in the root. There are no necessary to separate
> > > the test code from sub-module.
> > >
> > > I never see such design: two modules, one is main code, the other is
> > > test code.
> > >
> > > Is there some special reason?
> > >
> > > From Jinkui Shi
> > >
> >

Mime
View raw message