Mailing-List: contact dev-help@trafficserver.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@trafficserver.apache.org
Date: Tue, 26 Jan 2016 20:23:13 +0000 (UTC)
From: Jason Kenny <jkenny@yahoo-inc.com.INVALID>
Reply-To: Jason Kenny <jkenny@yahoo-inc.com>
To: Phil Sorber <sorber@apache.org>,
        "dev@trafficserver.apache.org" <dev@trafficserver.apache.org>
Message-ID: <192344497.450609.1453839793573.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: 
 <CABF6JR3ed0WOVZz6BZ5hO7O5cDVPknJp7C18iLbBjc56EMJi-w@mail.gmail.com>
References: <532468066.28515.1453761751840.JavaMail.yahoo.ref@mail.yahoo.com>
 <532468066.28515.1453761751840.JavaMail.yahoo@mail.yahoo.com>
 <CABF6JR0NdqxWLsPsMUo_L1ycduzsHsfY5mWdDp96X2K_N47tmg@mail.gmail.com>
 <1528330992.400755.1453835405904.JavaMail.yahoo@mail.yahoo.com>
 <CABF6JR3ed0WOVZz6BZ5hO7O5cDVPknJp7C18iLbBjc56EMJi-w@mail.gmail.com>
Subject: Re: Proposal for how to update source code layout.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


So the alternative build system is a low priority item as if we clean up th=
e automake files they will be easier to maintain and understand. For me it =
more about being able to easily use other systems to build ATS. Personally =
I would like to make a Scons/Parts build setup for ATS. I am a little bias =
on this honestly (as i wrote Parts and work with SCons), but I have had a l=
ot of luck with these systems, and honestly for larger systems that I know =
use this combo, people are generally happy about what it can do. The stuff =
is not perfect ( what is) but it is being improved. It still a big improvem=
ent to raw make and autotools in my view ( and from others that have used i=
t). I would like to provide such a system to the team so people could try i=
t out and praise or curse it as they like.=20

While I can go on about the many advantages for SCosn and Parts over autoto=
ols, the main difference between the two would be that autotools is a build=
 file generator which makes make files based on some configuration. SCons a=
nd Parts are a build system that can build differently based on a configura=
tion information passed in one of many different ways. This means if you ar=
e building ats on a linux box you can say:

scons --cfg=3Ddebug all=20

and get a debug build, then

scons --cfg=3Drelease


and get a release build, then realize you need to support building with Int=
el compiler v15 for 32-bit machines compatible with gcc 4.8 so you do a=20

scons --tc=3Dicc_15,gcc_4.8 --target=3Dx86

without wasting a minutes configuring new files rebuilding stuff, losing st=
uff if you did not run configure in a different directory, etc.
The main value of make in the unix/linux world is that it exists as part of=
 the system, SCons and Parts are not, and requires python run.=20

I am happy to talk more about it. However for me the issue is that having o=
ptions to only makes everything better and stronger. At the moment I think =
the current setup is not doing this.

Jason


________________________________
From: Phil Sorber <sorber@apache.org>
To: dev@trafficserver.apache.org; Jason Kenny <jkenny@yahoo-inc.com>=20
Sent: Tuesday, January 26, 2016 1:34 PM
Subject: Re: Proposal for how to update source code layout.


Can you talk more about the alternative build system? This is one that I do=
n't think we need to change really, but I am open to the idea and want to u=
nderstand it better.

Thanks.


On Tue, Jan 26, 2016 at 12:11 PM Jason Kenny <jkenny@yahoo-inc.com.invalid>=
 wrote:

Thank you for your input.=20
>Items like P_ I_ and other general code clean up I was hoping to do in a d=
ifferent phase. many of us could easy make a massive diff with all most of =
the issues corrected. However this diff would make most of our heads explod=
e. I think that it is very important to make each phase small and directed =
to help allow movement in a good direction. I know personally there are a n=
umber of item in the code itself I would like to address as well as fixing =
up the testing system. I believe that getting the relayout down first will =
help all other efforts as this will allow for clearer and more understandab=
le diffs when we address issues such as fixing up the P_ I_ stuff or adding=
 new tests, etc.
>Personally I would like to address:
>1) provide an alternative build system to the current auto-tools2) clean u=
p the testing. This is not small amount of work, and I would hope that I wo=
uld be able to provide a better subsystem that we all can use. One that is =
more accessible and useful than what we have at the moment.3) some stuff in=
 the code like the memory allocator and threaded event system, which i have=
 a background in doing
>But before this can be done correctly I really believe we have to clean up=
 the source layout. I want to get this done first and as quick as possible =
to reduce the pain caused by the source moves that would upset existing ( a=
nd future) pull requests.
>Jason
>
>
>      From: Phil Sorber <sorber@apache.org>
> To: dev@trafficserver.apache.org; Jason Kenny <jkenny@yahoo-inc.com>
> Sent: Tuesday, January 26, 2016 12:32 PM
> Subject: Re: Proposal for how to update source code layout.
>
>This seems generally good to me. I welcome a more modern structure. I also=
 prefer the first layout you mentioned, but it may be necessary to do the s=
econd layout and then move to the first. That might make others more comfor=
table. I personally see a lot of value in what you describe, but I know oth=
ers may still need more convincing and specific details.
>
>Also, you didn't mention some things specifically, but I assume we are doi=
ng away with things like the P_ and I_ in the header files? We should also =
make it so that the docs don't get needlessly built. I'm sure there are a b=
unch of other things, but those are the ones that come to mind.
>Another thing I think we need to be careful with is people who have long l=
ived feature branches. They really need to be rebasing against master a lot=
 or perhaps just pushing their changes upstream quicker.
>Be prepared for a lot of bike shedding. This is likely to be a long slow p=
rocess as we address the merits of each change and how it fits into the big=
ger picture and what are the pros and cons of each.
>
>On Mon, Jan 25, 2016 at 3:43 PM Jason Kenny <jkenny@yahoo-inc.com.invalid>=
 wrote:
>
>
>
>
>Hi everyone,
>
>Second try as the first attempt seems to be a blob of unformated text.
> I want to see about moving forward on the source clean up I believe we al=
l agreed to the last ATS summate. What I would like to get general agreemen=
t on what I plan to do to the source layout so it is clear why and where ev=
erything will go before I start making lots of small pull requests pulling =
moving source and cleaning up the build.
>First what are the goals?
>Cleaning up the source to will proved:
>1)            Better clarity on where to put new code.
>    a.            Making it easier to refactor code
>    b.            Making it easier for others to change code
>2)            simplify the build code
>    a.            making it easier to use other build system if desired
>    b.            making it easier for others to add or remove code
>3)            untangle the dependency mess of everything needing everythin=
g else
>4)            untangle what modules are being built
>5)            Make it easier for everyone to understand the Architecture a=
nd design
>6)            Make it easier to improve/cleanup  the testing systems
>7)            Allow a better setup to hopefully avoid any major source lay=
out refactors for a long time
>
>I want to clarify that this was generally agreed on by everyone at the ATS=
 summate. We want to avoid this unless needed. I believe people feel overal=
l this would be a good thing, but I want to make sure that I define the env=
isioned layout we will move to so we all understand the where we are going =
before doing this. Ideally we will want to do the major part of this work a=
s fast as we can to minimize the pain as much as possible.
>
>Before I go into the layout, I want to talk about process. I plan to break=
 this up into phases.
>Phase 1 =E2=80=93 move source to new location and any header cleanup that =
might be needed. Once this phase is done we will still have dependency issu=
es to deal with, however it should be clearer on what is wrong, and some of=
 the dependency issues should just go away
>Phase 2 =E2=80=93 refactor source to deal with dependency issues, to clean=
 up mega source files and start some cleanup of the tests. This will make i=
t easier to find objects and make changes in a way to that should not make =
compiling more complex. It should also make it easier to understand and lea=
rn the code.
>
>The hope I have is that will help minimize the size of any given diff. Wil=
l have more diff overall, but these will all be small and to the point.
>
>Now let talk about layout (finally :-) )
>
>The current layout has something like this at the top layer
>/build
>/ci
>/cmd
>/contrib
>/doc
>/example
>/iocore
>/lib
>/mgmt
>/plugins
>/proxy
>/rc
>/tools
>
>Source needed to build Traffics Server exits in /proxy, /mgmt, /lib, /cmd =
and /iocore. These directories also have tests in them as well. In some cas=
e the tests and the source is mixed. I would like to change the top level t=
o look like this instead:
>/build
>/ci
>/contrib
>/doc
>/plugins
>/rc
>/src
>/tests
>/tools
>
>We move the code in /examples, /proxy, /mgmt, /lib, /cmd and /iocore. The =
addition of a src and test directory will help clarify what is the source t=
o than traffic server application vs that of being a test or something else=
, such as an example or an optional plug-in. The code for /examples moves u=
nder /plugins, everything else under /src
>In the /src directory we would have a layout like this:
>src/
>  api/
>    c/
>    cpp/
>  config/
>  cmd/
>    =E2=80=A6
>  core/
>    =E2=80=A6
>  Infra/
>    =E2=80=A6
>
>Details:
>api =E2=80=93 contains all the API code for a given language. Any new lang=
uage API would be added here under a directory correctly names for the lang=
uage.
>config- Contains the default config file to run traffic server.
>cmd =E2=80=93 Contains the code for each program in a name directory.
>core =E2=80=93 Contains the main modules to build different programs under=
 the cmd/ directory. Modules under here depend on modules in Infra, or othe=
r modules in Core itself. Code here should be more Traffic server specific =
modules.
>infra =E2=80=93 Contains the support modules. Modules in here are leaf mod=
ules, may be reusable. If the module is not a leaf module it can only depen=
d on code in Infra and should be generally reusable.
>
>Dependencies:
>Modules should not depend on other modules that depend on them. The only e=
xpectations would be header only dependencies that don=E2=80=99t require li=
nk time circular dependencies.
>Modules in CMD can depend on modules in API, Core, Infra
>Modules in API can depend on modules in Core, Infra
>Modules in Core can depend on modules in Core, Infra
>Modules in Infra can depend on modules in Infra
>
>Header Patterns:
>The general include pattern will be changed to #include <module/header.h> =
for modules depending in on other headers. Files depending on headers in th=
e same module will be in the form of #include =E2=80=9Cheader.h=E2=80=9D
>
>An alternative to this layout would be
>Src/
>  api/
>    c/
>    cpp/
>  config/
>  mgmt/
>  proxy/
>    =E2=80=A6
>  iocore/
>    =E2=80=A6
>  lib/
>    =E2=80=A6
>
>The main difference is here is that it look like it did before. However mu=
ch of the code in proxy or mgmt would be moved to lib as it is a leaf compo=
nent, or the code would be moved in to a directory based on  module it woul=
d be used to generate, vs having lots of modules in a given directory. Beca=
use of this I felt it might be easier to use the first layout as it contain=
s a simpler set of logic on where code goes.
>Given that we agree with the layout. The process will be to move component=
s that are on the leaf first and slowly move up the chain. Each commit will=
 be on component at a time, with changes to the source files #include and a=
ny make file changes to allow the code to build as expected. After a compon=
ent is moved that changes that break up source can happen, where the commit=
 will include the makefile changes and the split source any header file cha=
nges need to support the split.
>
>I think that is it. I will deal with =E2=80=9Ctests=E2=80=9D after main so=
urce changes.
>
>Any question or concerns?
>
>Jason
>
>
>
>