Mailing-List: contact dev-help@trafficserver.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@trafficserver.apache.org
Date: Tue, 26 Jan 2016 19:57:55 +0000 (UTC)
From: Jason Kenny <jkenny@yahoo-inc.com.INVALID>
Reply-To: Jason Kenny <jkenny@yahoo-inc.com>
To: James Peach <jpeach@apache.org>,
        "dev@trafficserver.apache.org" <dev@trafficserver.apache.org>
Message-ID: <992144796.442553.1453838275319.JavaMail.yahoo@mail.yahoo.com>
In-Reply-To: <95ED521F-DA57-4170-B3EE-79951A7FE387@apache.org>
References: <532468066.28515.1453761751840.JavaMail.yahoo.ref@mail.yahoo.com>
 <532468066.28515.1453761751840.JavaMail.yahoo@mail.yahoo.com>
 <95ED521F-DA57-4170-B3EE-79951A7FE387@apache.org>
Subject: Re: Proposal for how to update source code layout.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I recall your concerns at the last summate.

So I believe this proposal is to avoid big bang by making a set of small an=
d lots of small pull requests. I do have a OneNote page with general depend=
s in it I can copy paste my notes here if people want to see it. ( it will =
be a little ugly FYI as there are notes not a presentation)

The layout I propose here is based on my talk with you BTW as you had the v=
ery useful feedback on what should be done. I will try to clarify a few mor=
e items.

the layout here is to simplify the layout and to remove a number of existin=
g macro dependency loops we have at the moment and make it a little harder =
to add them back in. By Macro I mean stuff like modules lib depends on modu=
les in iocore which depends on modules in lib. By fixing this we will have =
a clear macro depends between iocore and libs. Of course the layout I sugge=
st does not have a lib or iocore, but a simple directory for API's( ie api)=
, programs (ie cmds), ATS specific code (core), general reusable code (infr=
a). I cannot come up with a reason to have logic like proxy or mgmt given w=
hat i see in the depend tree at this time, and with any explanation of func=
tionality. I think this layout would make it easier for us to see duplicate=
 logic and help us better refactor code in a more reliable form.

Given the step is done there will still exists micro depend issues between =
modules and files, but that will be easier to address independently and aft=
er we relayout the source as and breaking up of source will have a clear pl=
ace for it to go.

Based on what you said to me with wanting to clean up xyz as we moved, I fo=
und that it will be easier for use to move the source as a phase 1 then as =
a phase 2 or 3 fix up a module more directly. The reason for this is to red=
uce overall pain and to keep the commits more understandable and directed t=
o a certain goal.=20

I think in the end all the items you talk about needing to fix are items we=
 all agree should be addressed in some form. Much of this will be technical=
ly easier and less painful to address after we relayout the source.

Again small directed fixes. This will not be a big bang, but a set of lot o=
f small pull requests as modules by module is moved over. If you want, for =
example tcl to disappear during the move. I can make that happen (It is eas=
y to have code fall out). Again to stress the pull request will be like "mo=
ve tsutil to src/infra/tsutil and updated makefiles". It will not be move e=
verything in a massive diff.
=20
I hope this helps address your concerns. If you still have questions or con=
cerns or suggestion about something please let me know. I would like to mak=
e sure we have a clear understand of why this is a needed pre-step to a num=
ber of other wanted and needed improvements to ATS.

Jason


________________________________
From: James Peach <jpeach@apache.org>
To: dev@trafficserver.apache.org; Jason Kenny <jkenny@yahoo-inc.com>=20
Sent: Tuesday, January 26, 2016 12:47 PM
Subject: Re: Proposal for how to update source code layout.


I really think that developer time would be better spent on projects like r=
emoving the Tcl dependency, figuring out whether proxy/StatSystem.cc is rea=
lly needed any more, cleaning up the XML dependencies, removing STL contain=
ers where it makes sense, improving test coverage and usability, making it =
easy to run TSQA on releases, decoupling specific subsystems, moving traffi=
c_manager to iocore, etc.

Neither of the proposed layouts seems to capture the dependency graph that =
well, but there's not a lot of detail here. For example, what are the modul=
es in core? Why is core not just traffic_server?

Overall, I don't think a big-bang approach makes sense; we should make incr=
emental improvements, being careful to avoid regressions and build problems=
.


> On Jan 25, 2016, at 2:42 PM, Jason Kenny <jkenny@yahoo-inc.com.INVALID> w=
rote:
>=20
>=20
>=20
>=20
> Hi everyone,
>=20
> Second try as the first attempt seems to be a blob of unformated text.
> I want to see about moving forward on the source clean up I believe we al=
l agreed to the last ATS summate. What I would like to get general agreemen=
t on what I plan to do to the source layout so it is clear why and where ev=
erything will go before I start making lots of small pull requests pulling =
moving source and cleaning up the build.
> First what are the goals?
> Cleaning up the source to will proved:
> 1)            Better clarity on where to put new code.
>    a.            Making it easier to refactor code
>    b.            Making it easier for others to change code
> 2)            simplify the build code
>    a.            making it easier to use other build system if desired
>    b.            making it easier for others to add or remove code
> 3)            untangle the dependency mess of everything needing everythi=
ng else
> 4)            untangle what modules are being built
> 5)            Make it easier for everyone to understand the Architecture =
and design
> 6)            Make it easier to improve/cleanup  the testing systems
> 7)            Allow a better setup to hopefully avoid any major source la=
yout refactors for a long time
>=20
> I want to clarify that this was generally agreed on by everyone at the AT=
S summate. We want to avoid this unless needed. I believe people feel overa=
ll this would be a good thing, but I want to make sure that I define the en=
visioned layout we will move to so we all understand the where we are going=
 before doing this. Ideally we will want to do the major part of this work =
as fast as we can to minimize the pain as much as possible.
>=20
> Before I go into the layout, I want to talk about process. I plan to brea=
k this up into phases.
> Phase 1 =E2=80=93 move source to new location and any header cleanup that=
 might be needed. Once this phase is done we will still have dependency iss=
ues to deal with, however it should be clearer on what is wrong, and some o=
f the dependency issues should just go away
> Phase 2 =E2=80=93 refactor source to deal with dependency issues, to clea=
n up mega source files and start some cleanup of the tests. This will make =
it easier to find objects and make changes in a way to that should not make=
 compiling more complex. It should also make it easier to understand and le=
arn the code.
>=20
> The hope I have is that will help minimize the size of any given diff. Wi=
ll have more diff overall, but these will all be small and to the point.
>=20
> Now let talk about layout (finally :-) )
>=20
> The current layout has something like this at the top layer
> /build
> /ci
> /cmd
> /contrib
> /doc
> /example
> /iocore
> /lib
> /mgmt
> /plugins
> /proxy
> /rc
> /tools
>=20
> Source needed to build Traffics Server exits in /proxy, /mgmt, /lib, /cmd=
 and /iocore. These directories also have tests in them as well. In some ca=
se the tests and the source is mixed. I would like to change the top level =
to look like this instead:
> /build
> /ci
> /contrib
> /doc
> /plugins
> /rc
> /src
> /tests
> /tools
>=20
> We move the code in /examples, /proxy, /mgmt, /lib, /cmd and /iocore. The=
 addition of a src and test directory will help clarify what is the source =
to than traffic server application vs that of being a test or something els=
e, such as an example or an optional plug-in. The code for /examples moves =
under /plugins, everything else under /src
> In the /src directory we would have a layout like this:
> src/
>  api/=20
>    c/
>    cpp/
>  config/
>  cmd/=20
>    =E2=80=A6
>  core/=20
>    =E2=80=A6
>  Infra/=20
>    =E2=80=A6
>=20
> Details:
> api =E2=80=93 contains all the API code for a given language. Any new lan=
guage API would be added here under a directory correctly names for the lan=
guage.
> config- Contains the default config file to run traffic server.
> cmd =E2=80=93 Contains the code for each program in a name directory.
> core =E2=80=93 Contains the main modules to build different programs unde=
r the cmd/ directory. Modules under here depend on modules in Infra, or oth=
er modules in Core itself. Code here should be more Traffic server specific=
 modules.
> infra =E2=80=93 Contains the support modules. Modules in here are leaf mo=
dules, may be reusable. If the module is not a leaf module it can only depe=
nd on code in Infra and should be generally reusable.
>=20
> Dependencies:
> Modules should not depend on other modules that depend on them. The only =
expectations would be header only dependencies that don=E2=80=99t require l=
ink time circular dependencies.
> Modules in CMD can depend on modules in API, Core, Infra
> Modules in API can depend on modules in Core, Infra
> Modules in Core can depend on modules in Core, Infra
> Modules in Infra can depend on modules in Infra
>=20
> Header Patterns:
> The general include pattern will be changed to #include <module/header.h>=
 for modules depending in on other headers. Files depending on headers in t=
he same module will be in the form of #include =E2=80=9Cheader.h=E2=80=9D
>=20
> An alternative to this layout would be
> Src/
>  api/
>    c/
>    cpp/
>  config/
>  mgmt/
>  proxy/=20
>    =E2=80=A6
>  iocore/
>    =E2=80=A6
>  lib/=20
>    =E2=80=A6
>=20
> The main difference is here is that it look like it did before. However m=
uch of the code in proxy or mgmt would be moved to lib as it is a leaf comp=
onent, or the code would be moved in to a directory based on  module it wou=
ld be used to generate, vs having lots of modules in a given directory. Bec=
ause of this I felt it might be easier to use the first layout as it contai=
ns a simpler set of logic on where code goes.
> Given that we agree with the layout. The process will be to move componen=
ts that are on the leaf first and slowly move up the chain. Each commit wil=
l be on component at a time, with changes to the source files #include and =
any make file changes to allow the code to build as expected. After a compo=
nent is moved that changes that break up source can happen, where the commi=
t will include the makefile changes and the split source any header file ch=
anges need to support the split.
>=20
> I think that is it. I will deal with =E2=80=9Ctests=E2=80=9D after main s=
ource changes.
>=20
> Any question or concerns?
>=20
> Jason