httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject Re: the wheel of httpd-dev life is surely slowing down, solutions please
Date Thu, 13 Nov 2003 13:47:17 GMT

Hi all...

I just have to jump in here since the topic is fascinating...
...and I think there's an opportunity here to review something
that has contributed to the 'slow down' at httpd-dev which
no one has seemed to grasp (yet).

I will call it... "The Covalent Factor".

If you look at what has REALLY happened in the past
3 years ( yes... going back that far since it's now 4 or
5 years since 2.0 became a real blib on the radar ) there's
no question that there was this intense period of 
development and 'new' things were happening at
a fast rate. As more and more developers got interested
in getting 2.0 cranked out the (limited) resources all
got eaten up in the 2.0 development cycle and 1.3
development virtually shut down. It was even 'officially'
shut down long before 2.0 was ready ( 1.3 officially went
into maintenance mode only ).

So now you had lots of legacy developers ( albeit... lots
of weekend-warriors, too, but WW's are the heartbeat
of Open Source ) who knew 1.3 very well but were now
totally put out to pasture. Very few of them 'came over'
to the 2.0 development. The majority of the developers
for 2.0 were the 'paid to play' kind that were either being
paid to directly work on Apache ( I'll get to the Covalent
factor in a second ) or,  at least, were being paid to
do something else but no one seemed to mind if they
sat there at their real jobs all day and did Apache
development. They might not admit to being directly
paid to work on Apache but when that's what you are
doing all day and still bringing home a paycheck then
that's exactly what that is.

The 'other' not-so-dedicated-but-certainly-interested 
developers felt 'shut out' of the 2.0 development cycle
because it was obvious a lot of it was taking place
'off line' and nothing was being documented so they
couldn't really get a good handle on what was going
on in order to make a contribution.

So they were mostly 'waiting in the wings' for a lot of
the 2.0 DESIGN level decisions and concepts to 
become clear so they could 'jump in'.

Well... we all know that it was a looong time before
some of the 2.0 design concepts became 'clear' enough
for a not-paid-to-work-on-apache person to 'jump in'.

There were basically only 2 big changes to Apache 2.0
versus 1.x.

1. MPM. Get out of the 'prefork' only 'go fork thyself' design model.

2. ( Came later ) Add some I/O filtering and get rid of BUFF.

Well, these are both fine goals to have but they are not for
the faint-of-heart ( or within the grasp of most weekend-warriors ).

That brings me to "The Covalent Factor".

Is anyone going to deny that at a certain point in the 2.0
development cycle the PRIMARY driving force was a 
group of people that ALL worked for one company? ( Covalent ).

That's certainly the way it looked.

...and these guys were crankin'... I mean they were
MOTIVATED, because Covalent's entire corporate
life revolves around Apache and they were actually 
high level deals going on with Compaq ( and other
corporate entities? ) that all revolved around getting
Apache 2.0 out the door.

At one point one of the Covalent guys ( I believe it
was that Randy Bloom fellow? ) was pretty much the
ONLY person who had any idea how the new 'filtering'
was even SUPPOSED to work. It was quite some time
before he even finished thinking it through and it went
through (too?) many re-works to even keep a good
grasp on it.

I remember the feeling for MONTHS was something like...
"let's just see what Covalent comes up with and let them
put their stamp of approval on it and then we'll see if
we understand it."

...and that's pretty much how the entire SECOND primary
goal of Apache 2.0 was achieved. Covalent just did it
and expected everyone else to 'go along'... and they did.

The point of all this is not to FLAME anyone or any corporate
entity. No one can deny that Covalent has been crucial to 
the life and times ( and success ) of Apache itself and they
should be proud of the contribution(s) they have made to
OSS. Covalent was started by one of the original Apache guys 
and some of the best developers left around here still work for 
Covalent. I think William Rowe still does, for example, and he's 
still pretty much the only human being who seems to understand 
the Windows version of Apache... but the legacy flood of 'covalent' 
email addresses showing up at httpd-dev has virtually
vanished so it's hard to tell what what with that anymore.

There is nothing in the rules of Open Source that says you
can't crank up a company that makes money off of Open
Source software. Indeed... if you look at almost ALL of the
major Open Source projects you will see the same sort
of 'Covalent' deal going on whereby the people that know
that OSS best have found a way to get paid to work on it.

My only real point here ( and the way all this relates to the
current thread ) is that maybe it's time to acknowledge that
what is happening now is what will always happen to a major
development project if you let too many of your eggs go into
the same 'corporate' basket.

If nothing else comes out of this thread maybe it should
just be a resolve to never let any future OSS development cycles
have 'too many eggs in the same basket'.

Corporations are motivated by contracts, deadlines, promises
and $$$$. Their 'motivations' and goals can co-exist with
OSS goals but it can lead to situations where TOO MANY of
the primary developers all end up working for the same
company and OSS project goals and one-company's 
corporate goals can get too entangled to unravel.

I think that pretty much already happened at httpd-dev
right under everyone's noses.

When Apache 2.0 was first released... it was a disaster.

The person who called for the release 'out of nowhere'
worked for Covalent... and within just a few hours he had
the required 2 or 3 'yes' votes to initiate the release... and
2 of those 3 votes were also from Covalent employees.

I think it can be safely said that 'Covalent' initiated/approved the
release of Apache 2.0 and they did it so fast ( In just one
day ) and without any chance for discussion that you have
to wonder to this day whether they all just walked out of
a meeting or something having 'decided' to do it because that's what
fit their contract schedule(s). Someone or something was
waiting for 2.0 to just be offically out of 'Beta' no matter
WHAT shape it was actually in.

DISCLAIMER: I don't know any of this for a fact. This is
just the way it 'looked' from simply following (public) 
httpd-dev mail messages. I have no idea what really 
went/goes on at Covalent.

I think even the Covalent developers who initiated the
release were suprised at how 'not ready' it really was.
Very little had even been tested and it was simply not
even usuable for quite some time.

There was a lot of major functionality that wasn't even
tested for the FIRST time until AFTER the freakin'
thing was officially released... and some (crucial) features
weren't even close to working at all.

This created a real feeling of disappoinment throughout
the entire Apache community... developers and users
alike. After all that hard work... it went out the door
with a whimper and was actually nothing short of

The silence over that whole 'not-ready-for-prime-time'
too-early release of Apache 2.0 was DEAFENING.

You could have heard a pin drop at httpd-dev for 
quite a while. I think people were just in shock or
too digusted to even say anything. Then the horror
set in and people just got down to work solving 
the problems... but for a while there it looked like
the httpd-dev list just came to grinding halt.

I think you lost a lot people ( developers included ) right then
and there.

There was then another flurry of activity to get the ( now
released ) Apache 2.0 into a state where you could even
call it usable... and after that is when the whole deal 
seemed to just put the brakes on.

I don't know if it was because of disgust over the whole way
it went down, or development burnout, or just that one
particular company that had an enormous amount of 
control ( too much ? ) over the product simply 'got what
it wanted'... but it sure looks like a lot of people have
just 'walked away' from Apache 2.0.

I mean... where is Covalent now? Where are all those guys
who had so much influence on what this Server has now
become and who knew it so well? Have they all
just vanished? Why aren't they helping to FINISH what
they so enthusiastically STARTED?

There are still messages floating by on httpd-dev even as
we speak regarding the 'filtering' and it seems like you are
hard-pressed to find anyone left at httpd-dev who REALLY 
understands it even now. You see a lot of "I think it works
like this" and "maybe we forgot to do this" and "have to
go and see how that works" responses but rarely anything
concrete. It's weird.

There are STILL some legacy 1.3 developers who don't
have their heads around MPM and BUCKET BRIGADES 
well enough to 'come back' and join the party again.

You might even consider re-opening the 1.3 tree for 
development again. I would. I never understood why it
was 'closed' in the first place. I find it hard to believe that
one of the most influenial OSS projects in the world 
wouldn't be able to develop a new version in the 
background and still keep the development of the
current version alive. Covalent wasn't interested in
seeing 1.3 advance but that shouldn't have made
any difference. 1.3  development should have stayed 
'open' to 'keep the fires burning'.

Reality would also dicatate keeping the 1.3 development
'alive' for patches and improvements.

Even now... almost 2 years after it was released... there
are VERY few people even interested in Apache 2.0
and what it has to offer. I will bet you that some 80 percent
of your 'customers' wouldn't have the faintest idea what
MPM means or what replacing BUFF means. They
just wouldn't get it.

Everyone understands SPEED, of course, but despite
a few postings in this regard on this thread itself there
isn't much concrete information out there about whether
Apache 2.0 really is faster. I will bet that if you just run
it in good 'ole pre-fork and simply add a filter or two 
that it is either just the same as 1.3 or even SLOWER.

Your high-end, high-dollar, high-transaction customers 
are, of course, going to understand what is different
about Apache 2.0 and why they should give it a shot
but that's still going to just be a fraction of your 
actual user-base.

The only way, at this point, to get the majority of your
customer base to switch to Apache 2.0 and help
iron out all the remaining bugs is if some serious 
security issue arises with 1.3 and you guys REFUSE
to add a patch to 1.3. That would FORCE people 
who don't even care about it to start using 2.0.

Sounds a little nasty... but that's what you could do
and, indeed, might be the ONLY way to crank up 
the usage on 2.0 and... as a consequence... the
development and improvement of it.

Go ahead and shut down even 'maintenance' mode
on 1.3 ( No more patches of any kind, security included )
and just sit back and wait. When it comes... you will
hardly have the bandwidth to handle the number of
people that are 'finally' switching to 2.0.

The reason things have 'slowed down' is because there
isn't any 'jazz' at the moment. If no one is using 2.0
and discovering new bugs that need to be squashed
and everyone seems happy with 1.3... then where's 
the problem? It's time to go lie on the beach and 
be proud of your success.

You have 2 Servers that seem to work fine and no
one is even asking for anything 'new' other than 
minor tweaks here and there. Life is good (??).

It's like being a fireman but nothing is catching fire...

...but even most fireman don't take the job to sit around
the station and eat pizza and watch soap operas... they
joined up to put out fires.

Software development is about problem solving. That's
the 'jazz' that most people get out of it. It's that feeling
that you got your head around something that most 
people would never understand and maybe noticed
something that no one else has noticed that makes
guys hang drywall for 5 days a week and then post
patches to OSS projects on the weekend. It's fun. It 
makes you forget for a while that you actually do 
just hang drywall most of the time.

So when any development cycle 'slows down' the
first question to ask is...

1. Where are the problems that need to be solved?

I don't think most people perceive the 'problems' 
as intently anymore. Apache 2.0 is out there and has
been freely available for quite some time to
anyone who is interested in using
it ( apparently not many ) and anyone who is actually
using 2.0 doesn't seem to be complaining about it much... 
and Apache 1.3 development is permanently dead.

So where's the fire?


View raw message