Return-Path: X-Original-To: apmail-cordova-dev-archive@www.apache.org Delivered-To: apmail-cordova-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 153CEDD34 for ; Thu, 3 Jan 2013 04:05:40 +0000 (UTC) Received: (qmail 23277 invoked by uid 500); 3 Jan 2013 04:05:39 -0000 Delivered-To: apmail-cordova-dev-archive@cordova.apache.org Received: (qmail 22998 invoked by uid 500); 3 Jan 2013 04:05:38 -0000 Mailing-List: contact dev-help@cordova.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cordova.apache.org Delivered-To: mailing list dev@cordova.apache.org Received: (qmail 22972 invoked by uid 99); 3 Jan 2013 04:05:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 04:05:37 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FRT_ADOBE2,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of mmocny@google.com designates 209.85.210.42 as permitted sender) Received: from [209.85.210.42] (HELO mail-da0-f42.google.com) (209.85.210.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2013 04:05:32 +0000 Received: by mail-da0-f42.google.com with SMTP id z17so6805091dal.29 for ; Wed, 02 Jan 2013 20:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=uWXzlNw4ISxYKsaJn0eiGhY+7tvNKR9ncAL38lIURKM=; b=Ja3j1aMpJBEPfuIpzILKsFUIachqoVOdf+iOfXSNPooK24rMHBVgrRzh48AIK8OZOS r3OJs0gHLFvrgFpuXA2HBbnzt46eE4GXkPIl/xi/T+KnL6hMXu2Uf+jtvVIvt2RWb5w1 b7UzpaYvHXthsbilbie0NblThc0yNS0mGQRmjCNHTv3h/5N2LdBTn/eif/dFmq73Ik8y 9zGz/VFOWt1+WeuUv/bGMDZsZEqf87ikSiA99DM/u0BVrfDahNExGdwWgLEDNCs5482n pZuquiGbnnZPqt4De4sJYSKFpmLnsL68dD4tBYJJflL8NPWDf5qT4Ed+zRrhCMU6k+tW c1HQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=uWXzlNw4ISxYKsaJn0eiGhY+7tvNKR9ncAL38lIURKM=; b=C7HONz8SESqSBn+94+/dQGV3bBnRCL4qKelhGasqeQ3FzXuIdv9v0xgtKeY2AmIHea jZkFZ+7yW8APAe2pQUI/Es/X60cGDyndzR7gpbwJro5Fj9n2uF3fgZN8jHHNZ/6aZoRv qWQFAuHQvC1B5giuIX35/0AtcPgLor7IPdnCc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :x-gm-message-state; bh=uWXzlNw4ISxYKsaJn0eiGhY+7tvNKR9ncAL38lIURKM=; b=PSBj3kKFDBQrow0lTVxvgsHi5jhU2RAY69xQAQ1htK0c5Rz1xhyzJog+EFy0qE9mlI s9+31YPRTH2JLoxfEzCokuGxJhNg/tq8uNjWEYnVoUZgHl44JyW7T5o/YboRuFSCRMrK Xr4Y6C4UGRzvzlU3r2IC/KXh6dUpXKP9ZWJhWOE+p9nTp2g+zvls/yRYonlT8xYCmpoC jewJjILRjO0fKM1kf5tLDrFTChxBMaBKLvdGjoNuNNuZlcdR6wIwX9YpXL2ZHoKZlakf Ppu+DVORjUJ/b0euL3oHq2rC1Yu1vClWJvXMHUvsmrmx8TARGmZPV/f1nEV+Uhg3MMpE Vdxg== MIME-Version: 1.0 Received: by 10.68.132.232 with SMTP id ox8mr149686981pbb.46.1357185912293; Wed, 02 Jan 2013 20:05:12 -0800 (PST) Sender: mmocny@google.com Received: by 10.68.220.234 with HTTP; Wed, 2 Jan 2013 20:05:12 -0800 (PST) In-Reply-To: References: Date: Wed, 2 Jan 2013 23:05:12 -0500 X-Google-Sender-Auth: VGiFeTwr4cnBK1mOH7-fvfh9Dn0 Message-ID: Subject: Re: too long to package a release? From: Michal Mocny To: dev@cordova.apache.org Content-Type: multipart/alternative; boundary=047d7b10cbcb3249fd04d25a78c1 X-Gm-Message-State: ALoCoQl/s1Jl5rB4o1EJvy3mH8KWkukIuyLRZyEf6IzaWLeIwmWmeOognwY9EvIr4yyzoIMLnYZGBvZvHDUqVwSGCxGUdPHOzOdaFpBFQhFgV9xzBullOLoDaBErlViBFwn/nPQkdSH1qSvS13PFrPYZoR7tT3Vue7/efpjteSsIsssNt2ei9aVEF30SQ7sTUUdGmFN8Kv2y X-Virus-Checked: Checked by ClamAV on apache.org --047d7b10cbcb3249fd04d25a78c1 Content-Type: text/plain; charset=ISO-8859-1 On Wed, Jan 2, 2013 at 7:27 PM, Andrew Grieve wrote: > From my understanding of git, there's nothing special about the master > branch, except that it's what gets checked out when someone doesn't > explicitly say which branch they want. > > Joe pointed out that sometimes random people check out the code and expect > it to be stable. > Gord pointed out that people tend to submit pull requests assuming that > master == dev branch. > > I think choosing between these two data points, I'd lean towards having > people submit more useful pull requests. > Well put, agreed. > > I don't think either option affects how you'd go about doing a point > release. You would: > 1. Check out the relevant release tag (e.g. 2.3.0) > 2. Give the branch a name (e.g. branch_2.3.1) > I'm not sure why you would name the branch after checkout? Wouldn't you name it by tagging after doing the merges in step 3? > 3. Merge in all of the changes that you want to put in the point release > (ideally, these would already be committed in the dev branch, be it "next" > or "master"). > 4. Tag the release. > > > On Wed, Jan 2, 2013 at 5:11 PM, Gord Tanner wrote: > > > Also a problem we have encountered with using a 'next' branch for active > > development is from third party commits. > > > > Every single 3rd party pull request is going to come into master. > > > > You can ether: > > 1. tell them to redo it on next > > 2. rebase it into next for them and let them know for next time. > > > > The cost of option 2 gets more the longer it takes to release. A 3rd > party > > pull request coming in could be based on code that is 2+ months old. > > > > This isn't a vote against a development branch, but a small annoyance we > > have run into. > > > > > > On Wed, Jan 2, 2013 at 5:08 PM, Gord Tanner wrote: > > > > > A hot fix is usually branched off of master, tested, merged and > released. > > > > > > We then rebase / merge the hotfix up into next. > > > > > > > > > On Wed, Jan 2, 2013 at 4:51 PM, Michal Mocny > > wrote: > > > > > >> Also: while I personally prefer master to be the dev channel, I will > say > > >> that I do like Gord's suggestion of how its done in ripple in that the > > >> name > > >> of the dev branch is 'next' and not '2.3.0' so that your dev setup > > doesn't > > >> need to change every month. > > >> > > >> Gord: how do you deal with bugfixes/point releases? Do you fix in > > feature > > >> branch, merge into next, and then cherry-pick/merge just that fix into > > >> master before doing a major release? Or, do you just offer bugfixes > via > > >> 'next'? > > >> > > >> > > >> On Wed, Jan 2, 2013 at 4:45 PM, Michal Mocny > > wrote: > > >> > > >> > I don't have much weight here, but personally I feel that this seems > > >> > backwards. > > >> > > > >> > With this proposal (if I understand it), when you do a fresh > checkout > > of > > >> > the codebase, instead of sitting on the bleeding edge, you would be > > >> sitting > > >> > at a "stable" release which is conceptually read-only for most > > >> contributors > > >> > (writes happen in the form of batch "releases" which itself would > just > > >> be > > >> > some git-fu to rebase master). > > >> > > > >> > I am happy enough to have features be worked on in branches etc, I > > just > > >> > think that it should be flipped and the stable release be the branch > > and > > >> > dev to be on master. > > >> > > > >> > > > >> > As a separate issue, I would suggest not using branches to "name" > > point > > >> > releases, but just tag them. If you have a 2.3.0 release, and you > > need > > >> to > > >> > fix a bug in 2.3.1, those should not become two logically separate > > code > > >> > branches with independent dev, but rather they are a logically > single > > >> > timeline with many names for each historically significant commit, > > >> right? > > >> > Thats what tags are for ( > > http://git-scm.com/book/en/Git-Basics-Tagging > > >> ). > > >> > > > >> > -Michal > > >> > > > >> > > > >> > On Wed, Jan 2, 2013 at 3:05 PM, Gord Tanner > > wrote: > > >> > > > >> >> This is what we have done in ripple (and webworks) > > >> >> > > >> >> master - always stable current shipping code > > >> >> next - always 'stable' next release. Expectation that code has been > > >> tested > > >> >> / run before merged into this branch. > > >> >> feature branches - branched off of next and merged into next when > > >> stable / > > >> >> done. Not expected to be stable or runnable until merge time. > > >> >> > > >> >> > > >> >> > > >> >> On Wed, Jan 2, 2013 at 2:32 PM, Filip Maj wrote: > > >> >> > > >> >> > Am I correct when I say that, with this approach, master becomes > a > > >> >> series > > >> >> > of merge commits coming from dev, then ? > > >> >> > > > >> >> > A couple questions to follow up: > > >> >> > > > >> >> > - "features get forked from stable" - forked from master, yes? > > >> >> > - "features, when ready, tested against dev branch" - what does > > this > > >> >> mean? > > >> >> > Does this mean, you would merge feature branch into dev branch > > >> (locally) > > >> >> > then run tests to make sure things work? > > >> >> > > > >> >> > On 1/2/13 11:19 AM, "Joe Bowser" wrote: > > >> >> > > > >> >> > >OK, Let's rethink this: > > >> >> > > > > >> >> > >After talking with Brian on the 21st, I think we agree on this: > > >> >> > > > > >> >> > > * Master remains stable and sits at the most recent released > code > > >> >> > >(i.e. 2.3.0 once we get 2.3.0 done) (Stable Channel) > > >> >> > > * Dev happens on branches for the releases (i.e. 2.4.0) (Dev > > >> Channel) > > >> >> > > * In the case of a point release, dev happens in the branch of > > the > > >> >> > >major release (i.e. 2.3.1 would happen in the 2.3.0 branch, not > > >> >> > >master) (Testing Channel) > > >> >> > > * Features get forked on stable then once the feature is ready, > > >> >> > >tested against the dev branch. If they work with stable, they > > >> SHOULD > > >> >> > >work with 2.4.0. If they don't, the tickets get added to 2.4.0 > to > > >> >> > >make it work with that release. That way things are more > > >> predictable > > >> >> > >as far as new features are concerned. (You will burn your face > > >> >> > >channel). > > >> >> > > > > >> >> > >Does that make sense? Working on master for things causes us > pain > > >> and > > >> >> > >we should use git conventions to make it simpler for people who > > >> expect > > >> >> > >our master to work all the time. I don't think this will speed > up > > >> the > > >> >> > >release as much as automating tagging of RCs so that when the JS > > is > > >> >> > >tagged, everything else is tagged. The week it takes to tag an > RC > > >> is > > >> >> > >way too long. > > >> >> > > > > >> >> > > > > >> >> > >On Wed, Jan 2, 2013 at 11:08 AM, Filip Maj > wrote: > > >> >> > >> Bumping this thread. I'd like Joe to clarify as well. > > >> >> > >> > > >> >> > >> On 12/20/12 12:26 PM, "Brian LeRoux" wrote: > > >> >> > >> > > >> >> > >>>Ok, I want to understand this, let me take a stab. > > >> >> > >>> > > >> >> > >>>You describe three long-lived branches like this: > > >> >> > >>> > > >> >> > >>>- Master: This is stable and frozen on the last tagged > release. > > >> >> > >>>- Dev: the next release to be tagged. Feature branches merged > > from > > >> >> > >>>master when confident. > > >> >> > >>>- Unstable: the current working branch for a particular tag. > > >> Feature > > >> >> > >>>branches merged as needed for collaboration. > > >> >> > >>> > > >> >> > >>>Everyone works from local feature branch rebasing and > committing > > >> to > > >> >> > >>>master. When that feature branch is considered good enough, it > > is > > >> >> > >>>merged into dev, and work continues. Whatever date we happen > to > > >> pick > > >> >> > >>>for a release that is what dev becomes, we tag, and move that > > sha > > >> to > > >> >> > >>>stable if its not an RC. > > >> >> > >>> > > >> >> > >>>? > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > >>>On Thu, Dec 20, 2012 at 10:52 AM, Joe Bowser < > bowserj@gmail.com > > > > > >> >> wrote: > > >> >> > >>>> I'm OK with this, but I think your example is off: > > >> >> > >>>> > > >> >> > >>>> Where n is the current released piece of the software: > > >> >> > >>>> > > >> >> > >>>> n.x.x = Stable > > >> >> > >>>> n+1.x.x = Dev > > >> >> > >>>> master = Unstable, can have things merged in from feature > > >> branches > > >> >> > >>>> > > >> >> > >>>> This fully uncouples features from release planning, which > is > > >> good > > >> >> > >>>> because it means the release will land in the version when > > it's > > >> >> ready, > > >> >> > >>>> and not for any other reason. I also propose that we keep > > using > > >> >> the > > >> >> > >>>> same RC tags and that for a final release we tag it > > x.x.xFinal. > > >> We > > >> >> > >>>> still need to tag an RC and re-tag it. > > >> >> > >>>> > > >> >> > >>>> Release Process: > > >> >> > >>>> 1. Tag the dev tree > > >> >> > >>>> 2. merge the dev tree back into master > > >> >> > >>>> 3. Create 2.5.0 branch > > >> >> > >>>> 4. File issues from 2.5.0 in JIRA > > >> >> > >>>> > > >> >> > >>>> I also propose that we automate the tagging. If an RC is > > >> broken, > > >> >> we > > >> >> > >>>> just cut another RC. A lot of our retagging is done to get > > >> around > > >> >> the > > >> >> > >>>> pain of having to do another RC. The biggest part of the > > delay > > >> is > > >> >> > >>>> waiting for every single platform maintainer to tag their > > >> platform > > >> >> > >>>> after the JS was tagged. For example, I tagged rc2 for the > JS > > >> and > > >> >> for > > >> >> > >>>> Android on Monday last week from my hotel room, and the > > release > > >> >> wasn't > > >> >> > >>>> fully tagged until this week. I'm fine with RCs going up to > > 10 > > >> as > > >> >> > >>>> long as we can release early, release often and release when > > we > > >> >> want > > >> >> > >>>> to and not run out of time and have to delay. > > >> >> > >>>> > > >> >> > >>>> On Thu, Dec 20, 2012 at 10:33 AM, Brian LeRoux > > >> wrote: > > >> >> > >>>>> Truth. Though lets not get hung up on the past and just > focus > > >> on > > >> >> the > > >> >> > >>>>> present. We've done a really good job getting where we are. > > >> >> > >>>>> > > >> >> > >>>>> So, Joe, are you saying you like the idea of three long > lived > > >> >> > >>>>>branches > > >> >> > >>>>> and merges happen from local feature branches? > > >> >> > >>>>> > > >> >> > >>>>> > > >> >> > >>>>> On Thu, Dec 20, 2012 at 10:22 AM, Joe Bowser < > > >> bowserj@gmail.com> > > >> >> > >>>>>wrote: > > >> >> > >>>>>> We are totally doing something wrong with the way that we > do > > >> >> > >>>>>>releases. > > >> >> > >>>>>> I personally think that we're not using git right, and > > here's > > >> >> why: > > >> >> > >>>>>> > > >> >> > >>>>>> Currently, when we do a release, we tag the RC, and we > test > > >> the > > >> >> RC. > > >> >> > >>>>>> There's nothing preventing us from putting things after > that > > >> tag > > >> >> and > > >> >> > >>>>>> if we don't want to those things in the release branching > > off > > >> >> that > > >> >> > >>>>>> tag. We've done it before and other than the problem with > > >> CoHo, > > >> >> it > > >> >> > >>>>>> worked really well. I propose that instead of tagging the > > >> >> release, > > >> >> > >>>>>>we > > >> >> > >>>>>> branch when we want to do a release, and we do all the bug > > >> fixes > > >> >> on > > >> >> > >>>>>> that branch. Once that branch is ready to roll, we merge > it > > >> back > > >> >> > >>>>>>into > > >> >> > >>>>>> master. In fact, nobody should be working on master > except > > >> to do > > >> >> > >>>>>> merges. The way we're doing this now feels dirty and > wrong. > > >> >> > >>>>>> > > >> >> > >>>>>> I honestly feel that this is a much faster way of working, > > and > > >> >> that > > >> >> > >>>>>> we're missing the point if we have to tell everyone to > jump > > >> out > > >> >> of > > >> >> > >>>>>>the > > >> >> > >>>>>> pool every time we do an RC. I know that we could be > > working > > >> on > > >> >> our > > >> >> > >>>>>> branches, but that work is almost entirely invisible to > the > > >> rest > > >> >> of > > >> >> > >>>>>> the project until it's time to merge it back in, which > takes > > >> >> > >>>>>>forever. > > >> >> > >>>>>> > > >> >> > >>>>>> > > >> >> > >>>>>> On Thu, Dec 20, 2012 at 10:07 AM, Michal Mocny < > > >> >> mmocny@chromium.org > > >> >> > > > > >> >> > >>>>>>wrote: > > >> >> > >>>>>>> So there is something to be said about having devs shift > > >> focus > > >> >> from > > >> >> > >>>>>>>dev to > > >> >> > >>>>>>> testing during an RC. However, as the team grows, not > all > > >> of us > > >> >> > >>>>>>>are > > >> >> > >>>>>>>really > > >> >> > >>>>>>> being responsible for cutting releases. Maybe that means > > we > > >> >> need > > >> >> > >>>>>>>to > > >> >> > >>>>>>>train > > >> >> > >>>>>>> the entire team to change current behavior, but that > > doesn't > > >> >> feel > > >> >> > >>>>>>> necessary/scalable. > > >> >> > >>>>>>> > > >> >> > >>>>>>> With growing external contributions, I would have to say > > >> that a > > >> >> > >>>>>>>code > > >> >> > >>>>>>>freeze > > >> >> > >>>>>>> on trunk doesn't seem to make as much sense. > > >> >> > >>>>>>> > > >> >> > >>>>>>> -Michal > > >> >> > >>>>>>> > > >> >> > >>>>>>> > > >> >> > >>>>>>> On Thu, Dec 20, 2012 at 9:47 AM, Andrew Grieve > > >> >> > >>>>>>> wrote: > > >> >> > >>>>>>> > > >> >> > >>>>>>>> I definitely think we'd get more done if we didn't have > > >> such a > > >> >> > >>>>>>>>long > > >> >> > >>>>>>>> code-freeze. I'm not sure this is the same as what you > > were > > >> >> > >>>>>>>>suggesting, but > > >> >> > >>>>>>>> have a script/tool to branch all of the platforms into > an > > rc > > >> >> > >>>>>>>>branch. Then, > > >> >> > >>>>>>>> each platform can fix themselves up a bit and tag their > > RC. > > >> >> > >>>>>>>>Meanwhile, dev > > >> >> > >>>>>>>> can continue to happen on edge. > > >> >> > >>>>>>>> > > >> >> > >>>>>>>> My main concern with our current approach is just that > the > > >> >> > >>>>>>>>code-freeze time > > >> >> > >>>>>>>> is super long. > > >> >> > >>>>>>>> > > >> >> > >>>>>>>> > > >> >> > >>>>>>>> On Wed, Dec 19, 2012 at 3:36 PM, Marcel Kinard > > >> >> > >>>>>>>> > > >> >> > >>>>>>>>wrote: > > >> >> > >>>>>>>> > > >> >> > >>>>>>>> > One of the things that strikes me here is the > difference > > >> >> between > > >> >> > >>>>>>>>calendar > > >> >> > >>>>>>>> > time and effort time. (This assumes folks already > > >> concurred > > >> >> that > > >> >> > >>>>>>>>the rc > > >> >> > >>>>>>>> is > > >> >> > >>>>>>>> > ready to release.) Based on my reading of > > >> >> > >>>>>>>>http://wiki.apache.org/** > > >> >> > >>>>>>>> > cordova/CuttingReleases > > >> >> > >>>>>>>>there > > >> >> > >>>>>>>> isn't a lot of effort time involved to cut a release. It > > >> seems > > >> >> > >>>>>>>>like > > >> >> > >>>>>>>>a > > >> >> > >>>>>>>> > good chunk of the calendar time is getting folks to > tag > > >> their > > >> >> > >>>>>>>>platform. > > >> >> > >>>>>>>> > Ideally the promotion from rc to final should take > very > > >> >> little > > >> >> > >>>>>>>>effort > > >> >> > >>>>>>>> time. > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > What I like about the rc is that it provides a > settling > > >> >> > >>>>>>>>mechanism > > >> >> > >>>>>>>>for the > > >> >> > >>>>>>>> > churn to calm down, run tests across more integration, > > and > > >> >> see > > >> >> > >>>>>>>>the bigger > > >> >> > >>>>>>>> > picture to assess release readiness. I would expect > that > > >> the > > >> >> > >>>>>>>>promotion > > >> >> > >>>>>>>> from > > >> >> > >>>>>>>> > edge to rc should take a decent amount of effort time, > > but > > >> >> not > > >> >> > >>>>>>>>because of > > >> >> > >>>>>>>> > the "cut" activities. > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > So when we are at rc and don't find any surprises, why > > >> does > > >> >> it > > >> >> > >>>>>>>>take a > > >> >> > >>>>>>>> week > > >> >> > >>>>>>>> > to promote to final? If we spend a week in rc1, > another > > >> week > > >> >> in > > >> >> > >>>>>>>>rc2, and > > >> >> > >>>>>>>> > another week to cut final, that leaves only 1 week in > a > > >> >> 4-week > > >> >> > >>>>>>>>cycle for > > >> >> > >>>>>>>> > active dev work? > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > I like the ideal of a channel/stream/branch/whatever > > where > > >> >> there > > >> >> > >>>>>>>>is a > > >> >> > >>>>>>>> > place for the rc to settle without necessarily > blocking > > >> >> commits > > >> >> > >>>>>>>>to edge. > > >> >> > >>>>>>>> > Where I'm going with this is that if there is an area > > >> where > > >> >> > >>>>>>>>commits to > > >> >> > >>>>>>>> the > > >> >> > >>>>>>>> > rc are carefully controlled, then perhaps one person > > (i.e, > > >> >> Steve > > >> >> > >>>>>>>>G) could > > >> >> > >>>>>>>> > cut the release for ALL platforms using scripts. This > > may > > >> >> > >>>>>>>>involve > > >> >> > >>>>>>>>that > > >> >> > >>>>>>>> one > > >> >> > >>>>>>>> > person tagging/branching/whatever across multiple > > >> platforms. > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > I also like putting the "how to cut" magic in each > > >> platform. > > >> >> > >>>>>>>>Then > > >> >> > >>>>>>>>perhaps > > >> >> > >>>>>>>> > a good chunk of coho is tests to make sure that the > > >> platform > > >> >> > >>>>>>>>magic > > >> >> > >>>>>>>> > delivered the correct format to it. > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > -- Marcel Kinard > > >> >> > >>>>>>>> > > > >> >> > >>>>>>>> > > >> >> > >> > > >> >> > > > >> >> > > > >> >> > > >> > > > >> > > > >> > > > > > > > > > --047d7b10cbcb3249fd04d25a78c1--