hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)
Date Tue, 03 Nov 2015 18:31:51 GMT
Maybe it got reenabled. I've disabled it now. Thanks Sean.
St.Ack

On Tue, Nov 3, 2015 at 9:50 AM, Sean Busbey <busbey@cloudera.com> wrote:

> I thought HBase_TRUNK was already disabled. if I missed it, disabling
> should be sufficient. There are a few older non-matrix (0.98 I think?)
> that we can delete once we're confident in the matrix version
>
> On Tue, Nov 3, 2015 at 8:44 AM, Stack <stack@duboce.net> wrote:
> > Stuff seems to be basically working.  I just committed a fix for this
> > message so it should be gone now:
> >
> > grep: unknown devices method
> > grep: write error: Broken pipe
> >
> >
> > Hopefully we've seen last of the errant surefire kills.
> >
> > We have an HBASE_TRUNK build and we have an HBASE_TRUNK_MATRIX. We should
> > get rid of the former now?
> >
> > St.Ack
> >
> > On Mon, Nov 2, 2015 at 11:03 PM, Stack <stack@duboce.net> wrote:
> >
> >> I just did an edit of all of our jenkins configuration. I changed the
> >> post-build zombie section. The surefire-killer was hanging out here it
> >> seems (For detail, see HBASE-14589 Looking for the surefire-killer;
> builds
> >> being killed...). I may have messed up builds. Will fix in morning if so
> >> (currently builds seem a bit backed up so won't wait around).
> >>
> >> St.Ack
> >>
> >> On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <busbey@cloudera.com>
> wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS]
> >>> reducing our load on builds.a.o' on reasoning or if you have concerns.
> >>>
> >>> Our project job view now has a brief summary of what's going on:
> >>>
> >>> https://builds.apache.org/view/H-L/view/HBase/
> >>>
> >>> * Where we had matrix builds, I confirmed they're doing the same thing
> >>> (in at least one test-configuration) as the non-matrix version
> >>> * I disabled jobs that were duplicating work done in matrix jobs, or
> >>> that only worked on a tag
> >>> * Made sure all non-disabled 0.98+ jobs are set to notify jira +
> >>> builds mailing list
> >>> * I updated pre-1.1 jobs to look for changes 1/day
> >>> * I updated 1.1 jobs to look for changes every 8 hours
> >>> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours
> >>>
> >>> There's still some non-pressing clean up work to do, so I'll file a
> >>> jira later today just to make sure it's all recorded somewhere.
> >>>
> >>> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <busbey@cloudera.com>
> >>> wrote:
> >>> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2
> >>> builds to
> >>> > use branch-1.2. (ref HBASE-13912)
> >>> >
> >>> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <busbey@cloudera.com>
> >>> wrote:
> >>> >>
> >>> >> I've add a new build test to run the ITs under sunny day conditions
> >>> using
> >>> >> minicluster for both java 7 and java 8 on the 1.2 line.
> >>> >>
> >>> >> https://builds.apache.org/job/HBase-1.2-IT/
> >>> >>
> >>> >> I haven't turned on notifications yet, because BulkIngest and
> >>> TestIngest
> >>> >> are read. details on HBASE-13750
> >>> >>
> >>> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <busbey@cloudera.com>
> >>> wrote:
> >>> >>>
> >>> >>> sigh. that should have been "is now a matrix build".
> >>> >>>
> >>> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <busbey@cloudera.com>
> >>> wrote:
> >>> >>>>
> >>> >>>> The HBase-1.2 build[1] is not a matrix build that runs
JDK7 and
> JDK8
> >>> in
> >>> >>>> parallel.
> >>> >>>>
> >>> >>>> I saved the old job for now and left it disabled[2].
> >>> >>>>
> >>> >>>>
> >>> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
> >>> >>>> [2]:
> >>> >>>>
> >>>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
> >>> >>>>
> >>> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <busbey@cloudera.com
> >
> >>> >>>> wrote:
> >>> >>>>>
> >>> >>>>> Sorry for the noise. I also updated the checkstyle
step on
> >>> HBase-TRUNK
> >>> >>>>>
> >>> >>>>> from
> >>> >>>>>     mvn checkstyle:checkstyle
> >>> >>>>> to
> >>> >>>>>     mvn -DskipTests package checkstyle:checkstyle
> >>> >>>>>
> >>> >>>>> to deal with the same issue. Looks all clear now.
> >>> >>>>>
> >>> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <
> busbey@cloudera.com>
> >>> >>>>> wrote:
> >>> >>>>>>
> >>> >>>>>> Updated the following builds:
> >>> >>>>>>
> >>> >>>>>> * HBase-TRUNK
> >>> >>>>>>
> >>> >>>>>> moved from
> >>> >>>>>>
> >>> >>>>>>     mvn clean compile findbugs:findbugs
> >>> >>>>>>
> >>> >>>>>> to
> >>> >>>>>>
> >>> >>>>>>    mvn clean -DskipTests package findbugs:findbugs
> >>> >>>>>>
> >>> >>>>>> To work around the known issue where we can't do
a bootstrap
> >>> compile
> >>> >>>>>> without previous install or remote SNAPSHOT artifacts.
(recently
> >>> triggered
> >>> >>>>>> by the addition of the procedure module on master)
> >>> >>>>>>
> >>> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino219@gmail.com>
> wrote:
> >>> >>>>>>>
> >>> >>>>>>> Make findbugs and checkstyle plugins always
run.
> >>> >>>>>>> The default behavior only publishes result
for stable builds,
> but
> >>> >>>>>>> master is
> >>> >>>>>>> not always stable and sometimes we introduce
new warnings in
> >>> unstable
> >>> >>>>>>> builds(see builds 6310-6314).
> >>> >>>>>>> And findbugs and checkstyle will not fail unless
we abort the
> >>> >>>>>>> building
> >>> >>>>>>> process I think, so it is safe to turn it on
every time.
> >>> >>>>>>>
> >>> >>>>>>> Thanks.
> >>> >>>>>>>
> >>> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> >>> >>>>>>>
> >>> >>>>>>> > +1 on letting HBase-TRUNK jenkins show
coverage report.
> >>> >>>>>>> >
> >>> >>>>>>> > Cheers
> >>> >>>>>>> >
> >>> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎
<palomino219@gmail.com>
> >>> wrote:
> >>> >>>>>>> >
> >>> >>>>>>> > > Going to change the config of HBase-TRUNK
jenkins to show
> >>> >>>>>>> > > findbugs,
> >>> >>>>>>> > > checkstyle and jacoco coverage report.
> >>> >>>>>>> > > The config has been tested on
> >>> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco
for
> nearly
> >>> 30
> >>> >>>>>>> > > times.
> >>> >>>>>>> > Not
> >>> >>>>>>> > > much different from HBase-TRUNK unless
it runs ~30%
> slower(the
> >>> >>>>>>> > > overhead
> >>> >>>>>>> > of
> >>> >>>>>>> > > collecting information for code coverage).
> >>> >>>>>>> > > Thanks.
> >>> >>>>>>> > >
> >>> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew
Purtell <
> apurtell@apache.org
> >>> >:
> >>> >>>>>>> > >
> >>> >>>>>>> > > > +1, thanks a lot for improving
our build hygiene.
> >>> >>>>>>> > > >
> >>> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01
PM, Enis Söztutar
> >>> >>>>>>> > > > <enis.soz@gmail.com>
> >>> >>>>>>> > > wrote:
> >>> >>>>>>> > > >
> >>> >>>>>>> > > > > Thanks Sean. This is great.
> >>> >>>>>>> > > > >
> >>> >>>>>>> > > > > Enis
> >>> >>>>>>> > > > >
> >>> >>>>>>> > > > > On Wed, Mar 11, 2015 at
1:54 PM, Sean Busbey
> >>> >>>>>>> > > > > <busbey@cloudera.com>
> >>> >>>>>>> > > > wrote:
> >>> >>>>>>> > > > >
> >>> >>>>>>> > > > > > FYI I just finished
chasing down the breakage for mvn
> >>> site
> >>> >>>>>>> > > > > > on all
> >>> >>>>>>> > > patch
> >>> >>>>>>> > > > > > builds.
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > HBASE-13191 consolidates
the few places in the
> >>> test-patch
> >>> >>>>>>> > > > > > code
> >>> >>>>>>> > where
> >>> >>>>>>> > > we
> >>> >>>>>>> > > > > > hard coded MAVEN_OPTS.
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > If you look at the
PreCommit job now, we use the "set
> >>> >>>>>>> > > > > > environment
> >>> >>>>>>> > > > > > variables" option
to set MAVEN_OPTS and then
> everything
> >>> >>>>>>> > > > > > else
> >>> >>>>>>> > respects
> >>> >>>>>>> > > > > that
> >>> >>>>>>> > > > > > setting.
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > I set the initial
value to be a combination of the
> >>> memory
> >>> >>>>>>> > limitations
> >>> >>>>>>> > > > > we've
> >>> >>>>>>> > > > > > been actually running
with (the ~6G was getting
> ignored)
> >>> >>>>>>> > > > > > and the
> >>> >>>>>>> > > > permgen
> >>> >>>>>>> > > > > > needed for site.
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M
-XX:-UsePerfData
> >>> -XX:MaxPermSize=256m
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > On Mon, Feb 23, 2015
at 11:46 PM, Stack <
> >>> stack@duboce.net>
> >>> >>>>>>> > > > > > wrote:
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > > I just made TRUNK
and branch-1 builds use same jvm
> as
> >>> >>>>>>> > > > > > > patch-build
> >>> >>>>>>> > > > > > > (hadoopqa) --
i.e. jdku51 -- and I set the
> MAVEN_OPTS
> >>> to
> >>> >>>>>>> > > > > > > be the
> >>> >>>>>>> > > same
> >>> >>>>>>> > > > as
> >>> >>>>>>> > > > > > > those of trunk
build too, setting
> >>> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"...
it
> >>> >>>>>>> > had
> >>> >>>>>>> > > > been
> >>> >>>>>>> > > > > > > 3000.
> >>> >>>>>>> > > > > > >
> >>> >>>>>>> > > > > > > Yours,
> >>> >>>>>>> > > > > > > St.Ack
> >>> >>>>>>> > > > > > >
> >>> >>>>>>> > > > > > > On Wed, Dec 31,
2014 at 4:22 PM, Stack <
> >>> stack@duboce.net>
> >>> >>>>>>> > > > > > > wrote:
> >>> >>>>>>> > > > > > >
> >>> >>>>>>> > > > > > > > I upped
hadoopqa retention to keep last 100
> builds
> >>> and
> >>> >>>>>>> > > > > > > > or last
> >>> >>>>>>> > 7
> >>> >>>>>>> > > > > days,
> >>> >>>>>>> > > > > > > > whichever
comes first.
> >>> >>>>>>> > > > > > > > St.Ack
> >>> >>>>>>> > > > > > > >
> >>> >>>>>>> > > > > > > > On Tue,
Nov 4, 2014 at 9:38 AM, Stack
> >>> >>>>>>> > > > > > > > <stack@duboce.net>
> >>> >>>>>>> > wrote:
> >>> >>>>>>> > > > > > > >
> >>> >>>>>>> > > > > > > >> Branch-1
and master have stabilized and now run
> >>> mostly
> >>> >>>>>>> > > > > > > >> blue
> >>> >>>>>>> > > (give
> >>> >>>>>>> > > > or
> >>> >>>>>>> > > > > > > take
> >>> >>>>>>> > > > > > > >> the
odd failure) [1][2]. Having a mostly blue
> >>> branch-1
> >>> >>>>>>> > > > > > > >> has
> >>> >>>>>>> > > helped
> >>> >>>>>>> > > > us
> >>> >>>>>>> > > > > > > >> identify
at least one destabilizing commit in
> the
> >>> last
> >>> >>>>>>> > > > > > > >> few
> >>> >>>>>>> > days,
> >>> >>>>>>> > > > > maybe
> >>> >>>>>>> > > > > > > two;
> >>> >>>>>>> > > > > > > >> this
is as it should be (smile).
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >> Lets
keep our builds blue. If you commit a
> patch,
> >>> make
> >>> >>>>>>> > > > > > > >> sure
> >>> >>>>>>> > > > > subsequent
> >>> >>>>>>> > > > > > > >> builds
stay blue. You can subscribe to
> >>> >>>>>>> > builds@hbase.apache.org
> >>> >>>>>>> > > to
> >>> >>>>>>> > > > > get
> >>> >>>>>>> > > > > > > >> notice
of failures if not already subscribed.
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >> Thanks,
> >>> >>>>>>> > > > > > > >> St.Ack
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >> 1.
> >>> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> >>> >>>>>>> > > > > > > >> 2.
> >>> >>>>>>> > >
> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >> On Mon,
Oct 13, 2014 at 4:41 PM, Stack
> >>> >>>>>>> > > > > > > >> <stack@duboce.net>
> >>> >>>>>>> > > wrote:
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >>>
A few notes on testing.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
Too long to read, infra is more capable now and
> >>> after
> >>> >>>>>>> > > > > > > >>>
some
> >>> >>>>>>> > > work,
> >>> >>>>>>> > > > we
> >>> >>>>>>> > > > > > are
> >>> >>>>>>> > > > > > > >>>
seeing branch-1 and trunk mostly running blue.
> >>> Lets
> >>> >>>>>>> > > > > > > >>>
try and
> >>> >>>>>>> > > keep
> >>> >>>>>>> > > > it
> >>> >>>>>>> > > > > > > this
> >>> >>>>>>> > > > > > > >>>
way going forward.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
Apache Infra has new, more capable hardware.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
A recent spurt of test fixing combined with
> more
> >>> >>>>>>> > > > > > > >>>
capable
> >>> >>>>>>> > > hardware
> >>> >>>>>>> > > > > > seems
> >>> >>>>>>> > > > > > > >>>
to have gotten us to a new place; tests are
> mostly
> >>> >>>>>>> > > > > > > >>>
passing
> >>> >>>>>>> > now
> >>> >>>>>>> > > on
> >>> >>>>>>> > > > > > > branch-1
> >>> >>>>>>> > > > > > > >>>
and master.  Lets try and keep it this way and
> >>> start
> >>> >>>>>>> > > > > > > >>>
to trust
> >>> >>>>>>> > > our
> >>> >>>>>>> > > > > > test
> >>> >>>>>>> > > > > > > runs
> >>> >>>>>>> > > > > > > >>>
again.  Just a few flakies remain.  Lets try
> and
> >>> nail
> >>> >>>>>>> > > > > > > >>>
them.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
Our tests now run in parallel with other test
> >>> suites
> >>> >>>>>>> > > > > > > >>>
where
> >>> >>>>>>> > > > previous
> >>> >>>>>>> > > > > > we
> >>> >>>>>>> > > > > > > >>>
ran alone. You can see this sometimes when our
> >>> zombie
> >>> >>>>>>> > detector
> >>> >>>>>>> > > > > > reports
> >>> >>>>>>> > > > > > > >>>
tests from another project altogether as
> lingerers
> >>> >>>>>>> > > > > > > >>>
(To be
> >>> >>>>>>> > > fixed).
> >>> >>>>>>> > > > > > > Some of
> >>> >>>>>>> > > > > > > >>>
our tests are failing because a concurrent
> hbase
> >>> run
> >>> >>>>>>> > > > > > > >>>
is
> >>> >>>>>>> > undoing
> >>> >>>>>>> > > > > > > classes and
> >>> >>>>>>> > > > > > > >>>
data from under it. Also, lets fix.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
Our tests are brittle. It takes 75minutes for
> >>> them to
> >>> >>>>>>> > complete.
> >>> >>>>>>> > > > > Many
> >>> >>>>>>> > > > > > > >>>
are heavy-duty integration tests starting up
> >>> multiple
> >>> >>>>>>> > clusters
> >>> >>>>>>> > > > and
> >>> >>>>>>> > > > > > > >>>
mapreduce all in the one JVM. It is a miracle
> they
> >>> >>>>>>> > > > > > > >>>
pass at
> >>> >>>>>>> > all.
> >>> >>>>>>> > > > > > > Usually
> >>> >>>>>>> > > > > > > >>>
integration tests have been cast as unit tests
> >>> >>>>>>> > > > > > > >>>
because there
> >>> >>>>>>> > > was
> >>> >>>>>>> > > > no
> >>> >>>>>>> > > > > > > where
> >>> >>>>>>> > > > > > > >>>
else for them to get an airing.  We have the
> >>> hbase-it
> >>> >>>>>>> > > > > > > >>>
suite
> >>> >>>>>>> > now
> >>> >>>>>>> > > > > which
> >>> >>>>>>> > > > > > > would
> >>> >>>>>>> > > > > > > >>>
be a more apt place but until these are run on
> a
> >>> >>>>>>> > > > > > > >>>
regular
> >>> >>>>>>> > basis
> >>> >>>>>>> > > in
> >>> >>>>>>> > > > > > > public
> >>> >>>>>>> > > > > > > >>>
for all to see, the fat integration tests
> >>> disguised
> >>> >>>>>>> > > > > > > >>>
as unit
> >>> >>>>>>> > > tests
> >>> >>>>>>> > > > > > will
> >>> >>>>>>> > > > > > > >>>
remain.  A review of our current unit tests
> >>> weeding
> >>> >>>>>>> > > > > > > >>>
the old
> >>> >>>>>>> > > cruft
> >>> >>>>>>> > > > > and
> >>> >>>>>>> > > > > > > the
> >>> >>>>>>> > > > > > > >>>
no longer relevant or duplicates would be a
> nice
> >>> >>>>>>> > > > > > > >>>
undertaking
> >>> >>>>>>> > if
> >>> >>>>>>> > > > > > > someone is
> >>> >>>>>>> > > > > > > >>>
looking to contribute.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
Alex Newman has been working on making our
> tests
> >>> work
> >>> >>>>>>> > > > > > > >>>
up on
> >>> >>>>>>> > > > travis
> >>> >>>>>>> > > > > > and
> >>> >>>>>>> > > > > > > >>>
circle-ci.  That'll be sweet when it goes
> >>> end-to-end.
> >>> >>>>>>> > > > > > > >>>
He
> >>> >>>>>>> > also
> >>> >>>>>>> > > > > added
> >>> >>>>>>> > > > > > in
> >>> >>>>>>> > > > > > > >>>
some "type" categorizations -- client, filter,
> >>> >>>>>>> > > > > > > >>>
mapreduce --
> >>> >>>>>>> > > > > alongside
> >>> >>>>>>> > > > > > > our
> >>> >>>>>>> > > > > > > >>>
old "sizing" categorizations of
> >>> small/medium/large.
> >>> >>>>>>> > > > > > > >>>
His
> >>> >>>>>>> > > thinking
> >>> >>>>>>> > > > > is
> >>> >>>>>>> > > > > > > that
> >>> >>>>>>> > > > > > > >>>
we can run these categorizations in parallel
> so we
> >>> >>>>>>> > > > > > > >>>
could run
> >>> >>>>>>> > > the
> >>> >>>>>>> > > > > > total
> >>> >>>>>>> > > > > > > >>>
suite in about the time of the longest test,
> say
> >>> >>>>>>> > 20-30minutes?
> >>> >>>>>>> > > > We
> >>> >>>>>>> > > > > > > could
> >>> >>>>>>> > > > > > > >>>
even change Apache to run them this way.
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
FYI,
> >>> >>>>>>> > > > > > > >>>
St.Ack
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>>
> >>> >>>>>>> > > > > > > >>
> >>> >>>>>>> > > > > > > >
> >>> >>>>>>> > > > > > >
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > > > --
> >>> >>>>>>> > > > > > Sean
> >>> >>>>>>> > > > > >
> >>> >>>>>>> > > > >
> >>> >>>>>>> > > >
> >>> >>>>>>> > > >
> >>> >>>>>>> > > >
> >>> >>>>>>> > > > --
> >>> >>>>>>> > > > Best regards,
> >>> >>>>>>> > > >
> >>> >>>>>>> > > >    - Andy
> >>> >>>>>>> > > >
> >>> >>>>>>> > > > Problems worthy of attack prove
their worth by hitting
> >>> back. -
> >>> >>>>>>> > > > Piet
> >>> >>>>>>> > Hein
> >>> >>>>>>> > > > (via Tom White)
> >>> >>>>>>> > > >
> >>> >>>>>>> > >
> >>> >>>>>>> >
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> --
> >>> >>>>>> Sean
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>> Sean
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> --
> >>> >>>> Sean
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Sean
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Sean
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Sean
> >>>
> >>>
> >>>
> >>> --
> >>> Sean
> >>>
> >>
> >>
>
>
>
> --
> Sean
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message