Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0B59182B7 for ; Tue, 3 Nov 2015 14:44:53 +0000 (UTC) Received: (qmail 51392 invoked by uid 500); 3 Nov 2015 14:44:53 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 51302 invoked by uid 500); 3 Nov 2015 14:44:53 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 51264 invoked by uid 99); 3 Nov 2015 14:44:52 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2015 14:44:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 54F18C88FA for ; Tue, 3 Nov 2015 14:44:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 4V3xuQTZMdOy for ; Tue, 3 Nov 2015 14:44:44 +0000 (UTC) Received: from mail-io0-f179.google.com (mail-io0-f179.google.com [209.85.223.179]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id DB19342B91 for ; Tue, 3 Nov 2015 14:44:43 +0000 (UTC) Received: by iody8 with SMTP id y8so20552404iod.1 for ; Tue, 03 Nov 2015 06:44:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=mxhd6taW1Z7aqEIVtsSxDsFEzvjMqXsHWX9x+DWgoOU=; b=M0l1rcr5QVt7SEr5BFV/NgQB1K452iKKmbzl3VLJSSKhpd/PnA9xNmymWAZ5N8xpOG ljVNxg8Kidr8F6krIHKdhlVYwDL5SYhiNMq+AXZEduzLn6R+hpNYYyBBaXIcy0EOcBaW gctSypx6D/4+qTh4ZS+PEoPQP/Ua8ZliB76NSEUnMhahckQFlMC8oWgUcpJN0L9vZRlC PK4mjdDDL20uFfrG78YoXJCMbnVYY3GdhEkPR2i2oQjCY9bj/N5k5X+El3y++qtMV814 Wk7+zgJzQlGBooBAdq6dAWPeezhyG5vl540K4qfaasPHqQoEmKxP9q1KJnvxTDjmbw+U JkRA== MIME-Version: 1.0 X-Received: by 10.107.162.197 with SMTP id l188mr20041683ioe.57.1446561883458; Tue, 03 Nov 2015 06:44:43 -0800 (PST) Sender: saint.ack@gmail.com Received: by 10.64.146.8 with HTTP; Tue, 3 Nov 2015 06:44:43 -0800 (PST) In-Reply-To: References: Date: Tue, 3 Nov 2015 06:44:43 -0800 X-Google-Sender-Auth: B3BoPCQiyG_9voXGWgY-iMfJ9Qk Message-ID: Subject: Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing) From: Stack To: HBase Dev List Content-Type: multipart/alternative; boundary=001a1141b65635b9310523a3efb6 --001a1141b65635b9310523a3efb6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Stuff seems to be basically working. I just committed a fix for this message so it should be gone now: grep: unknown devices method grep: write error: Broken pipe Hopefully we've seen last of the errant surefire kills. We have an HBASE_TRUNK build and we have an HBASE_TRUNK_MATRIX. We should get rid of the former now? St.Ack On Mon, Nov 2, 2015 at 11:03 PM, Stack wrote: > I just did an edit of all of our jenkins configuration. I changed the > post-build zombie section. The surefire-killer was hanging out here it > seems (For detail, see HBASE-14589 Looking for the surefire-killer; build= s > being killed...). I may have messed up builds. Will fix in morning if so > (currently builds seem a bit backed up so won't wait around). > > St.Ack > > On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey wrote: > >> Hi Folks, >> >> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS] >> reducing our load on builds.a.o' on reasoning or if you have concerns. >> >> Our project job view now has a brief summary of what's going on: >> >> https://builds.apache.org/view/H-L/view/HBase/ >> >> * Where we had matrix builds, I confirmed they're doing the same thing >> (in at least one test-configuration) as the non-matrix version >> * I disabled jobs that were duplicating work done in matrix jobs, or >> that only worked on a tag >> * Made sure all non-disabled 0.98+ jobs are set to notify jira + >> builds mailing list >> * I updated pre-1.1 jobs to look for changes 1/day >> * I updated 1.1 jobs to look for changes every 8 hours >> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours >> >> There's still some non-pressing clean up work to do, so I'll file a >> jira later today just to make sure it's all recorded somewhere. >> >> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey >> wrote: >> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2 >> builds to >> > use branch-1.2. (ref HBASE-13912) >> > >> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey >> wrote: >> >> >> >> I've add a new build test to run the ITs under sunny day conditions >> using >> >> minicluster for both java 7 and java 8 on the 1.2 line. >> >> >> >> https://builds.apache.org/job/HBase-1.2-IT/ >> >> >> >> I haven't turned on notifications yet, because BulkIngest and >> TestIngest >> >> are read. details on HBASE-13750 >> >> >> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey >> wrote: >> >>> >> >>> sigh. that should have been "is now a matrix build". >> >>> >> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey >> wrote: >> >>>> >> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK= 8 >> in >> >>>> parallel. >> >>>> >> >>>> I saved the old job for now and left it disabled[2]. >> >>>> >> >>>> >> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/ >> >>>> [2]: >> >>>> >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix= / >> >>>> >> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey >> >>>> wrote: >> >>>>> >> >>>>> Sorry for the noise. I also updated the checkstyle step on >> HBase-TRUNK >> >>>>> >> >>>>> from >> >>>>> mvn checkstyle:checkstyle >> >>>>> to >> >>>>> mvn -DskipTests package checkstyle:checkstyle >> >>>>> >> >>>>> to deal with the same issue. Looks all clear now. >> >>>>> >> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey >> >>>>> wrote: >> >>>>>> >> >>>>>> Updated the following builds: >> >>>>>> >> >>>>>> * HBase-TRUNK >> >>>>>> >> >>>>>> moved from >> >>>>>> >> >>>>>> mvn clean compile findbugs:findbugs >> >>>>>> >> >>>>>> to >> >>>>>> >> >>>>>> mvn clean -DskipTests package findbugs:findbugs >> >>>>>> >> >>>>>> To work around the known issue where we can't do a bootstrap >> compile >> >>>>>> without previous install or remote SNAPSHOT artifacts. (recently >> triggered >> >>>>>> by the addition of the procedure module on master) >> >>>>>> >> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, =E5=BC=A0=E9=93=8E wrote: >> >>>>>>> >> >>>>>>> Make findbugs and checkstyle plugins always run. >> >>>>>>> The default behavior only publishes result for stable builds, bu= t >> >>>>>>> master is >> >>>>>>> not always stable and sometimes we introduce new warnings in >> unstable >> >>>>>>> builds(see builds 6310-6314). >> >>>>>>> And findbugs and checkstyle will not fail unless we abort the >> >>>>>>> building >> >>>>>>> process I think, so it is safe to turn it on every time. >> >>>>>>> >> >>>>>>> Thanks. >> >>>>>>> >> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu : >> >>>>>>> >> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report. >> >>>>>>> > >> >>>>>>> > Cheers >> >>>>>>> > >> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, =E5=BC=A0=E9=93=8E >> wrote: >> >>>>>>> > >> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show >> >>>>>>> > > findbugs, >> >>>>>>> > > checkstyle and jacoco coverage report. >> >>>>>>> > > The config has been tested on >> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly >> 30 >> >>>>>>> > > times. >> >>>>>>> > Not >> >>>>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(t= he >> >>>>>>> > > overhead >> >>>>>>> > of >> >>>>>>> > > collecting information for code coverage). >> >>>>>>> > > Thanks. >> >>>>>>> > > >> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell > >: >> >>>>>>> > > >> >>>>>>> > > > +1, thanks a lot for improving our build hygiene. >> >>>>>>> > > > >> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis S=C3=B6ztutar >> >>>>>>> > > > >> >>>>>>> > > wrote: >> >>>>>>> > > > >> >>>>>>> > > > > Thanks Sean. This is great. >> >>>>>>> > > > > >> >>>>>>> > > > > Enis >> >>>>>>> > > > > >> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey >> >>>>>>> > > > > >> >>>>>>> > > > wrote: >> >>>>>>> > > > > >> >>>>>>> > > > > > FYI I just finished chasing down the breakage for mvn >> site >> >>>>>>> > > > > > on all >> >>>>>>> > > patch >> >>>>>>> > > > > > builds. >> >>>>>>> > > > > > >> >>>>>>> > > > > > HBASE-13191 consolidates the few places in the >> test-patch >> >>>>>>> > > > > > code >> >>>>>>> > where >> >>>>>>> > > we >> >>>>>>> > > > > > hard coded MAVEN_OPTS. >> >>>>>>> > > > > > >> >>>>>>> > > > > > If you look at the PreCommit job now, we use the "set >> >>>>>>> > > > > > environment >> >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then everythin= g >> >>>>>>> > > > > > else >> >>>>>>> > respects >> >>>>>>> > > > > that >> >>>>>>> > > > > > setting. >> >>>>>>> > > > > > >> >>>>>>> > > > > > I set the initial value to be a combination of the >> memory >> >>>>>>> > limitations >> >>>>>>> > > > > we've >> >>>>>>> > > > > > been actually running with (the ~6G was getting ignore= d) >> >>>>>>> > > > > > and the >> >>>>>>> > > > permgen >> >>>>>>> > > > > > needed for site. >> >>>>>>> > > > > > >> >>>>>>> > > > > > MAVEN_OPTS=3D-Xmx3100M -XX:-UsePerfData >> -XX:MaxPermSize=3D256m >> >>>>>>> > > > > > >> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack < >> stack@duboce.net> >> >>>>>>> > > > > > wrote: >> >>>>>>> > > > > > >> >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm a= s >> >>>>>>> > > > > > > patch-build >> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPT= S >> to >> >>>>>>> > > > > > > be the >> >>>>>>> > > same >> >>>>>>> > > > as >> >>>>>>> > > > > > > those of trunk build too, setting >> >>>>>>> > > > > > > MAVEN_OPTS=3D"-Xmx6100m"... it >> >>>>>>> > had >> >>>>>>> > > > been >> >>>>>>> > > > > > > 3000. >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > Yours, >> >>>>>>> > > > > > > St.Ack >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack < >> stack@duboce.net> >> >>>>>>> > > > > > > wrote: >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds >> and >> >>>>>>> > > > > > > > or last >> >>>>>>> > 7 >> >>>>>>> > > > > days, >> >>>>>>> > > > > > > > whichever comes first. >> >>>>>>> > > > > > > > St.Ack >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack >> >>>>>>> > > > > > > > >> >>>>>>> > wrote: >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run >> mostly >> >>>>>>> > > > > > > >> blue >> >>>>>>> > > (give >> >>>>>>> > > > or >> >>>>>>> > > > > > > take >> >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue >> branch-1 >> >>>>>>> > > > > > > >> has >> >>>>>>> > > helped >> >>>>>>> > > > us >> >>>>>>> > > > > > > >> identify at least one destabilizing commit in the >> last >> >>>>>>> > > > > > > >> few >> >>>>>>> > days, >> >>>>>>> > > > > maybe >> >>>>>>> > > > > > > two; >> >>>>>>> > > > > > > >> this is as it should be (smile). >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch, >> make >> >>>>>>> > > > > > > >> sure >> >>>>>>> > > > > subsequent >> >>>>>>> > > > > > > >> builds stay blue. You can subscribe to >> >>>>>>> > builds@hbase.apache.org >> >>>>>>> > > to >> >>>>>>> > > > > get >> >>>>>>> > > > > > > >> notice of failures if not already subscribed. >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> Thanks, >> >>>>>>> > > > > > > >> St.Ack >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> 1. >> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/ >> >>>>>>> > > > > > > >> 2. >> >>>>>>> > > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack >> >>>>>>> > > > > > > >> >> >>>>>>> > > wrote: >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >>> A few notes on testing. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now and >> after >> >>>>>>> > > > > > > >>> some >> >>>>>>> > > work, >> >>>>>>> > > > we >> >>>>>>> > > > > > are >> >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. >> Lets >> >>>>>>> > > > > > > >>> try and >> >>>>>>> > > keep >> >>>>>>> > > > it >> >>>>>>> > > > > > > this >> >>>>>>> > > > > > > >>> way going forward. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with more >> >>>>>>> > > > > > > >>> capable >> >>>>>>> > > hardware >> >>>>>>> > > > > > seems >> >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are most= ly >> >>>>>>> > > > > > > >>> passing >> >>>>>>> > now >> >>>>>>> > > on >> >>>>>>> > > > > > > branch-1 >> >>>>>>> > > > > > > >>> and master. Lets try and keep it this way and >> start >> >>>>>>> > > > > > > >>> to trust >> >>>>>>> > > our >> >>>>>>> > > > > > test >> >>>>>>> > > > > > > runs >> >>>>>>> > > > > > > >>> again. Just a few flakies remain. Lets try and >> nail >> >>>>>>> > > > > > > >>> them. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test >> suites >> >>>>>>> > > > > > > >>> where >> >>>>>>> > > > previous >> >>>>>>> > > > > > we >> >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our >> zombie >> >>>>>>> > detector >> >>>>>>> > > > > > reports >> >>>>>>> > > > > > > >>> tests from another project altogether as lingere= rs >> >>>>>>> > > > > > > >>> (To be >> >>>>>>> > > fixed). >> >>>>>>> > > > > > > Some of >> >>>>>>> > > > > > > >>> our tests are failing because a concurrent hbase >> run >> >>>>>>> > > > > > > >>> is >> >>>>>>> > undoing >> >>>>>>> > > > > > > classes and >> >>>>>>> > > > > > > >>> data from under it. Also, lets fix. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for >> them to >> >>>>>>> > complete. >> >>>>>>> > > > > Many >> >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up >> multiple >> >>>>>>> > clusters >> >>>>>>> > > > and >> >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle th= ey >> >>>>>>> > > > > > > >>> pass at >> >>>>>>> > all. >> >>>>>>> > > > > > > Usually >> >>>>>>> > > > > > > >>> integration tests have been cast as unit tests >> >>>>>>> > > > > > > >>> because there >> >>>>>>> > > was >> >>>>>>> > > > no >> >>>>>>> > > > > > > where >> >>>>>>> > > > > > > >>> else for them to get an airing. We have the >> hbase-it >> >>>>>>> > > > > > > >>> suite >> >>>>>>> > now >> >>>>>>> > > > > which >> >>>>>>> > > > > > > would >> >>>>>>> > > > > > > >>> be a more apt place but until these are run on a >> >>>>>>> > > > > > > >>> regular >> >>>>>>> > basis >> >>>>>>> > > in >> >>>>>>> > > > > > > public >> >>>>>>> > > > > > > >>> for all to see, the fat integration tests >> disguised >> >>>>>>> > > > > > > >>> as unit >> >>>>>>> > > tests >> >>>>>>> > > > > > will >> >>>>>>> > > > > > > >>> remain. A review of our current unit tests >> weeding >> >>>>>>> > > > > > > >>> the old >> >>>>>>> > > cruft >> >>>>>>> > > > > and >> >>>>>>> > > > > > > the >> >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a nice >> >>>>>>> > > > > > > >>> undertaking >> >>>>>>> > if >> >>>>>>> > > > > > > someone is >> >>>>>>> > > > > > > >>> looking to contribute. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Alex Newman has been working on making our tests >> work >> >>>>>>> > > > > > > >>> up on >> >>>>>>> > > > travis >> >>>>>>> > > > > > and >> >>>>>>> > > > > > > >>> circle-ci. That'll be sweet when it goes >> end-to-end. >> >>>>>>> > > > > > > >>> He >> >>>>>>> > also >> >>>>>>> > > > > added >> >>>>>>> > > > > > in >> >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter, >> >>>>>>> > > > > > > >>> mapreduce -- >> >>>>>>> > > > > alongside >> >>>>>>> > > > > > > our >> >>>>>>> > > > > > > >>> old "sizing" categorizations of >> small/medium/large. >> >>>>>>> > > > > > > >>> His >> >>>>>>> > > thinking >> >>>>>>> > > > > is >> >>>>>>> > > > > > > that >> >>>>>>> > > > > > > >>> we can run these categorizations in parallel so = we >> >>>>>>> > > > > > > >>> could run >> >>>>>>> > > the >> >>>>>>> > > > > > total >> >>>>>>> > > > > > > >>> suite in about the time of the longest test, say >> >>>>>>> > 20-30minutes? >> >>>>>>> > > > We >> >>>>>>> > > > > > > could >> >>>>>>> > > > > > > >>> even change Apache to run them this way. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> FYI, >> >>>>>>> > > > > > > >>> St.Ack >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > -- >> >>>>>>> > > > > > Sean >> >>>>>>> > > > > > >> >>>>>>> > > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > -- >> >>>>>>> > > > Best regards, >> >>>>>>> > > > >> >>>>>>> > > > - Andy >> >>>>>>> > > > >> >>>>>>> > > > Problems worthy of attack prove their worth by hitting >> back. - >> >>>>>>> > > > Piet >> >>>>>>> > Hein >> >>>>>>> > > > (via Tom White) >> >>>>>>> > > > >> >>>>>>> > > >> >>>>>>> > >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Sean >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Sean >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Sean >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Sean >> >> >> >> >> >> >> >> >> >> -- >> >> Sean >> > >> > >> > >> > >> > -- >> > Sean >> >> >> >> -- >> Sean >> > > --001a1141b65635b9310523a3efb6--