Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@locus.apache.org Received: (qmail 61160 invoked from network); 4 Apr 2008 17:13:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 4 Apr 2008 17:13:02 -0000 Received: (qmail 72670 invoked by uid 500); 4 Apr 2008 17:13:02 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 72647 invoked by uid 500); 4 Apr 2008 17:13:02 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 72638 invoked by uid 99); 4 Apr 2008 17:13:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 10:13:02 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jim@powerset.com designates 207.5.72.93 as permitted sender) Received: from [207.5.72.93] (HELO exhub015-1.exch015.msoutlookonline.net) (207.5.72.93) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 17:12:10 +0000 Received: from EXVMBX015-1.exch015.msoutlookonline.net ([207.5.72.71]) by exhub015-1.exch015.msoutlookonline.net ([207.5.72.93]) with mapi; Fri, 4 Apr 2008 10:12:30 -0700 From: Jim Kellerman To: "hbase-dev@hadoop.apache.org" Date: Fri, 4 Apr 2008 10:12:29 -0700 Subject: RE: StackOverFlow Error in HBase Thread-Topic: StackOverFlow Error in HBase Thread-Index: AciWdguN49t0HqU3StCS7QaijdkK5gAAM38A Message-ID: <84E2AE771361E9419DD0EFBD31F09C4D4F5A2A3E57@EXVMBX015-1.exch015.msoutlookonline.net> References: <13050.217.74.68.2.1206621641.squirrel@poczta2.xg.pl> <47EBB1CE.9030505@duboce.net> <1206645120.6724.80.camel@da-laptop-csw> <47EBF75B.30208@duboce.net> <1206906868.15138.5.camel@da-laptop-csw> <47EFF9BA.6080201@duboce.net> <000c01c892e6$51b8ad50$105bd659@pcdavidalves> <1206967156.6479.33.camel@da-laptop-csw> <000901c8941f$f909b880$a201a8c0@pcdavidalves> <84E2AE771361E9419DD0EFBD31F09C4D4F5A1A30CB@EXVMBX015-1.exch015.msoutlookonline.net> <002901c89440$645c5690$e782364d@pcdavidalves> <84E2AE771361E9419DD0EFBD31F09C4D4F5A1A381B@EXVMBX015-1.exch015.msoutlookonline.net> <1207239641.9743.2.camel@da-laptop-csw> <84E2AE771361E9419DD0EFBD31F09C4D4F5A1A387D@EXVMBX015-1.exch015.msoutlookonline.net> <1207315741.6541.5.camel@da-laptop-csw> <84E2AE771361E9419DD0EFBD31F09C4D4F5A2A3E09@EXVMBX015-1.exch015.msoutlookonline.net> <1207328676.6541.8.camel@da-laptop-csw> In-Reply-To: <1207328676.6541.8.camel@da-laptop-csw> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Sorry. I misunderstood the question. It would be better if you checked out = HBase branch 0.1 --- Jim Kellerman, Senior Engineer; Powerset > -----Original Message----- > From: David Alves [mailto:dr-alves@criticalsoftware.com] > Sent: Friday, April 04, 2008 10:05 AM > To: hbase-dev@hadoop.apache.org > Subject: RE: StackOverFlow Error in HBase > > Hi Jim > > Of course, my questions was regarding whether I > should use HEAD or some branch or tag. > Anyways I currently running Hbase HEAD patched > against Hadoop HEAD, I'll know if its ok soon. > > Regards > David Alves > On Fri, 2008-04-04 at 09:18 -0700, Jim Kellerman wrote: > > After applying the patch, you have to rebuild and deploy on your > > cluster, run your test that was failing and verify that it > now works. > > See > > > http://hadoop.apache.org/hbase/docs/current/api/overview-summary.html# > > overview_description > > > > > > > > --- > > Jim Kellerman, Senior Engineer; Powerset > > > > > > > -----Original Message----- > > > From: David Alves [mailto:dr-alves@criticalsoftware.com] > > > Sent: Friday, April 04, 2008 6:29 AM > > > To: hbase-dev@hadoop.apache.org > > > Subject: RE: StackOverFlow Error in HBase > > > > > > Hi all again > > > > > > I've never used the patch system you guys use, so > in order > > > to test the patch submitted by Clint what do I have to do? I mean > > > I've updated HEAD and applied the patch, is this it? > > > > > > Regards > > > David Alves > > > > > > > > > > > > On Thu, 2008-04-03 at 10:18 -0700, Jim Kellerman wrote: > > > > Thanks David. I'll add 554 as a blocker for 0.1.1 > > > > > > > > --- > > > > Jim Kellerman, Senior Engineer; Powerset > > > > > > > > > > > > > -----Original Message----- > > > > > From: David Alves [mailto:dr-alves@criticalsoftware.com] > > > > > Sent: Thursday, April 03, 2008 9:21 AM > > > > > To: hbase-user@hadoop.apache.org > > > > > Subject: RE: StackOverFlow Error in HBase > > > > > > > > > > Hi Jim and all > > > > > > > > > > I'll commit to test the patch under the same > > > conditions as > > > > > it failed before, (with around 36000 records) but in this > > > > > precise moment I preparing my next development > iteration, which > > > means a lot > > > > > of meetings. > > > > > By the end of the day tomorrow (friday) I > should have a > > > > > confirmation whether the patch worked (or not). > > > > > > > > > > Regards > > > > > David Alves > > > > > > > > > > On Thu, 2008-04-03 at 09:12 -0700, Jim Kellerman wrote: > > > > > > David, > > > > > > > > > > > > Have you had a chance to try this patch? We are about to > > > > > release hbase-0.1.1 and until we receive a confirmation in > > > > > HBASE-554 from another person who has tried it and > > > verifies that it > > > > > works, we cannot include it in this release. If it is not in > > > > > this release, there will be a significant wait for it > to appear > > > > > in an hbase release. hbase-0.1.2 will not happen anytime soon > > > unless there > > > > > are critical issues that arise that have not been > fixed in 0.1.1. > > > > > hbase-0.2.0 is also some time in the future. There are a > > > significant > > > > > number of issues to address before that release is ready. > > > > > > > > > > > > Frankly, I'd like to see this patch in 0.1.1, because it is > > > > > an issue for people that use filters. > > > > > > > > > > > > The alternative would be for Clint to supply a test case > > > > > that fails without the patch but passes with the patch. > > > > > > > > > > > > We will hold up the release, but need a commitment either > > > > > from David to test the patch or for Clint to supply a test. > > > > > We need that commitment by the end of the day today > > > > > 2008/04/03 along with an eta as to when it will be completed. > > > > > > > > > > > > --- > > > > > > Jim Kellerman, Senior Engineer; Powerset > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: David Alves [mailto:dr-alves@criticalsoftware.com] > > > > > > > Sent: Tuesday, April 01, 2008 2:36 PM > > > > > > > To: hbase-user@hadoop.apache.org > > > > > > > Subject: RE: StackOverFlow Error in HBase > > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > I just deployed the unpatched version. > > > > > > > Tomorrow I'll rebuild the system with the patch > > > > > and try it > > > > > > > out. > > > > > > > Thanks again. > > > > > > > > > > > > > > Regards > > > > > > > David Alves > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Jim Kellerman [mailto:jim@powerset.com] > > > > > > > > Sent: Tuesday, April 01, 2008 10:04 PM > > > > > > > > To: hbase-user@hadoop.apache.org > > > > > > > > Subject: RE: StackOverFlow Error in HBase > > > > > > > > > > > > > > > > David, > > > > > > > > > > > > > > > > Have you tried this patch and does it work for > you? If so > > > > > > > > we'll include it > > > > > > > > hbase-0.1.1 > > > > > > > > > > > > > > > > --- > > > > > > > > Jim Kellerman, Senior Engineer; Powerset > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: David Alves > [mailto:dr-alves@criticalsoftware.com] > > > > > > > > > Sent: Tuesday, April 01, 2008 10:44 AM > > > > > > > > > To: hbase-user@hadoop.apache.org > > > > > > > > > Subject: RE: StackOverFlow Error in HBase > > > > > > > > > > > > > > > > > > Hi > > > > > > > > > Thanks for the prompt path Clint, St.Ack and > > > > > all you guys. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > David Alves > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: clint.a.m@gmail.com > [mailto:clint.a.m@gmail.com] > > > > > > > On Behalf > > > > > > > > > > Of Clint Morgan > > > > > > > > > > Sent: Tuesday, April 01, 2008 2:04 AM > > > > > > > > > > To: hbase-user@hadoop.apache.org > > > > > > > > > > Subject: Re: StackOverFlow Error in HBase > > > > > > > > > > > > > > > > > > > > Try the patch at > > > > > > > https://issues.apache.org/jira/browse/HBASE-554. > > > > > > > > > > > > > > > > > > > > cheers, > > > > > > > > > > -clint > > > > > > > > > > > > > > > > > > > > On Mon, Mar 31, 2008 at 5:39 AM, David Alves > > > > > > > > > > wrote: > > > > > > > > > > > Hi ... again > > > > > > > > > > > > > > > > > > > > > > In my previous mail I stated that > > > increasing the > > > > > > > > > stack size > > > > > > > > > > solved the > > > > > > > > > > > problem, well I jumped a little bit to the > > > conclusion, > > > > > > > > > in fact it > > > > > > > > > > > didn't, the StackOverFlowError always occurs > > > at the end > > > > > > > > > of the cycle > > > > > > > > > > > when no more records match the filter. Anyway I've > > > > > > > rewritten my > > > > > > > > > > > application to use a normal scanner and and do the > > > > > > > > > "filtering" after > > > > > > > > > > > which is not optimal but it works. > > > > > > > > > > > I'm just saying this because it might > > > be a clue, > > > > > > > > > in previous > > > > > > > > > > versions > > > > > > > > > > > (!=3D 0.1.0) even though a more serious > > > problem happened > > > > > > > > > > > (regionservers became irresponsive after so many > > > > > > > records) this > > > > > > > > > > > didn't happen. Btw in current version I > notice no, > > > > > > > > > > > or > > > > > > > > > very small, > > > > > > > > > > > decrease of thoughput with time, great work! > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > David Alves > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 2008-03-31 at 05:18 +0100, David > Alves wrote: > > > > > > > > > > > > Hi again > > > > > > > > > > > > > > > > > > > > > > > > As I was almost at the end (80%) > of indexable > > > > > > > > > docs, for the > > > > > > > > > > time > > > > > > > > > > > > being I simply increased the stack size, which > > > > > > > seemed to work. > > > > > > > > > > > > Thanks for your input St.Ack > really helped me > > > > > > > > > solve the problem > > > > > > > > > > at > > > > > > > > > > > > least for the moment. > > > > > > > > > > > > On another note in the same > method I changed > > > > > > > the way the > > > > > > > > > > scanner was > > > > > > > > > > > > obtained when htable.getStartKeys() > would be more > > > > > > > > > > > than > > > > > > > > > 1, so that > > > > > > > > > > > I > > > > > > > > > > could > > > > > > > > > > > > limit the records read each time to a single > > > > > > > region, and the > > > > > > > > > > > scanning > > > > > > > > > > would > > > > > > > > > > > > start at the last region, strangely > the number of > > > > > > > > > > > keys > > > > > > > > > obtained > > > > > > > > > > > by > htable.getStartKeys() was always 1 > even though > > > > > > > by the end > > > > > > > > > > > there are > > > > > > > > > > already > > > > > > > > > > > > 21 regions. > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > David Alves > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > From: stack > > > > > > > > > > > [mailto:stack@duboce.net] > > Sent: > > > > > > > > > Sunday, March > > > > > > > > > > > 30, 2008 9:36 PM > > To: > > > hbase-user@hadoop.apache.org > > > > > > > > > > > > > > > > > > > > > > Subject: > > > > > > > > > > > Re: StackOverFlow Error in HBase > > > > You're > > > > > > > doing nothing > > > > > > > > > > > wrong. > > > > > > > > > > > > > > > > > > > > > > > > > > The filters as written recurse until > they find > > > > > > > > > > > a > > > > > > > match. If > > > > > > > > > > > long > > stretches between matching rows, > > > then you will > > > > > > > > > get a > > > > > > > > > > > > > StackOverflowError. Filters need to be changed. > > > > > Thanks for > > > > > > > > > > pointing > > > > > > > > > > > > > this out. Can you do without them for the > > > > > > > > > > > moment > > > > > > > > > until we get > > > > > > > > > > > a > > > > > > > > > > chance > > > > > > > > > > > > > to fix it? (HBASE-554) > > > > Thanks, > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > David Alves wrote: > > > > > > > > > > > > > > Hi St.Ack and all > > > > > > > > > > > > > > > > > The error always occurs when > trying to see if > > > > > > > > > there are more > > > > > > > > > > rows to > > > > > > > > > > > > > > process. > > > > > > > > > > > > > > Yes I'm using a > filter(RegExpRowFilter) to > > > > > > > > > select only the rows > > > > > > > > > > (any > > > > > > > > > > > > > > row key) that match a specific > value in one > > > > > > > > > > > of > > > > > > > the columns. > > > > > > > > > > > > > > Then I obtain the scanner just test > > > the hasNext > > > > > > > > > method, close > > > > > > > > > > the > > > > > > > > > > > > > > scanner and return. > > > > > > > > > > > > > > Am I doing something wrong? > > > > > > > > > > > > > > Still StackOverflowError is not > supposed to > > > > > > > happen right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > David Alves > > > > > > > > > > > > > > On Thu, 2008-03-27 at 12:36 -0700, > stack wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> You are using a filter? If so, tell us > > > > > more about it. > > > > > > > > > > > > > >> St.Ack > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> David Alves wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >>> Hi guys > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> I 'm using HBase to keep > data that is > > > > > > > > > later indexed. > > > > > > > > > > > > > >>> The data is indexed in > chunks so the > > > > > > > > > cycle is get XXXX > > > > > > > > > > records index > > > > > > > > > > > > > >>> them check for more records etc... > > > > > > > > > > > > > >>> When I tryed the candidate-2 > > > instead of > > > > > > > > > the old 0.16.0 > > > > > > > > > > (which I > > > > > > > > > > > > > >>> switched to do to the regionservers > > > > > > > > > > > becoming > > > > > > > > > unresponsive) > > > > > > > > > > > I > > > > > > > > > > got the > > > > > > > > > > > > > >>> error in the end of this email > well into an > > > > > > > indexing job. > > > > > > > > > > > > > >>> So you have any idea > why? Am I doing > > > > > > > > > something wrong? > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> David Alves > > >>> > > >>> > > > > > > > > > > > java.lang.RuntimeException: > > > > > > > > > > org.apache.hadoop.ipc.RemoteException: > > > > > > > > > > > > > >>> java.io.IOException: > > > java.lang.StackOverflowError > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > java.io.DataInputStream.readFully(DataInputStream.java:178 > > > > > > > > > > ) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > java.io.DataInputStream.readLong(DataInputStream.java:399) > > > > > > > > > > > > > >>> at > org.apache.hadoop.dfs.DFSClient > > > > > > > > > > > > > >>> > $BlockReader.readChunk(DFSClient.java:735) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInpu > > > > > > > > > tC > > > > > > > > > hecker.java: > > > > > > > > > > > > > 234) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176 > > > > > > > ) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:19 > > > > > > > 3) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:157 > > > > > > > ) > > > > > > > > > > > > > >>> at > org.apache.hadoop.dfs.DFSClient > > > > > > > > > > > > > >>> $BlockReader.read(DFSClient.java:658) > > > > > > > > > > > > > >>> at > org.apache.hadoop.dfs.DFSClient > > > > > > > > > > > > > >>> > > > $DFSInputStream.readBuffer(DFSClient.java:1130) > > > > > > > > > > > > > >>> at > org.apache.hadoop.dfs.DFSClient > > > > > > > > > > > > > >>> $DFSInputStream.read(DFSClient.java:1166) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > java.io.DataInputStream.readFully(DataInputStream.java:178 > > > > > > > > > > ) > > > > > > > > > > > > > >>> at > > > org.apache.hadoop.io.DataOutputBuffer > > > > > > > > > > > > > >>> $Buffer.write(DataOutputBuffer.java:56) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90 > > > > > > > > > ) > > > > > > > > > > > > > >>> at > org.apache.hadoop.io.SequenceFile > > > > > > > > > > > > > >>> $Reader.next(SequenceFile.java:1829) > > > > > > > > > > > > > >>> at > org.apache.hadoop.io.SequenceFile > > > > > > > > > > > > > >>> $Reader.next(SequenceFile.java:1729) > > > > > > > > > > > > > >>> at > org.apache.hadoop.io.SequenceFile > > > > > > > > > > > > > >>> $Reader.next(SequenceFile.java:1775) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > org.apache.hadoop.io.MapFile$Reader.next(MapFile.java:461) > > > > > > > > > > > > > >>> at org.apache.hadoop.hbase.HStore > > > > > > > > > > > > > >>> > $StoreFileScanner.getNext(HStore.java:2350) > > > > > > > > > > > > > >>> at > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.HAbstractScanner.next(HAbstractScanner.java: > > > > > > > > > 25 > > > > > > > > > > 6) > > > > > > > > > > > > > >>> at org.apache.hadoop.hbase.HStore > > > > > > > > > > > > > >>> $HStoreScanner.next(HStore.java:2561) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1807) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > > > > > > > > > >>> at > org.apache.hadoop.hbase.HRegion > > > > > > > > > > > > > >>> $HScanner.next(HRegion.java:1843) > > > > > >>> ... > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No virus found in this incoming message. > > > > > > > > > Checked by AVG. > > > > > > > > > Version: 7.5.519 / Virus Database: 269.22.3/1354 - > > > > > > > > > Release > > > > > > > > > Date: 4/1/2008 5:38 AM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No virus found in this outgoing message. > > > > > > > > Checked by AVG. > > > > > > > > Version: 7.5.519 / Virus Database: 269.22.3/1354 - > > > Release Date: > > > > > > > > 4/1/2008 > > > > > > > > 5:38 AM > > > > > > > > > > > > > > > > > > > > > No virus found in this incoming message. > > > > > > > Checked by AVG. > > > > > > > Version: 7.5.519 / Virus Database: 269.22.3/1354 - Release > > > > > > > Date: 4/1/2008 5:38 AM > > > > > > > > > > > > > > > > > > > > > > > > > > No virus found in this outgoing message. > > > > > > Checked by AVG. > > > > > > Version: 7.5.519 / Virus Database: 269.22.5/1357 - > Release Date: > > > > > > 4/3/2008 10:48 AM > > > > > > > > > > > > > > > > > > > > > No virus found in this incoming message. > > > > > Checked by AVG. > > > > > Version: 7.5.519 / Virus Database: 269.22.5/1357 - Release > > > > > Date: 4/3/2008 10:48 AM > > > > > > > > > > > > > > > > > > No virus found in this outgoing message. > > > > Checked by AVG. > > > > Version: 7.5.519 / Virus Database: 269.22.5/1357 - Release Date: > > > > 4/3/2008 10:48 AM > > > > > > > > > > > > > No virus found in this incoming message. > > > Checked by AVG. > > > Version: 7.5.519 / Virus Database: 269.22.5/1359 - Release > > > Date: 4/4/2008 8:23 AM > > > > > > > > > > No virus found in this outgoing message. > > Checked by AVG. > > Version: 7.5.519 / Virus Database: 269.22.5/1359 - Release Date: > > 4/4/2008 8:23 AM > > > > > No virus found in this incoming message. > Checked by AVG. > Version: 7.5.519 / Virus Database: 269.22.5/1359 - Release > Date: 4/4/2008 8:23 AM > > No virus found in this outgoing message. Checked by AVG. Version: 7.5.519 / Virus Database: 269.22.5/1359 - Release Date: 4/4/2008 8= :23 AM