hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Build failed in Jenkins: HBase-0.94 #151
Date Thu, 09 May 2013 08:23:00 GMT
I can get TestMetaReaderEditor to periodically fail (1 in 5 runs or so),
bisecting now.


On Thu, May 9, 2013 at 3:35 PM, Andrew Purtell <apurtell@apache.org> wrote:

> Looking into the state of the 0.94 branch.
>
>
> On Thu, May 9, 2013 at 3:25 PM, <apurtell@apache.org> wrote:
>
>> See <http://54.241.6.143/job/HBase-0.94/151/>
>>
>> ------------------------------------------
>> [...truncated 31888 lines...]
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.635 sec
>> Running org.apache.hadoop.hbase.client.TestFromClientSide3
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.479 sec
>> Running org.apache.hadoop.hbase.client.TestSnapshotFromClient
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.749 sec
>> Running org.apache.hadoop.hbase.client.TestMetaScanner
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.219 sec
>> Running org.apache.hadoop.hbase.client.TestMultiParallel
>> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.787
>> sec
>> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
>> Tests run: 5, Failures: 1, Errors: 3, Skipped: 0, Time elapsed: 257.635
>> sec <<< FAILURE!
>> Running org.apache.hadoop.hbase.client.TestAdmin
>> Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 122.143
>> sec
>> Running org.apache.hadoop.hbase.client.TestScannerTimeout
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.391 sec
>> Running org.apache.hadoop.hbase.client.TestHCM
>> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.307 sec
>> Running org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.625 sec
>> Running org.apache.hadoop.hbase.client.TestMultipleTimestamps
>> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.292 sec
>> Running org.apache.hadoop.hbase.client.TestTimestampsFilter
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.202 sec
>> Running org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 141.11 sec
>> Running org.apache.hadoop.hbase.client.TestShell
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.229 sec
>> Running org.apache.hadoop.hbase.client.TestHTableUtil
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.749 sec
>> Running org.apache.hadoop.hbase.master.TestMXBean
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.502 sec
>> Running org.apache.hadoop.hbase.master.TestActiveMasterManager
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.376 sec
>> Running org.apache.hadoop.hbase.master.TestClockSkewDetection
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.461 sec
>> Running org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.426 sec
>> Running org.apache.hadoop.hbase.TestClusterBootOrder
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.397 sec
>> Running
>> org.apache.hadoop.hbase.master.handler.TestTableDescriptorModification
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.281 sec
>> Running org.apache.hadoop.hbase.master.handler.TestCreateTableHandler
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.171 sec
>> Running
>> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.488 sec
>> Running org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
>> Tests run: 59, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 192.84
>> sec
>> Running org.apache.hadoop.hbase.master.TestMaster
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.44 sec
>> Running
>> org.apache.hadoop.hbase.master.handler.TestTableDeleteFamilyHandler
>> Running org.apache.hadoop.hbase.master.TestDefaultLoadBalancer
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.345 sec
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.602 sec
>> Running org.apache.hadoop.hbase.master.TestDeadServer
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.39 sec
>> Running org.apache.hadoop.hbase.master.TestHMasterRPCException
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.073 sec
>> Running org.apache.hadoop.hbase.master.TestMasterTransitions
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 21.095 sec
>> Running org.apache.hadoop.hbase.client.TestFromClientSide
>> Tests run: 59, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 188.263
>> sec
>> Running org.apache.hadoop.hbase.master.cleaner.TestLogsCleaner
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.29 sec
>> Running org.apache.hadoop.hbase.master.cleaner.TestHFileCleaner
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.634 sec
>> Running org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.363 sec
>> Running org.apache.hadoop.hbase.master.TestMasterZKSessionRecovery
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.567 sec
>> Running org.apache.hadoop.hbase.master.TestRestartCluster
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.005 sec
>> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.457 sec
>> Running org.apache.hadoop.hbase.master.TestMasterFileSystem
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.471 sec
>> Running org.apache.hadoop.hbase.master.TestSplitLogManager
>> Running org.apache.hadoop.hbase.master.TestRollingRestart
>> Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.977
>> sec
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.002 sec
>> Running org.apache.hadoop.hbase.master.TestAssignmentManager
>> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.481 sec
>> Running org.apache.hadoop.hbase.master.TestMasterFailover
>> Running org.apache.hadoop.hbase.master.TestMasterStatusServlet
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.541 sec
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.283 sec
>> Running org.apache.hadoop.hbase.master.TestOpenedRegionHandler
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.376 sec
>> Running org.apache.hadoop.hbase.master.TestMasterShutdown
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.917 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestChecksum
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.416 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex
>> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.405
>> sec
>> Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.022 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.5 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock
>> Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.409
>> sec
>> Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility
>> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.921 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
>> Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.074 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.77 sec
>> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 156.566
>> sec
>> Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache
>> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.731 sec
>> Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.9 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
>> Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.462
>> sec
>> Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.275 sec
>> Running org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 73.349 sec
>> Running org.apache.hadoop.hbase.io.TestFileLink
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.588 sec
>> Running
>> org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.357
>> sec
>> Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders
>> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 167.193
>> sec
>> Running org.apache.hadoop.hbase.TestInfoServers
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.594 sec
>> Running org.apache.hadoop.hbase.thrift.TestThriftServer
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.404 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
>> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.7 sec
>> Running org.apache.hadoop.hbase.TestFullLogReconstruction
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.539 sec
>> Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.209
>> sec
>> Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine
>> Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 376.209
>> sec
>> Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 182.42 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.369 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 166.412
>> sec
>> Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.119 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
>> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.066 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
>> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 461.92 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
>> Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 452.382
>> sec
>> Running
>> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.041 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestImportExport
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 121.374
>> sec
>> Running org.apache.hadoop.hbase.TestMultiVersions
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.135 sec
>> Running org.apache.hadoop.hbase.constraint.TestConstraint
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.243 sec
>> Running org.apache.hadoop.hbase.TestAcidGuarantees
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 107.309
>> sec
>> Running org.apache.hadoop.hbase.procedure.TestZKProcedureControllers
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.27 sec
>> Running org.apache.hadoop.hbase.procedure.TestZKProcedure
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.774 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
>> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 526.822
>> sec
>> Running org.apache.hadoop.hbase.TestZooKeeper
>> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.744 sec
>> Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
>> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.456 sec
>> Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.142
>> sec
>> Running org.apache.hadoop.hbase.backup.TestHFileArchiving
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 178.892
>> sec
>> Running org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
>> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 603.965
>> sec
>>
>> Results :
>>
>> Failed tests:
>> testRetrying(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor): reader:
>> count=1, t=null
>>
>> Tests in error:
>>   testTableExists(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
>> Failed after attempts=10, exceptions:(..)
>>   testGetRegion(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor):
>> Failed after attempts=10, exceptions:(..)
>>
>> testScanMetaForTable(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor)
>>
>> Tests run: 1348, Failures: 1, Errors: 3, Skipped: 13
>>
>> [JENKINS] Recording test results
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:06:05.589s
>> [INFO] Finished at: Thu May 09 07:25:26 UTC 2013
>> [INFO] Final Memory: 53M/254M
>> [INFO]
>> ------------------------------------------------------------------------
>> [JENKINS] Archiving <http://54.241.6.143/job/HBase-0.94/ws/pom.xml> to
>> /var/lib/jenkins/jobs/HBase-0.94/modules/org.apache.hbase$hbase/builds/2013-05-09_06-19-13/archive/org.apache.hbase/hbase/0.94.8-SNAPSHOT/hbase-0.94.8-SNAPSHOT.pom
>> Waiting for Jenkins to finish collecting data
>> mavenExecutionResult exceptions not empty
>> message : Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
>> (secondPartTestsExecution) on project hbase: There are test failures.
>>
>> Please refer to <
>> http://54.241.6.143/job/HBase-0.94/ws/target/surefire-reports> for the
>> individual test results.
>> cause : There are test failures.
>>
>> Please refer to <
>> http://54.241.6.143/job/HBase-0.94/ws/target/surefire-reports> for the
>> individual test results.
>> Stack trace :
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
>> goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
>> (secondPartTestsExecution) on project hbase: There are test failures.
>>
>> Please refer to <
>> http://54.241.6.143/job/HBase-0.94/ws/target/surefire-reports> for the
>> individual test results.
>>         at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
>>         at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>         at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>         at
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
>>         at
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
>>         at
>> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
>>         at
>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
>>         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
>>         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
>>         at
>> org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
>>         at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
>>         at
>> org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:158)
>>         at hudson.maven.Maven3Builder.call(Maven3Builder.java:100)
>>         at hudson.maven.Maven3Builder.call(Maven3Builder.java:66)
>>         at hudson.remoting.UserRequest.perform(UserRequest.java:118)
>>         at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>         at hudson.remoting.Request$2.run(Request.java:326)
>>         at
>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>>         at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: org.apache.maven.plugin.MojoFailureException: There are test
>> failures.
>>
>> Please refer to <
>> http://54.241.6.143/job/HBase-0.94/ws/target/surefire-reports> for the
>> individual test results.
>>         at
>> org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87)
>>         at
>> org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:651)
>>         at
>> org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:625)
>>         at
>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137)
>>         at
>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98)
>>         at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>>         at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
>>         ... 27 more
>> channel stopped
>> Archiving artifacts
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message