hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pi song" <pi.so...@gmail.com>
Subject Re: Slow unit testing
Date Tue, 13 May 2008 14:35:00 GMT
FYI,

Starting Jetty in Hadoop on my machine takes much longer than you.
Normally, it takes at least 15 seconds each time here when it takes
sub-second in your logs.
I will have to dig more.

08/05/10 23:22:01 INFO fs.FSNamesystem: Registered FSNamesystemStatusMBean
08/05/10 23:22:16 INFO util.Credential: Checking Resource aliases
08/05/10 23:22:16 INFO http.HttpServer: Version Jetty/5.1.x


08/05/10 23:22:17 INFO dfs.DataNode: Balancing bandwith is 1048576 bytes/s
08/05/10 23:22:32 INFO http.HttpServer: Version Jetty/5.1.x
08/05/10 23:22:33 INFO util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@147358f



08/05/10 23:22:33 INFO dfs.DataNode: BlockReport of 0 blocks got processed
in 5 msecs
08/05/10 23:22:48 INFO http.HttpServer: Version Jetty/5.1.x
08/05/10 23:22:48 INFO util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@1d87b85



08/05/10 23:22:49 INFO dfs.DataNode: BlockReport of 0 blocks got processed
in 6 msecs
08/05/10 23:23:04 INFO http.HttpServer: Version Jetty/5.1.x
08/05/10 23:23:04 INFO util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@1412e75



08/05/10 23:23:04 INFO dfs.DataNode: BlockReport of 0 blocks got processed
in 8 msecs
08/05/10 23:23:19 INFO http.HttpServer: Version Jetty/5.1.x
08/05/10 23:23:20 INFO util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@15d252d



08/05/10 23:23:36 INFO mapred.MiniMRCluster: mapred.local.dir is
/tmp/hadoop-root/mapred/local/0_0
08/05/10 23:23:51 INFO http.HttpServer: Version Jetty/5.1.x
08/05/10 23:23:51 INFO util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@903025



On Tue, May 13, 2008 at 8:54 AM, pi song <pi.songs@gmail.com> wrote:

> Thanks for your information.
> I'll look at this tonight.
>
>
> On 5/13/08, Olga Natkovich <olgan@yahoo-inc.com> wrote:
> >
> > We do see similar slowdown testing with hadoop 0.17 and still trying to
> > figure out why this is happening. I wonder if this could be related.
> >
> > > -----Original Message-----
> > > From: Alan Gates [mailto:gates@yahoo-inc.com]
> > > Sent: Monday, May 12, 2008 9:41 AM
> > > To: pig-dev@incubator.apache.org
> > > Subject: Re: Slow unit testing
> > >
> > > Sorry, I should have been more clear.  Time results similar
> > > to Olga's, not Pi's.
> > >
> > > Alan.
> > >
> > > Alan Gates wrote:
> > > > I run the unit tests on a mac laptop and am seeing similar
> > > time results.
> > > >
> > > > Alan.
> > > >
> > > > Olga Natkovich wrote:
> > > >> Pi,
> > > >>
> > > >> Inline my stdout content. Attached test output files. I ran this
> > > >> against latest trunk on a linux box.
> > > >>
> > > >> test-local:
> > > >>
> > > >> testOneMode:
> > > >>     [mkdir] Created dir:
> > > >> /home/olgan/src/pig-apache3/trunk/build/test/logs-local
> > > >>     [junit] Running org.apache.pig.test.TestAlgebraicEval
> > > >>     [junit] Tests run: 5, Failures: 0, Errors: 0, Time
> > > elapsed: 2.93 sec
> > > >>     [junit] Running org.apache.pig.test.TestBinaryStorage
> > > >>     [junit] Tests run: 2, Failures: 0, Errors: 0, Time
> > > elapsed: 0.662
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestBuiltin
> > > >>     [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed:
> > > >> 3.849 sec
> > > >>     [junit] Running org.apache.pig.test.TestCmdLineParser
> > > >>     [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed:
> > > >> 0.011 sec
> > > >>     [junit] Running org.apache.pig.test.TestCombiner
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.421
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestCompressedFiles
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.337
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestCustomSlicer
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.214
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestDataBag
> > > >>     [junit] Tests run: 18, Failures: 0, Errors: 0, Time elapsed:
> > > >> 1.461 sec
> > > >>     [junit] Running org.apache.pig.test.TestDataModel
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time
> > > elapsed: 0.126
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestEvalPipeline
> > > >>     [junit] Tests run: 7, Failures: 0, Errors: 0, Time
> > > elapsed: 5.035
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestExGenCogroup
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time
> > > elapsed: 1.246
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestExGenEval
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 1.172
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestFilterOpNumeric
> > > >>     [junit] Tests run: 8, Failures: 0, Errors: 0, Time
> > > elapsed: 1.76 sec
> > > >>     [junit] Running org.apache.pig.test.TestFilterOpString
> > > >>     [junit] Tests run: 6, Failures: 0, Errors: 0, Time
> > > elapsed: 1.385
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestGrunt
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.238
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestInfixArithmetic
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed:
> > > >> 57.608 sec
> > > >>     [junit] Running org.apache.pig.test.TestLogicalPlanBuilder
> > > >>     [junit] Tests run: 49, Failures: 0, Errors: 0, Time
> > > elapsed: 1.04
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestMapReduce
> > > >>     [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed:
> > > >> 73.303 sec
> > > >>     [junit] Running
> > > org.apache.pig.test.TestMapReduceResultRecycling
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 1.542
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestParamSubPreproc
> > > >>     [junit] Tests run: 28, Failures: 0, Errors: 0, Time elapsed:
> > > >> 0.633 sec
> > > >>     [junit] Running org.apache.pig.test.TestParser
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.206
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestPigContext
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 0.806
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestPigFile
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 3.344
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestPigScriptParser
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.441
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestPigServer
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time
> > > elapsed: 5.76 sec
> > > >>     [junit] Running org.apache.pig.test.TestPigSplit
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.546
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestReversibleLoadStore
> > > >>     [junit] Tests run: 2, Failures: 0, Errors: 0, Time
> > > elapsed: 0.622
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestStore
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 2.139
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestStreaming
> > > >>     [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed:
> > > >> 1.074 sec
> > > >>
> > > >> test-mapreduce:
> > > >>
> > > >> testOneMode:
> > > >>     [mkdir] Created dir:
> > > >> /home/olgan/src/pig-apache3/trunk/build/test/logs-mapreduce
> > > >>     [junit] Running org.apache.pig.test.TestAlgebraicEval
> > > >>     [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed:
> > > >> 74.738 sec
> > > >>     [junit] Running org.apache.pig.test.TestBinaryStorage
> > > >>     [junit] Tests run: 2, Failures: 0, Errors: 0, Time
> > > elapsed: 36.99
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestBuiltin
> > > >>     [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed:
> > > >> 4.513 sec
> > > >>     [junit] Running org.apache.pig.test.TestCmdLineParser
> > > >>     [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed:
> > > >> 0.034 sec
> > > >>     [junit] Running org.apache.pig.test.TestCombiner
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 24.9 sec
> > > >>     [junit] Running org.apache.pig.test.TestCompressedFiles
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> > > >> 29.658 sec
> > > >>     [junit] Running org.apache.pig.test.TestCustomSlicer
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> > > >> 90.949 sec
> > > >>     [junit] Running org.apache.pig.test.TestDataBag
> > > >>     [junit] Tests run: 18, Failures: 0, Errors: 0, Time elapsed:
> > > >> 1.605 sec
> > > >>     [junit] Running org.apache.pig.test.TestDataModel
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time
> > > elapsed: 0.194
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestEvalPipeline
> > > >>     [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed:
> > > >> 203.826 sec
> > > >>     [junit] Running org.apache.pig.test.TestExGenCogroup
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed:
> > > >> 15.116 sec
> > > >>     [junit] Running org.apache.pig.test.TestExGenEval
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 8.609
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestFilterOpNumeric
> > > >>     [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed:
> > > >> 104.898 sec
> > > >>     [junit] Running org.apache.pig.test.TestFilterOpString
> > > >>     [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed:
> > > >> 83.079 sec
> > > >>     [junit] Running org.apache.pig.test.TestGrunt
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.296
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestInfixArithmetic
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed:
> > > >> 50.802 sec
> > > >>     [junit] Running org.apache.pig.test.TestLogicalPlanBuilder
> > > >>     [junit] Tests run: 49, Failures: 0, Errors: 0, Time elapsed:
> > > >> 1.328 sec
> > > >>     [junit] Running org.apache.pig.test.TestMapReduce
> > > >>     [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed:
> > > >> 52.169 sec
> > > >>     [junit] Running
> > > org.apache.pig.test.TestMapReduceResultRecycling
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> > > >> 43.827 sec
> > > >>     [junit] Running org.apache.pig.test.TestParamSubPreproc
> > > >>     [junit] Tests run: 28, Failures: 0, Errors: 0, Time elapsed:
> > > >> 0.724 sec
> > > >>     [junit] Running org.apache.pig.test.TestParser
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 8.08 sec
> > > >>     [junit] Running org.apache.pig.test.TestPigContext
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 1.209
> > > >> sec
> > > >>     [junit] Running org.apache.pig.test.TestPigFile
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time
> > > elapsed: 0.95 sec
> > > >>     [junit] Running org.apache.pig.test.TestPigScriptParser
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > > elapsed: 0.26 sec
> > > >>     [junit] Running org.apache.pig.test.TestPigServer
> > > >>     [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed:
> > > >> 10.313 sec
> > > >>     [junit] Running org.apache.pig.test.TestPigSplit
> > > >>     [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed:
> > > >> 109.971 sec
> > > >>     [junit] Running org.apache.pig.test.TestReversibleLoadStore
> > > >>     [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed:
> > > >> 11.803 sec
> > > >>     [junit] Running org.apache.pig.test.TestStore
> > > >>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed:
> > > >> 125.27 sec
> > > >>     [junit] Running org.apache.pig.test.TestStreaming
> > > >>     [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed:
> > > >> 142.835 sec
> > > >>
> > > >> test:
> > > >>
> > > >> BUILD SUCCESSFUL
> > > >> Total time: 26 minutes 4 seconds
> > > >> ucdev3:~/src/pig-apache3/trunk> cd
> > > >> ~/src/pig-yahoo/trunk/pig/conf/piglet/hadoop-site.xml
> > >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message