pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pi song" <pi.so...@gmail.com>
Subject Re: Slow unit testing
Date Mon, 12 May 2008 14:49:45 GMT
Below is stat from my latest test run which took 83 minutes. Is there anyone
else having the same problem?

Pi

test-local:

testOneMode:
    [mkdir] Created dir: /root/IdeaProjects/Pig/build/test/logs-local
    [junit] Running org.apache.pig.test.TestAlgebraicEval
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 1.523 sec
    [junit] Running org.apache.pig.test.TestBinaryStorage
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.507 sec
    [junit] Running org.apache.pig.test.TestBuiltin
    [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed: 4.414 sec
    [junit] Running org.apache.pig.test.TestCmdLineParser
    [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed: 0.047 sec
    [junit] Running org.apache.pig.test.TestCombiner
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.486 sec
    [junit] Running org.apache.pig.test.TestCompressedFiles
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.257 sec
    [junit] Running org.apache.pig.test.TestCustomSlicer
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.208 sec
    [junit] Running org.apache.pig.test.TestDataBag
    [junit] Tests run: 18, Failures: 0, Errors: 0, Time elapsed: 1.303 sec
    [junit] Running org.apache.pig.test.TestDataModel
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.201 sec
    [junit] Running org.apache.pig.test.TestEvalPipeline
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 8.903 sec
    [junit] Running org.apache.pig.test.TestExGenCogroup
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 2.09 sec
    [junit] Running org.apache.pig.test.TestExGenEval
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.579 sec
    [junit] Running org.apache.pig.test.TestFilterOpNumeric
    [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 3.581 sec
    [junit] Running org.apache.pig.test.TestFilterOpString
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 1.677 sec
    [junit] Running org.apache.pig.test.TestGrunt
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.553 sec
    [junit] Running org.apache.pig.test.TestInfixArithmetic
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 53.791 sec
    [junit] Running org.apache.pig.test.TestLogicalPlanBuilder
    [junit] Tests run: 49, Failures: 0, Errors: 0, Time elapsed: 1.17 sec
    [junit] Running org.apache.pig.test.TestMapReduce
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 57.749 sec
    [junit] Running org.apache.pig.test.TestMapReduceResultRecycling
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.428 sec
    [junit] Running org.apache.pig.test.TestParamSubPreproc
    [junit] Tests run: 28, Failures: 0, Errors: 0, Time elapsed: 0.317 sec
    [junit] Running org.apache.pig.test.TestParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.213 sec
    [junit] Running org.apache.pig.test.TestPigContext
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.938 sec
    [junit] Running org.apache.pig.test.TestPigFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.501 sec
    [junit] Running org.apache.pig.test.TestPigScriptParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.283 sec
    [junit] Running org.apache.pig.test.TestPigServer
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 1.338 sec
    [junit] Running org.apache.pig.test.TestPigSplit
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.477 sec
    [junit] Running org.apache.pig.test.TestReversibleLoadStore
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.661 sec
    [junit] Running org.apache.pig.test.TestStore
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.196 sec
    [junit] Running org.apache.pig.test.TestStreaming
    [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 1.187 sec

test-mapreduce:

testOneMode:
    [mkdir] Created dir: /root/IdeaProjects/Pig/build/test/logs-mapreduce
    [junit] Running org.apache.pig.test.TestAlgebraicEval
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 258.721 sec
    [junit] Running org.apache.pig.test.TestBinaryStorage
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 184.114 sec
    [junit] Running org.apache.pig.test.TestBuiltin
    [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed: 4.538 sec
    [junit] Running org.apache.pig.test.TestCmdLineParser
    [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed: 0.07 sec
    [junit] Running org.apache.pig.test.TestCombiner
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 173.987 sec
    [junit] Running org.apache.pig.test.TestCompressedFiles
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 174.538 sec
    [junit] Running org.apache.pig.test.TestCustomSlicer
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 236.571 sec
    [junit] Running org.apache.pig.test.TestDataBag
    [junit] Tests run: 18, Failures: 0, Errors: 0, Time elapsed: 1.478 sec
    [junit] Running org.apache.pig.test.TestDataModel
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.165 sec
    [junit] Running org.apache.pig.test.TestEvalPipeline
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 360.121 sec
    [junit] Running org.apache.pig.test.TestExGenCogroup
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 160.593 sec
    [junit] Running org.apache.pig.test.TestExGenEval
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 158.003 sec
    [junit] Running org.apache.pig.test.TestFilterOpNumeric
    [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 262.014 sec
    [junit] Running org.apache.pig.test.TestFilterOpString
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 234.839 sec
    [junit] Running org.apache.pig.test.TestGrunt
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.159 sec
    [junit] Running org.apache.pig.test.TestInfixArithmetic
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 53.881 sec
    [junit] Running org.apache.pig.test.TestLogicalPlanBuilder
    [junit] Tests run: 49, Failures: 0, Errors: 0, Time elapsed: 1.396 sec
    [junit] Running org.apache.pig.test.TestMapReduce
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 54.656 sec
    [junit] Running org.apache.pig.test.TestMapReduceResultRecycling
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 198.565 sec
    [junit] Running org.apache.pig.test.TestParamSubPreproc
    [junit] Tests run: 28, Failures: 0, Errors: 0, Time elapsed: 0.305 sec
    [junit] Running org.apache.pig.test.TestParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 158.771 sec
    [junit] Running org.apache.pig.test.TestPigContext
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.762 sec
    [junit] Running org.apache.pig.test.TestPigFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3.69 sec
    [junit] Running org.apache.pig.test.TestPigScriptParser
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.336 sec
    [junit] Running org.apache.pig.test.TestPigServer
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 158.031 sec
    [junit] Running org.apache.pig.test.TestPigSplit
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 258.747 sec
    [junit] Running org.apache.pig.test.TestReversibleLoadStore
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 160.657 sec
    [junit] Running org.apache.pig.test.TestStore
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 280.73 sec
    [junit] Running org.apache.pig.test.TestStreaming
    [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 286.489 sec

test:

BUILD SUCCESSFUL
Total time: 83 minutes 8 seconds



On Tue, May 13, 2008 at 12:40 AM, pi song <pi.songs@gmail.com> wrote:

> When you run your tests next time, could you please zip all logs and send
> to me?
> This has been really annoying. I'll have to fix it.
>
> Pi
>
>
> On Tue, May 13, 2008 at 12:36 AM, Olga Natkovich <olgan@yahoo-inc.com>
> wrote:
>
> > I also see a lot of "noise" in my logs. I don't remember seeing this
> > before  though it did not see to effect speed of my tests. I wonder if
> > this is somehow related to the test changes that happened recently.
> >
> >
> > ________________________________
> >
> >        From: pi song [mailto:pi.songs@gmail.com]
> >         Sent: Monday, May 12, 2008 7:30 AM
> >         To: pig-dev@incubator.apache.org
> >         Subject: Re: Slow unit testing
> >
> >
> >        After looking at the log files:-
> >        1) It is not between tests but within the tests. Just my screen
> > is updating too slowly.
> >        2) My log files look very messy. Is your file as messy as mine?
> >
> >        Pi
> >
> >
> >        On Tue, May 13, 2008 at 12:20 AM, pi song <pi.songs@gmail.com>
> > wrote:
> >
> >
> >                This started to happen since the introduction of
> > Hadoop16 (which includes changes in HOD). The time shown in ANT-Junit is
> > not accurate. What I can see here is that  time spent doing nothing
> > between each tests is long but not taken into account.
> >
> >                Is it possible to be something to do with connecting to
> > HOD?
> >
> >                    [junit] Tests run: 8, Failures: 0, Errors: 0, Time
> > elapsed: 262.014 sec
> >                    [junit] Running
> > org.apache.pig.test.TestFilterOpString
> >                    [junit] Tests run: 6, Failures: 0, Errors: 0, Time
> > elapsed: 234.839 sec
> >                    [junit] Running org.apache.pig.test.TestGrunt
> >                    [junit] Tests run: 1, Failures: 0, Errors: 0, Time
> > elapsed: 0.159 sec
> >
> >                Pi
> >
> >
> >                On Tue, May 13, 2008 at 12:15 AM, pi song
> > <pi.songs@gmail.com> wrote:
> >
> >
> >                        Already more than 40 minutes.
> >
> >
> >                        On Tue, May 13, 2008 at 12:10 AM, Olga Natkovich
> > <olgan@yahoo-inc.com> wrote:
> >
> >
> >                                I run the tests on the regular basis and
> > don't see a proble. MR tests
> >                                take about 20 minites for me. How long
> > does it take for you?
> >
> >                                Olga
> >
> >
> >                                > -----Original Message-----
> >                                > From: pi song
> > [mailto:pi.songs@gmail.com]
> >                                > Sent: Monday, May 12, 2008 6:57 AM
> >                                > To: pig-dev@incubator.apache.org
> >                                > Subject: Slow unit testing
> >                                >
> >                                > It has been a while since we moved
> > from Hadoop15 to Hadoop16
> >                                > that all the MapReduce unit tests run
> > significantly slower
> >                                > than before (at least 2-3x).
> >                                > What is the reason behind this? Is
> > there a way to fix?
> >                                >
> >                                > Since everyone has to run at least a
> > full test before
> >                                > submitting a patch.
> >                                > This is not productive from
> > developer's point of view.
> >                                >
> >                                > Pi
> >                                >
> >
> >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message