pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2898) Parallel execution of e2e tests
Date Wed, 26 Sep 2012 05:28:07 GMT

    [ https://issues.apache.org/jira/browse/PIG-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463537#comment-13463537

Rohini Palaniswamy commented on PIG-2898:

   Thanks it worked for local mode. But there were some additional test failures than usual
in H23. H20 tests are still running and I will update status on them tomorrow.

Few comments:
1) Can you change yarn.nodemanager.local-dirs to mapreduce.cluster.local.dir. yarn.nodemanager.local-dirs
was a wrong suggestion from me. Tested with mapreduce.cluster.local.dir and it works. 
2) Can you change name of hadoop.mapred.dir to hadoop.mapred.local.dir as it is slightly confusing
and make it configurable through commandline. In many cases /tmp gets full and would be good
to have the ability to point to some other dir.
3) I had some comments in the reviewboard. Can you incorporate them too and post an updated
patch in reviewboard.
> Parallel execution of e2e tests
> -------------------------------
>                 Key: PIG-2898
>                 URL: https://issues.apache.org/jira/browse/PIG-2898
>             Project: Pig
>          Issue Type: Improvement
>          Components: e2e harness
>    Affects Versions: 0.10.0
>            Reporter: Andrey Klochkov
>            Assignee: Ivan A. Veselovsky
>              Labels: test
>         Attachments: PIG-2898-against-trunk-2.patch, PIG-2898-fix-sub-prototypes.patch,
> Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The bottleneck
here is the client side, and per our observations it can help a lot if the e2e harness would
be able to run tests in parallel threads.
> We prototyped changes in e2e harness allowing to run tests in a configurable number of
threads. Preliminary results show more than 6x reduction in execution time when using a small
3-nodes M/R cluster with modest configuration. Going to share a patch shortly.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message