hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu Zhang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-72) Porting Pig unit tests to use MiniDFSCluster and MiniMRCluster on the local machine
Date Tue, 29 Jan 2008 18:44:41 GMT

    [ https://issues.apache.org/jira/browse/PIG-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563613#action_12563613
] 

Xu Zhang commented on PIG-72:
-----------------------------

PigTestCase does things very similar to ClusterMapReduceTestCase.  The problem now is that
when PigTestCase is used there are exceptions as follows in the unit test summary even though
the summary shows the test runs are successful.  

XXXXXXXXX 10:25:55 ~/workspace/pig-trunk $ ant test
Buildfile: build.xml

test:
    [junit] Running org.apache.pig.test.TestAlgebraicEval
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 205.489 sec
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)
    [junit] java.io.IOException: Filesystem closed
    [junit]     at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:177)
    [junit]     at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:411)
    [junit]     at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:143)
    [junit]     at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:223)

BUILD SUCCESSFUL
Total time: 3 minutes 26 seconds


> Porting Pig unit tests to use MiniDFSCluster and MiniMRCluster on the local machine
> -----------------------------------------------------------------------------------
>
>                 Key: PIG-72
>                 URL: https://issues.apache.org/jira/browse/PIG-72
>             Project: Pig
>          Issue Type: Test
>          Components: tools
>            Reporter: Xu Zhang
>         Attachments: PortUnitTestToMiniClusters.patch
>
>
> We have the need to port the Pig unit tests to use MiniDFSCluster and MiniMRCluster,
so that tests can be executed with the DFS and MR threads on the local machine.   This feature
will eliminate the need to set up a real distributed hadoop cluster before running the unit
tests, as everything will now be carried out with the (mini) cluster on the user's local machine.
 
> One prerequisite for using this feature is a hadoop jar that has the class files for
MiniDFSCluster, MiniMRCluster and other supporting components.  I have been able to generate
such a jar file with a special target added by myself to hadoop's build.xml and have also
logged a hadoop jira to request this target be a permanent part of that build file.  If possible,
we can just replace hadoop15.jar with this jar file on the SVN source tree and then the users
will never need to worry about the availability of this jar file. Please find such a hadoop
jar file in the attachment.
> To use the feature in unit tests, the user just need to call MiniClusterBuilder.buildCluster()
before a PigServer instance is created with the string "mapreduce" as the parameter to its
constructor.  Here is an example of how the MiniClusterBuilder is used in a test case class:
>         public class TestWhatEver extends TestCase {
> 	        private String initString = "mapreduce";
> 	        private MiniClusterBuilder cluster = MiniClusterBuilder.buildCluster();
> 	
>                 @Test
>                 public void testGroupCountWithMultipleFields() throws Exception {
>                         PigServer pig = new PigServer(initString);
>                         // Do something with the pig server, such as registering and
executing Pig 
>                         // queries. The queries will executed with the local cluster.

>                 }
>      
>                 // More test cases if needed
>         }
> To run the unit tests with the local cluster, under the top directory of the source tree,
issue the command "ant test". Notice that you do not need to specify the location of the hadoop-site.xml
file with the command line option "-Djunit.hadoop.conf=<dir>" anymore. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message