hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xu Zhang ...@yahoo-inc.com>
Subject Re: [jira] Commented: (PIG-152) Solution for PIG-123 might have introduced parsing errors on Windows with the Pig unit tests
Date Mon, 17 Mar 2008 21:36:13 GMT
Thanks Pi.  I do not think there is any special tricks besides those with
Linux (such as you will need ant, junit, java, svn, etc).  If you want, you
can also send me your pig.jar and then I will try it out for you.

Best regards,
Xu


> From: "Pi Song (JIRA)" <jira@apache.org>
> Reply-To: <pig-dev@incubator.apache.org>
> Date: Mon, 17 Mar 2008 14:12:24 -0700 (PDT)
> To: <pig-dev@incubator.apache.org>
> Subject: [jira] Commented: (PIG-152) Solution for PIG-123 might have
> introduced parsing errors on Windows with the Pig unit tests
> 
> 
>     [ 
> https://issues.apache.org/jira/browse/PIG-152?page=com.atlassian.jira.plugin.s
> ystem.issuetabpanels:comment-tabpanel&focusedCommentId=12579606#action_1257960
> 6 ] 
> 
> Pi Song commented on PIG-152:
> -----------------------------
> 
> Xu, I'll be looking at this as soon as possible. In the mean time, can you
> tell me if there is any trick to get Pig running on Windows?
> 
>> Solution for PIG-123 might have introduced parsing errors on Windows with the
>> Pig unit tests
>> -----------------------------------------------------------------------------
>> ---------------
>> 
>>                 Key: PIG-152
>>                 URL: https://issues.apache.org/jira/browse/PIG-152
>>             Project: Pig
>>          Issue Type: Bug
>>          Components: impl
>>            Reporter: Xu Zhang
>> 
>> *Since the changes for PIG-123 were checked in, we got lots of parsing errors
>> when running the unit tests on Windows.*
>> It seems all these errors are generated when a query string that contains a
>> single quote for the path of the data file is passed into the
>> PigServer::registerQuery() method.
>> Here is an example of such an error and the location where it is generated in
>> the unit test code (on the line "pig.registerQuery(query);" in the following
>> code):
>> {noformat}
>> org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line
>> 1, column 39.  Encountered: "W" (87), after : "\'file:c:\\"
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParserTokenManager.getNextToken(
>> QueryParserTokenManager.java:1599)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.jj_consume_token(QueryPar
>> ser.java:4069)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.FileName(QueryParser.java
>> :717)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.ja
>> va:615)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :497)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.ja
>> va:425)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.GroupItem(QueryParser.jav
>> a:1043)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser
>> .java:996)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :523)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedExpr(QueryParser.ja
>> va:425)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser
>> .java:1364)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java
>> :552)
>> at 
>> 
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:373>>
)
>> at 
>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:22
>> 7)
>> at 
>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.
>> java:47)
>> at org.apache.pig.PigServer.registerQuery(PigServer.java:237)
>> at 
>> org.apache.pig.test.TestAlgebraicEval.testSimpleCount(TestAlgebraicEval.java:
>> 50)
>> Standard Output
>> Starting DataNode 0 with dfs.data.dir: dfs\data\data1,dfs\data\data2
>> Starting DataNode 1 with dfs.data.dir: dfs\data\data3,dfs\data\data4
>> Starting DataNode 2 with dfs.data.dir: dfs\data\data5,dfs\data\data6
>> Starting DataNode 3 with dfs.data.dir: dfs\data\data7,dfs\data\data8
>> myid =  foreach (group (load 'file:c:\WINDOWS\TEMP\test39837txt') all)
>> generate COUNT($1);
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39838txt') all)
>> generate group, COUNT($1) ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39839txt') all)
>> generate COUNT($1), group ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39840txt' using
>> org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1) ;
>> myid = foreach (group (load 'file:c:\WINDOWS\TEMP\test39841txt' using
>> org.apache.pig.builtin.PigStorage(':')) by $0) generate group, COUNT($1.$1),
>> COUNT($1.$0) ;
>> {noformat}
>> Here is the corresponding code:
>> {code}
>> public class TestAlgebraicEval extends TestCase {
>>     
>> MiniCluster cluster = MiniCluster.buildCluster();
>>     @Test
>>     public void testSimpleCount() throws Throwable {
>>         int LOOP_COUNT = 1024;
>>         PigServer pig = new PigServer(MAPREDUCE);
>>         File tmpFile = File.createTempFile("test", "txt");
>>         PrintStream ps = new PrintStream(new FileOutputStream(tmpFile));
>>         for(int i = 0; i < LOOP_COUNT; i++) {
>>             ps.println(i);
>>         }
>>         ps.close();
>>         String query = "myid =  foreach (group (load 'file:" + tmpFile + "')
>> all) generate COUNT($1);";
>>         System.out.println(query);
>>         pig.registerQuery(query);
>>         Iterator it = pig.openIterator("myid");
>>         tmpFile.delete();
>>         Tuple t = (Tuple)it.next();
>>         Double count = t.getAtomField(0).numval();
>>         assertEquals(count, (double)LOOP_COUNT);
>>     }
>>     .
>>     .
>>     .
>> {code}
> 
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
> 


Mime
View raw message