hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pi Song (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-114) store one alias/logicalPlan twice leads to instantiation of StoreFunc as LoadFunc
Date Fri, 07 Mar 2008 12:47:47 GMT

    [ https://issues.apache.org/jira/browse/PIG-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576168#action_12576168
] 

Pi Song commented on PIG-114:
-----------------------------

Ben,

Sorry that I led to a long story again. I think whether to do marker interfaces or not can
be addressed when we implement optimization.
At this stage I just want to fix this bug which as discussed above will be fixed by
(1) Add a structure to identify reversible property of StoreFunc
(2) Fix optimization engine not to do reverse if it's not safe
(3) Unit testing

I have implemented (1) using what I've proposed before
{noformat}
LoadFunc
StoreFunc
ReversibleLoadStoreFunc extends LoadFunc, StoreFunc
{noformat}

As you've said
{quote}
Alternatively we could add a method to StoreFunc:
LoadFunc getReloader();
{quote}
We still cannot conclude your way or my way which way is better. But since I've already implemented
using my way. Let's use my one. ( Meritocrazy :D )

For (2) I've fixed optimization in both local and mapreduce engines

For (3) This time I do an end-to-end test. I'm sorry for that. I find it much more complex
to do only plan compiler test (This involves both local and mapreduce engines and physical
ops are different). Also a usual way to do optimization test is to compare actual output structure
with expected output structure (deterministic optimization engine). We still don't have that
framework yet. This should be done as a part of optimization work.




> store one alias/logicalPlan twice leads to instantiation of StoreFunc as LoadFunc
> ---------------------------------------------------------------------------------
>
>                 Key: PIG-114
>                 URL: https://issues.apache.org/jira/browse/PIG-114
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Johannes Zillmann
>            Assignee: Pi Song
>         Attachments: PIG114_FixOptimize_Sample.patch, pigPatch-storeTwice-620665.patch
>
>
> Calling PigServer#store() twice for an alias results in following exception :
> {noformat}
> java.lang.RuntimeException: java.lang.ClassCastException: org.apache.pig.test.DummyStoreFunc
cannot be cast to org.apache.pig.LoadFunc
> 	at org.apache.pig.backend.local.executionengine.POLoad.<init>(POLoad.java:59)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:167)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:184)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:184)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:111)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:90)
> 	at org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:1)
> 	at org.apache.pig.PigServer.store(PigServer.java:330)
> 	at org.apache.pig.PigServer.store(PigServer.java:317)
> 	at org.apache.pig.test.StoreTwiceTest.testIt(StoreTwiceTest.java:31)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:589)
> 	at junit.framework.TestCase.runTest(TestCase.java:164)
> 	at junit.framework.TestCase.runBare(TestCase.java:130)
> 	at junit.framework.TestResult$1.protect(TestResult.java:110)
> 	at junit.framework.TestResult.runProtected(TestResult.java:128)
> 	at junit.framework.TestResult.run(TestResult.java:113)
> 	at junit.framework.TestCase.run(TestCase.java:120)
> 	at junit.framework.TestSuite.runTest(TestSuite.java:228)
> 	at junit.framework.TestSuite.run(TestSuite.java:223)
> 	at org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:35)
> 	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:45)
> 	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
> 	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
> Caused by: java.lang.ClassCastException: org.apache.pig.test.DummyStoreFunc cannot be
cast to org.apache.pig.LoadFunc
> 	at org.apache.pig.backend.local.executionengine.POLoad.<init>(POLoad.java:57)
> 	... 28 more
> {noformat}
> I will attach a patch with a test scenario for this. Basically the code is as follow:
> {noformat}PigServer pig = new PigServer(ExecType.LOCAL);
>         pig
>                 .registerQuery("A = LOAD 'test/org/apache/pig/test/StoreTwiceTest.java'
USING "
>                         + DummyLoadFunc.class.getName() + "();");
>         pig.registerQuery("B = FOREACH A GENERATE * ;");
>         File outputFile = new File("/tmp/testPigOutput");
>         outputFile.delete();
>         pig.store("A", outputFile.getAbsolutePath(), DummyStoreFunc.class
>                 .getName()
>                 + "()");
>         outputFile.delete();
>         pig.store("B", outputFile.getAbsolutePath(), DummyStoreFunc.class
>                 .getName()
>                 + "()");
>         outputFile.delete();
>         assertEquals(2, _storedTuples.size());
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message