pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From deepak kumar v <deepu....@gmail.com>
Subject Re: PIG-671
Date Wed, 09 Mar 2011 19:03:08 GMT
Hi Dmitriy,
Thanks for the help.

I was able to set up the project in eclipse after playing a bit with build
path. I am now trying to run unit test cases (TestBuiltin.java) with eclipse
so that debugging becomes easy and i can walk through bit of code while
running test cases in debug mode. (It runs from cmd line) However it fails
after sometime. PFA the exceptions.

Any help.

Regards,
Deepak

On Wed, Mar 9, 2011 at 9:52 AM, Dmitriy Ryaboy <dvryaboy@gmail.com> wrote:

> Deepak,
> Don't create from Ant Buildfile. Just import an existing project (create
> new java project -> uncheck "create in default location" -> point it at
> where you have the project checked out and where you ran ant eclipse-files).
> You might need to change the build path a bit, adding / removing jars and
> all that, but it should be pretty straightforward from there.
>
> D
>
> On Tue, Mar 8, 2011 at 7:43 PM, deepak kumar v <deepu.pig@gmail.com>wrote:
>
>> Hi,
>> Sure, will do that.
>>
>> Meanwhile, I tried to import pig src into eclipse using New project wizard
>> with "Create a Java Project from Ant Buildfile" option, i see the following
>> error
>> "Reference ${cp} not found" and it fails to import src code. A empty New
>> project "Pig" is created as a result.
>>
>> Before trying above i ran
>>
>> ant eclipse-files
>>
>> which ran to successful.
>>
>>
>> Regards,
>>
>> Deepak
>>
>>
>> On Wed, Mar 9, 2011 at 1:37 AM, Dmitriy Ryaboy <dvryaboy@gmail.com>wrote:
>>
>>> Nice work.
>>>
>>> You are going to want to make sure COUNT also works on the scenarios it's
>>> supposed to work on. So far you only seem to be testing failures?
>>>
>>> Also, write it up as proper unit tests so we don't get regressions.
>>>
>>> D
>>>
>>>
>>> On Tue, Mar 8, 2011 at 10:40 AM, deepak kumar v <deepu.pig@gmail.com>wrote:
>>>
>>>> Hi Dmitriy,
>>>> Will checkout TestBuiltins.java once my eclipse setup is ready.
>>>> Meanwhile i tried the couple of scenarios that you mentioned.
>>>>
>>>> 1) Schema defined for a
>>>> grunt> a = load 'test.txt' as (data:chararray);
>>>>  grunt> b = group a all;
>>>> grunt> describe a;
>>>> a: {data: chararray}
>>>> grunt> describe b;
>>>> b: {group: chararray,a: {(data: chararray)}}
>>>> grunt> x = foreach b generate COUNT(a.data, a.data);
>>>> grunt> dump x;
>>>> 2011-03-09 00:06:40,953 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>>>> ERROR 1045: Could not infer the matching function for
>>>> org.apache.pig.builtin.COUNT as multiple or none of them fit. Please use
an
>>>> explicit cast.
>>>>
>>>> 2) Schema not defined for a
>>>> grunt> a = load 'test.txt';
>>>> grunt> b = group a all;
>>>> grunt> describe a;
>>>> Schema for a unknown.
>>>> grunt> describe b;
>>>> b: {group: chararray,a: {(null)}}
>>>> grunt> x = foreach b generate COUNT(a.$0, a.$0);
>>>> grunt> dump x;
>>>> 2011-03-09 00:07:58,715 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>>>> ERROR 1045: Could not infer the matching function for
>>>> org.apache.pig.builtin.COUNT as multiple or none of them fit. Please use
an
>>>> explicit cast.
>>>>
>>>>
>>>> Changes seems to be working with both scenarios.
>>>>
>>>> Regards,
>>>> Deepak
>>>>
>>>>
>>>>
>>>> On Tue, Mar 8, 2011 at 10:45 PM, Dmitriy Ryaboy <dvryaboy@gmail.com>wrote:
>>>>
>>>>> ant test doesn't hang, it just runs for a very long time. If you want
>>>>> to test something specific, you can name the test class like so:
>>>>>
>>>>> ant test -Dtestcase=TestBuiltins
>>>>> (this will run the tests in TestBuiltins.java)
>>>>>
>>>>> COUNT tests are probably in TestBuiltins or in TestAlgebraic. Look
>>>>> around.
>>>>>
>>>>> You definitely want to add some tests to make sure that COUNT still
>>>>> works on the cases where it's supposed to work, and that the Pig parser
no
>>>>> longer allows COUNT with the wrong number or type of arguments.
>>>>>
>>>>> I would test in particular what happens when a bag is supplied for
>>>>> which a schema is known -- Pig might be making a distinction between
a bag
>>>>> with a known schema and a bag with an unknown schema, and we definitely
want
>>>>> both of those to work.
>>>>>
>>>>> D
>>>>>
>>>>>
>>>>> On Tue, Mar 8, 2011 at 1:58 AM, deepak kumar v <deepu.pig@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>> PFA patch of fix for PIG-671. Used the approach mentioned in previous
>>>>>> email.
>>>>>> I could not find any test cases for Count.java, besides ant test
just
>>>>>> hung up.
>>>>>>
>>>>>> Output:
>>>>>> grunt> a = load 'test.txt';
>>>>>> grunt> x = foreach a generate COUNT(a.$0,a.$0);
>>>>>> grunt> dump x;
>>>>>> 2011-03-08 14:45:03,408 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>>>> - ERROR 1045: Could not infer the matching function for
>>>>>> org.apache.pig.builtin.COUNT as multiple or none of them fit. Please
use an
>>>>>> explicit cast.
>>>>>> Details at logfile:
>>>>>> /Users/deepakkv/Documents/opensource/pig/working/pig_1299575686422.log
>>>>>> grunt> b = group a all;
>>>>>> grunt> x = foreach b generate COUNT(a.$0,a.$0);
>>>>>> grunt> dump x;
>>>>>> 2011-03-08 14:45:19,668 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>>>> - ERROR 1045: Could not infer the matching function for
>>>>>> org.apache.pig.builtin.COUNT as multiple or none of them fit. Please
use an
>>>>>> explicit cast.
>>>>>> Details at logfile:
>>>>>> /Users/deepakkv/Documents/opensource/pig/working/pig_1299575686422.log
>>>>>> grunt> quit
>>>>>>
>>>>>> Regards,
>>>>>> Deepak
>>>>>>
>>>>>> On Tue, Mar 8, 2011 at 12:12 PM, deepak kumar v <deepu.pig@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Dmitriy,
>>>>>>>
>>>>>>> I was looking SUBSTRING.java and thats exactly(getArgToFuncMapping)
>>>>>>> what i am trying now with COUNT.
>>>>>>> Waiting for the build to complete and test out my changes before
i
>>>>>>> could post this option.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Deepak
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 8, 2011 at 11:56 AM, Dmitriy Ryaboy <dvryaboy@gmail.com>wrote:
>>>>>>>
>>>>>>>> Actually I think if you just implement getArgToFuncMapping
for
>>>>>>>> COUNT, where you only return a mapping for a single bag argument,
pig will
>>>>>>>> notice that the wrong number of args is supplied during the
compilation
>>>>>>>> phase and no runtime exceptions will be required.
>>>>>>>>
>>>>>>>> I haven't checked how well the funcSpec mapping works with
Bags,
>>>>>>>> that's something to experiment with.
>>>>>>>>
>>>>>>>> D
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 7, 2011 at 9:55 PM, deepak kumar v <deepu.pig@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Pig Developers,
>>>>>>>>> This is my first dive into open source contribution and
i hope to
>>>>>>>>> dive deep.
>>>>>>>>>
>>>>>>>>> I was going through https://issues.apache.org/jira/browse/PIG-671and
>>>>>>>>> observed the following with COUNT.java
>>>>>>>>>
>>>>>>>>> COUNT.exec() always retrieves the first item from input
tuple which
>>>>>>>>> it
>>>>>>>>> assumes is a bag and counts the numbers of items in the
bag.
>>>>>>>>> Even if we pass multiple arguments to COUNT(), it will
always pick
>>>>>>>>> the first
>>>>>>>>> argument.
>>>>>>>>>
>>>>>>>>> There are few ways we go through this
>>>>>>>>> a) Leave as is cause it returns correct result for counting
the
>>>>>>>>> number of
>>>>>>>>> items in the first argument.
>>>>>>>>> OR
>>>>>>>>> b) Make a check for the size of the input tuple in COUNT.exec()
and
>>>>>>>>> if it is
>>>>>>>>> not 1 then throw ExecException()  or IllegalArgumentException
>>>>>>>>> {might be
>>>>>>>>> correct}
>>>>>>>>> which will cause the Map job to fail.
>>>>>>>>>
>>>>>>>>> Let me know how to we go about it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Deepak
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message