hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: Custom Loadfunc problem!
Date Wed, 28 Oct 2009 14:57:54 GMT
Richard,
I would strongly advise you against trying to mess with PigInputFormat
and JobControlCompiler.  We know that slicers work (after all, people
manage to write hbase loaders and such). Plus the whole
input/output/slice/slicer thing is being revised right now, there is a
proposal in PIG-966 you can look at if you would like to comment.

In fact I just took your code and substituted some placeholders for
various methods and classes (like HSSchema, etc), and I can see Slice
methods getting called and everything.

What do your location strings look like? Pig tries to be helpful by
transforming locations into HDFS paths if the locations are not
qualified by a scheme ( something:/ ). This makes file-based LoadFuncs
easier to write, but it also means that you have to specify a scheme
if you are doing something that's not file-based.  Try changing your
code to expect the location string to start with some scheme, such as
abcd:/ (or perhaps something more meaningful) and things should work.

-D

2009/10/28 RichardGUO Fei <gladiatorcn@hotmail.com>:
>
> It seems that the real problem is JobControlCompiler.getJob, because this method removes
POLoad and sets PigInputFormat as the input format but the path passed to my storage is actually
a table name rather than a file path! I think that the LoadFunc should be more flexible. I
am trying to fix this at this moment.
>
>
>
> Thanks,
>
> Richard
>
>> From: zjffdu@gmail.com
>> To: pig-dev@hadoop.apache.org
>> Subject: RE: Custom Loadfunc problem!
>> Date: Tue, 27 Oct 2009 23:40:43 -0800
>>
>>
>> I mean hadoop's local mode not pig's own local mode
>>
>>
>> -----Original Message-----
>> From: Dmitriy Ryaboy [mailto:dvryaboy@gmail.com]
>> Sent: 2009年10月26日 6:33
>> To: pig-dev@hadoop.apache.org; pig-dev@hadoop.apache.org
>> Subject: RE: Custom Loadfunc problem!
>>
>> Jeff,
>> Slicers dont work in local mode, there is an ancient ticket for that on the
>> Jira.
>>
>> Richard -- hard to say whats going on without more code. Think you can come
>> up with a simplified version of your loadfunc that fails in a similar
>> manner, and share it?
>>
>>
>>
>> -----Original Message-----
>> From: "zjffdu" <zjffdu@gmail.com>
>> To: pig-dev@hadoop.apache.org
>> Sent: 10/27/2009 1:45 AM
>> Subject: RE: Custom Loadfunc problem!
>>
>> Illustrate will not execute the job, while dump and store will execute it.
>> So I think there must be something wrong with your custom slicer. I suggest
>> you set breakpoint in your slicer and debug it in map reduce mode locally
>>
>>
>>
>> -----Original Message-----
>> From: RichardGUO Fei [mailto:gladiatorcn@hotmail.com]
>> Sent: 20091026 0:43
>> To: pig-dev@hadoop.apache.org
>> Subject: RE: Custom Loadfunc problem!
>>
>>
>> Hi,
>>
>>
>>
>> Btw, my program works with ILLUSTRATE but n
>>
>> [truncated by sender]
>>
>
> _________________________________________________________________
> MSN十周年庆典,查看MSN注册时间,赢取神秘大奖
> http://10.msn.com.cn

Mime
View raw message