hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From RichardGUO Fei <gladiato...@hotmail.com>
Subject RE: Custom Loadfunc problem!
Date Wed, 28 Oct 2009 16:55:03 GMT

I think that you are right. But the most weird thing is that there is no LOG outputed.

I can see that the last method invoked was getLocations. And anything after that

was not recorded anywhere. Even in the pseudo-distributed mode of Hadoop,

if I use System.out.println, the stdout under userlogs should contain a job's log, right?

But that did not happen after getLocations. It drove me crazy. I have no idea of if an

important method like PigInputFormat.getRecordReader has ever been invoked.

 

Please enlighten me!

 

Thanks,

Richard
 
> Date: Wed, 28 Oct 2009 10:57:54 -0400
> Subject: Re: Custom Loadfunc problem!
> From: dvryaboy@gmail.com
> To: pig-dev@hadoop.apache.org
> 
> Richard,
> I would strongly advise you against trying to mess with PigInputFormat
> and JobControlCompiler. We know that slicers work (after all, people
> manage to write hbase loaders and such). Plus the whole
> input/output/slice/slicer thing is being revised right now, there is a
> proposal in PIG-966 you can look at if you would like to comment.
> 
> In fact I just took your code and substituted some placeholders for
> various methods and classes (like HSSchema, etc), and I can see Slice
> methods getting called and everything.
> 
> What do your location strings look like? Pig tries to be helpful by
> transforming locations into HDFS paths if the locations are not
> qualified by a scheme ( something:/ ). This makes file-based LoadFuncs
> easier to write, but it also means that you have to specify a scheme
> if you are doing something that's not file-based. Try changing your
> code to expect the location string to start with some scheme, such as
> abcd:/ (or perhaps something more meaningful) and things should work.
> 
> -D
> 
> 2009/10/28 RichardGUO Fei <gladiatorcn@hotmail.com>:
> >
> > It seems that the real problem is JobControlCompiler.getJob, because this method
removes POLoad and sets PigInputFormat as the input format but the path passed to my storage
is actually a table name rather than a file path! I think that the LoadFunc should be more
flexible. I am trying to fix this at this moment.
> >
> >
> >
> > Thanks,
> >
> > Richard
> >
> >> From: zjffdu@gmail.com
> >> To: pig-dev@hadoop.apache.org
> >> Subject: RE: Custom Loadfunc problem!
> >> Date: Tue, 27 Oct 2009 23:40:43 -0800
> >>
> >>
> >> I mean hadoop's local mode not pig's own local mode
> >>
> >>
> >> -----Original Message-----
> >> From: Dmitriy Ryaboy [mailto:dvryaboy@gmail.com]
> >> Sent: 2009年10月26日 6:33
> >> To: pig-dev@hadoop.apache.org; pig-dev@hadoop.apache.org
> >> Subject: RE: Custom Loadfunc problem!
> >>
> >> Jeff,
> >> Slicers dont work in local mode, there is an ancient ticket for that on the
> >> Jira.
> >>
> >> Richard -- hard to say whats going on without more code. Think you can come
> >> up with a simplified version of your loadfunc that fails in a similar
> >> manner, and share it?
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: "zjffdu" <zjffdu@gmail.com>
> >> To: pig-dev@hadoop.apache.org
> >> Sent: 10/27/2009 1:45 AM
> >> Subject: RE: Custom Loadfunc problem!
> >>
> >> Illustrate will not execute the job, while dump and store will execute it.
> >> So I think there must be something wrong with your custom slicer. I suggest
> >> you set breakpoint in your slicer and debug it in map reduce mode locally
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: RichardGUO Fei [mailto:gladiatorcn@hotmail.com]
> >> Sent: 20091026 0:43
> >> To: pig-dev@hadoop.apache.org
> >> Subject: RE: Custom Loadfunc problem!
> >>
> >>
> >> Hi,
> >>
> >>
> >>
> >> Btw, my program works with ILLUSTRATE but n
> >>
> >> [truncated by sender]
> >>
> >
> > _________________________________________________________________
> > MSN十周年庆典,查看MSN注册时间,赢取神秘大奖
> > http://10.msn.com.cn
 		 	   		  
_________________________________________________________________
约会说不清地方?来试试微软地图最新msn互动功能!
http://ditu.live.com/?form=TL&swm=1
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message