hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pi song" <pi.so...@gmail.com>
Subject Re: A question regarding the execution engine
Date Fri, 07 Mar 2008 12:52:39 GMT
So what about when after the execution is finished? Does the current
implementation also copies output files from HDFS back to local file system?

On Fri, Mar 7, 2008 at 10:35 PM, Craig Macdonald <craigm@dcs.gla.ac.uk>
wrote:

> No idea, sorry. Try on list?
>
> C
>
> pi song wrote:
> > So what about when after the execution is finished? Does the current
> > implementation also copies output files from HDFS back to local file
> > system?
> >
> > On Fri, Mar 7, 2008 at 1:51 AM, Craig Macdonald <craigm@dcs.gla.ac.uk
> > <mailto:craigm@dcs.gla.ac.uk>> wrote:
> >
> >     Hi Pi,
> >
> >     I have a JIRA on this issue: PIG-102
> >     It needs feedback from the community whether it should be a
> >     configuration property or a high-level command.
> >
> >     Craig
> >
> >     Benjamin Reed wrote:
> >     > This uses the FileLocalizer. All file references are sent
> >     through the
> >     > FileLocalizer. If we are doing MAPREDUCE and a file reference
> >     starts with
> >     > file:, we copy it to a temp file in HDFS before we start the job
> >     and use that
> >     > temp file as the input or output of the map reduce job.
> >     >
> >     > ben
> >     >
> >     > On Thursday 06 March 2008 04:07:41 pi song wrote:
> >     >
> >     >> Dear pig-dev mailling-list,
> >     >>
> >     >> I just wanna understand this bit quickly. Below is the code from
> >     >> TestMapReduce.java. As you can see the temp file is created in
> >     local
> >     >> machine but I don't understand how Hadoop MapReduce pick up the
> >     file from
> >     >> local file system rather than HDFS?
> >     >>
> >     >>         PigServer pig = new PigServer(MAPREDUCE);
> >     >>         File tmpFile = File.createTempFile("test", ".txt");
> >     >>         PrintStream ps = new PrintStream(new
> >     FileOutputStream(tmpFile));
> >     >>         for(int i = 0; i < 10; i++) {
> >     >>             ps.println(i+"\t"+i);
> >     >>         }
> >     >>         ps.close();
> >     >>         String query = "foreach (load 'file:"+tmpFile+"')
> >     generate $0,$1;";
> >     >>         System.out.println(query);
> >     >>         pig.registerQuery("asdf_id = " + query);
> >     >>         try {
> >     >>             pig.deleteFile("frog");
> >     >>         } catch(Exception e) {}
> >     >>
> >     >> Cheers,
> >     >> Pi
> >     >>
> >     >
> >     >
> >     >
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message