hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peng, Wei" <Wei.P...@xerox.com>
Subject RE: Needs a simple answer
Date Mon, 20 Dec 2010 08:17:30 GMT
Maha,

If you want to access hdfs file in mapper class, use
context.getConfiguration() in FileSystem.get. In the main method, you
need to create configuration first using new configuration().
In your case, you cannot create configuration in main method, and access
it in mapper class because the main method and the mapper are not in the
same jvm.

Wei

-----Original Message-----
From: Hari Sreekumar [mailto:hsreekumar@clickable.com] 
Sent: Sunday, December 19, 2010 8:52 AM
To: common-user@hadoop.apache.org
Subject: Re: Needs a simple answer

You should not be creating new configuration instances in the
mapper/reducer. Instead, get the job configuration using
context.getConfiguration() [API 0.20.2]

On Sun, Dec 19, 2010 at 1:16 PM, maha <maha@umail.ucsb.edu> wrote:

> Thanks Ping :)  actually my mistake was that I used to create a
FileSystem
> in the Mapper class as:
>
>       fs = FileSystem.get( myConf) ;   //  where myConf is defined in
the
> main class WordCount.
>
>  Somehow, that wasn't working .. but as soon as I changed it to be
with new
> Configurations:
>
>  FileSystem fs = FileSystem.get(new Configuration())    //   it worked
!
>
>
>   Any reason for that?
>
>   Thank you,
>       Maha
>
>
>
> On Dec 17, 2010, at 2:59 PM, Peng, Wei wrote:
>
> >
> > You can put your local file to distributed file system by hadoop fs
-put
> localfile DFSfile.
> > Then access it by
> > Configuration conf = new Configuration();
> >               try {
> >                       FileSystem fs =
FileSystem.get(URI.create(DFSfile),
> conf);
> >                       FSDataInputStream in = null;
> >                       in = fs.open(new Path(path));
> >                       String content = in.readUTF();
> >
> >               } catch (IOException e) {
> >
> >               }
> > -----Original Message-----
> > From: maha [mailto:maha@umail.ucsb.edu]
> > Sent: Fri 12/17/2010 2:20 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: Needs a simple answer
> >
> > Hi Ted, thanks for the hint , but I'm using only one machine now ..
so
> the file is definitely in there.
> >
> > So, How can I open an hdfs- file in my InputFormat class (in
GetSplits)
> so I can pass the contents to the mappers ?
> >
> > Now, the splits are splits of files-paths from the input directory (
eg.
> split1: /tmp/f1, split2:/tmp/f2 .... split4: /tmp/f4)  instead I want
----->
>  ( split1: content of file1 , ....).
> >
> > Thank you,
> >
> > Maha
> >
> >
> > On Dec 16, 2010, at 2:49 PM, Ted Dunning wrote:
> >
> >> Maha,
> >>
> >> Remember that the mapper is not running on the same machine as the
main
> >> class.  Thus local files aren't where you think.
> >>
> >> On Thu, Dec 16, 2010 at 1:06 PM, maha <maha@umail.ucsb.edu> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Why the following lines would work in the main class (WordCount)
and
> not
> >>> in Mapper ? even though " myconf " is set in WordCount to point to
the
> >>> getConf() returned object.
> >>>
> >>>           try{
> >>>                  FileSystem hdfs =
FileSystem.get(wc.WordCount.myconf);
> >>>                  hdfs.copyFromLocalFile(new Path("/Users/file"),
new
> >>> Path("/tmp/file"));
> >>>                 }catch(Exception e) {
System.err.print("\nError");}
> >>>
> >>>
> >>> Also, the print statement will never print on console unless it's
in my
> >>> run function..
> >>>
> >>>    Appreciate it :)
> >>>
> >>>      Maha
> >>>
> >>>
> >
> >
>
>

Mime
View raw message