hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rottinghuis, Joep" <jrottingh...@ebay.com>
Subject RE: Compiling programs with Hadoop 0.21.0
Date Fri, 02 Sep 2011 03:13:47 GMT
Hadoop-hdfs will contain the HDFS (Hadoop FileSystem) classes.
In your code you point to hdfs:// in your path, therefore you use HDFS, hence they are needed.

Add the hdfs jar to your classpath and see if that improves things.

Iif you organize imports on you class in an IDE such as Eclipse you'll see that when you restrict
the org.apache.hadoop.* import only to packages you need, that indeed you are using hdfs classes.

Thanks,

Joep

From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: Wednesday, August 31, 2011 5:40 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Compiling programs with Hadoop 0.21.0

Hi,

Thanks for the kind reply.

If only the common and mapred are used, then what is hadoop-hdfs-0.21.0.jar used for?

My code looks like this:


import java.util.*;

import java.lang.*;

import java.io.*;

import org.apache.hadoop.*;



class MapReduce  {



public static void main(String[] args) throws Exception  {

        try  {

                Path pt=new Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");

                FileSystem fs = FileSystem.get(new Configuration());

                BufferedWriter br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));

                String line;

                line="Testing";

                System.out.println(line);

                br.write(line);

                br.close();

        }catch(Exception e){

                System.out.println("File not found");

        }

    }

}

When I compile the code, I get the following erros:
$javac -classpath /Users/arko/Documents/hadoop-0.21.0/hadoop-common-0.21.0.jar -d class/ FileSystemCat.java
clear

FileTest.java:16: cannot find symbol
symbol  : class Path
location: class FileTest
                Path pt=new Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
                ^
FileTest.java:16: cannot find symbol
symbol  : class Path
location: class FileTest
                Path pt=new Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
                            ^
FileTest.java:17: java.io.FileSystem is not public in java.io<http://java.io>; cannot
be accessed from outside package
                FileSystem fs = FileSystem.get(new Configuration());
                ^
FileTest.java:17: cannot find symbol
symbol  : class Configuration
location: class FileTest
                FileSystem fs = FileSystem.get(new Configuration());
                                                   ^
FileTest.java:17: java.io.FileSystem is not public in java.io<http://java.io>; cannot
be accessed from outside package
                FileSystem fs = FileSystem.get(new Configuration());
                                ^
5 errors

Thanks again for help!
Warm Regards
Arko
On Wed, Aug 31, 2011 at 5:00 PM, Robert Evans <evans@yahoo-inc.com<mailto:evans@yahoo-inc.com>>
wrote:
You should be able to use hadoop-common-0.21.0.jar for accessing HDFS APIS and use hadoop-mapred.0.21.0.jar
for accessing the mapreduce APIs.  I cannot really comment further on compilation errors without
seeing the code/error messages.

--Bobby Evans


On 8/31/11 4:34 PM, "Arko Provo Mukherjee" <arkoprovomukherjee@gmail.com<http://arkoprovomukherjee@gmail.com>>
wrote:
Hello,

I am trying to learn Hadoop and doing a project on it.

I need to update some files in my project and hence wanted to use version 0.21.0

However, I am confused as to how I can compile my programs on version 0.21.0 as it doesn't
have any hadoop-core-0.21.0.jar file. What option should I have to give in the -classpath
option?
I can see three different JAR files namely hadoop-common-0.21.0.jar, hadoop-hdfs-0.21.0.jar
& hadoop-mapred-0.21.0.jar.

I am getting compilation errors which using the the common jar file.

Also I would like to learn the FileContext Library as the docs that I have read says that
it has a simpler interface than the FileSystem. However, I cannot find any links to some example
code which can help me to grasp the API. Request you to kindly share any link / code snippet
to demonstrate the use of the new API.

Many thanks in advance for your kind response.

Warm regards
Arko


Mime
View raw message