hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subroto Sanyal <subrotosan...@huawei.com>
Subject RE: Compiling programs with Hadoop 0.21.0
Date Fri, 02 Sep 2011 05:35:35 GMT
Dear Arko,

 

The class org.apache.hadoop.fs.Path is available in the
hadoop-common-0.21.0.jar.

>From your code I can see you are trying to create a file in HDFS and write
into it.

 

The code snippet needs the hdfs and common jar for compiling.

 

Regards, 
Subroto Sanyal

  _____  

From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com] 
Sent: Friday, September 02, 2011 10:18 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Compiling programs with Hadoop 0.21.0

 

Hello,

 

Thanks for your reply. 

Following your suggestion, I tried to compile using the following command:
$ javac -classpath
/Users/arko/Documents/hadoop-0.21.0/hadoop-hdfs-0.21.0.jar  -d class/
FileTest.java

I got the following errors:
FileTest.java:9: package org.apache.hadoop does not exist
import org.apache.hadoop.*;
^
FileTest.java:16: cannot find symbol
symbol  : class Path
location: class FileTest
               Path pt=new
Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
               ^
FileTest.java:16: cannot find symbol
symbol  : class Path
location: class FileTest
               Path pt=new
Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
                           ^
FileTest.java:17: java.io.FileSystem is not public in java.io; cannot be
accessed from outside package
               FileSystem fs = FileSystem.get(new Configuration());
               ^
FileTest.java:17: cannot find symbol
symbol  : class Configuration
location: class FileTest
               FileSystem fs = FileSystem.get(new Configuration());
                                                  ^
FileTest.java:17: java.io.FileSystem is not public in java.io; cannot be
accessed from outside package
               FileSystem fs = FileSystem.get(new Configuration());
                               ^
6 errors

Many thanks in advance!
Warm regards

Arko

On Thu, Sep 1, 2011 at 10:13 PM, Rottinghuis, Joep <jrottinghuis@ebay.com>
wrote:
> Hadoop-hdfs will contain the HDFS (Hadoop FileSystem) classes.
>
> In your code you point to hdfs:// in your path, therefore you use HDFS,
> hence they are needed.
>
>  
>
> Add the hdfs jar to your classpath and see if that improves things.
>
>  
>
> Iif you organize imports on you class in an IDE such as Eclipse you'll see
> that when you restrict the org.apache.hadoop.* import only to packages you
> need, that indeed you are using hdfs classes.
>
>  
>
> Thanks,
>
>  
>
> Joep
>
>  
>
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: Wednesday, August 31, 2011 5:40 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: Compiling programs with Hadoop 0.21.0
>
>  
>
> Hi,
>
>  
>
> Thanks for the kind reply.
>
>  
>
> If only the common and mapred are used, then what is
hadoop-hdfs-0.21.0.jar
> used for?
>
>  
>
> My code looks like this:
>
>  
>
> import java.util.*;
>
> import java.lang.*;
>
> import java.io.*;
>
> import org.apache.hadoop.*;
>
>  
>
> class MapReduce  {
>
>     
>
> public static void main(String[] args) throws Exception  {
>
>         try  {
>
>                 Path pt=new
> Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
>
>                 FileSystem fs = FileSystem.get(new Configuration());
>
>                 BufferedWriter br=new BufferedWriter(new
> OutputStreamWriter(fs.create(pt,true)));
>
>                 String line;
>
>                 line="Testing";
>
>                 System.out.println(line);
>
>                 br.write(line);
>
>                 br.close();
>
>         }catch(Exception e){
>
>                 System.out.println("File not found");
>
>         }
>
>     }
>
> }
>
>  
>
> When I compile the code, I get the following erros:
>
> $javac -classpath
> /Users/arko/Documents/hadoop-0.21.0/hadoop-common-0.21.0.jar -d class/
> FileSystemCat.java clear
>
>  
>
> FileTest.java:16: cannot find symbol
>
> symbol  : class Path
>
> location: class FileTest
>
>                 Path pt=new
> Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
>
>                 ^
>
> FileTest.java:16: cannot find symbol
>
> symbol  : class Path
>
> location: class FileTest
>
>                 Path pt=new
> Path("hdfs://localhost:54310//Users/arko/Documents/Research/HDFS/abc");
>
>                             ^
>
> FileTest.java:17: java.io.FileSystem is not public in java.io; cannot be
> accessed from outside package
>
>                 FileSystem fs = FileSystem.get(new Configuration());
>
>                 ^
>
> FileTest.java:17: cannot find symbol
>
> symbol  : class Configuration
>
> location: class FileTest
>
>                 FileSystem fs = FileSystem.get(new Configuration());
>
>                                                    ^
>
> FileTest.java:17: java.io.FileSystem is not public in java.io; cannot be
> accessed from outside package
>
>                 FileSystem fs = FileSystem.get(new Configuration());
>
>                                 ^
>
> 5 errors
>
>  
>
> Thanks again for help!
> Warm Regards
>
> Arko
>
> On Wed, Aug 31, 2011 at 5:00 PM, Robert Evans <evans@yahoo-inc.com> wrote:
>
> You should be able to use hadoop-common-0.21.0.jar for accessing HDFS APIS
> and use hadoop-mapred.0.21.0.jar for accessing the mapreduce APIs.  I
cannot
> really comment further on compilation errors without seeing the code/error
> messages.
>
> --Bobby Evans
>
> On 8/31/11 4:34 PM, "Arko Provo Mukherjee" <arkoprovomukherjee@gmail.com>
> wrote:
>
> Hello,
>
> I am trying to learn Hadoop and doing a project on it.
>
> I need to update some files in my project and hence wanted to use version
> 0.21.0
>
> However, I am confused as to how I can compile my programs on version
0.21.0
> as it doesn't have any hadoop-core-0.21.0.jar file. What option should I
> have to give in the -classpath option?
> I can see three different JAR files namely hadoop-common-0.21.0.jar,
> hadoop-hdfs-0.21.0.jar & hadoop-mapred-0.21.0.jar.
>
> I am getting compilation errors which using the the common jar file. 
>
> Also I would like to learn the FileContext Library as the docs that I have
> read says that it has a simpler interface than the FileSystem. However, I
> cannot find any links to some example code which can help me to grasp the
> API. Request you to kindly share any link / code snippet to demonstrate
the
> use of the new API.
>
> Many thanks in advance for your kind response.
>
> Warm regards
> Arko
>
>  


Mime
View raw message