hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Gauthier <alexgauthie...@gmail.com>
Subject Re: writing to hdfs via java api
Date Sat, 29 Oct 2011 03:43:01 GMT
On Fri, Oct 28, 2011 at 8:41 PM, Tom Melendez <tom@supertom.com> wrote:

> Hi Jay,
>
> Are you able to look at the logs or the web interface?  Can you find
> out why it's getting killed?
>
> Also, can you verify that these ports are open and a process is
> connected to them (maybe with netstat)?
>
> http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/
>
> Thanks,
>
> Tom
>
> On Fri, Oct 28, 2011 at 7:57 PM, Jay Vyas <jayunit100@gmail.com> wrote:
> > Thanks tom : Thats interesting....
> >
> > First, I tried, and it complained that the input directory didnt exist,
> so I
> > ran
> > $> hadoop fs -mkdir /user/cloudera/input
> >
> > Then, I tried to do this :
> >
> > $> hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar grep input output2
> > 'dfs[a-z.]+'
> >
> > And it seemed to start working ...... But then it abruptly printed
> "killed"
> > somehow at the end of the job [scroll down] ?
> >
> > Maybe this is related to why i cant connect ..... ?!
> >
> > 1) the hadoop jar 11/10/14 21:34:43 WARN util.NativeCodeLoader: Unable to
> > load native-hadoop library for your platform... using builtin-java
> classes
> > where applicable
> > 11/10/14 21:34:43 WARN snappy.LoadSnappy: Snappy native library not
> loaded
> > 11/10/14 21:34:43 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 0
> > 11/10/14 21:34:44 INFO mapred.JobClient: Running job:
> job_201110142010_0009
> > 11/10/14 21:34:45 INFO mapred.JobClient:  map 0% reduce 0%
> > 11/10/14 21:34:55 INFO mapred.JobClient:  map 0% reduce 100%
> > 11/10/14 21:34:57 INFO mapred.JobClient: Job complete:
> job_201110142010_0009
> > 11/10/14 21:34:57 INFO mapred.JobClient: Counters: 14
> > 11/10/14 21:34:57 INFO mapred.JobClient:   Job Counters
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Launched reduce tasks=1
> > 11/10/14 21:34:57 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5627
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Total time spent by all
> reduces
> > waiting after reserving slots (ms)=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Total time spent by all maps
> > waiting after reserving slots (ms)=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=5050
> > 11/10/14 21:34:57 INFO mapred.JobClient:   FileSystemCounters
> > 11/10/14 21:34:57 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=53452
> > 11/10/14 21:34:57 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=86
> > 11/10/14 21:34:57 INFO mapred.JobClient:   Map-Reduce Framework
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce input groups=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Combine output records=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce shuffle bytes=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce output records=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Spilled Records=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Combine input records=0
> > 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce input records=0
> > 11/10/14 21:34:57 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> > 11/10/14 21:34:58 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 11/10/14 21:34:58 INFO mapred.JobClient: Running job:
> job_201110142010_0010
> > 11/10/14 21:34:59 INFO mapred.JobClient:  map 0% reduce 0%
> > Killed
> >
> >
> > On Fri, Oct 28, 2011 at 8:24 PM, Tom Melendez <tom@supertom.com> wrote:
> >
> >> Hi Jay,
> >>
> >> Some questions for you:
> >>
> >> - Does the hadoop client itself work from that same machine?
> >> - Are you actually able to run the hadoop example jar (in other words,
> >> your setup is valid otherwise)?
> >> - Is port 8020 actually available?  (you can telnet or nc to it?)
> >> - What does jps show on the namenode?
> >>
> >> Thanks,
> >>
> >> Tom
> >>
> >> On Fri, Oct 28, 2011 at 4:04 PM, Jay Vyas <jayunit100@gmail.com> wrote:
> >> > Hi guys : Made more progress debugging my hadoop connection, but still
> >> > haven't got it working......  It looks like my VM (cloudera hadoop)
> won't
> >> > let me in.  I find that there is no issue connecting to the name node
> -
> >> that
> >> > is , using hftp and 50070......
> >> >
> >> > via standard HFTP as in here :
> >> >
> >> > //This method works fine - connecting directly to hadoop's namenode
> and
> >> > querying the filesystem
> >> > public static void main1(String[] args) throws Exception
> >> >    {
> >> >        String uri = "hftp://155.37.101.76:50070/";
> >> >
> >> >        System.out.println( "uri: " + uri );
> >> >        Configuration conf = new Configuration();
> >> >
> >> >        FileSystem fs = FileSystem.get( URI.create( uri ), conf );
> >> >        fs.printStatistics();
> >> >    }
> >> >
> >> >
> >> > But unfortunately, I can't get into hdfs ..... Any thoughts on this ?
>  I
> >> am
> >> > modifying the uri to access port 8020
> >> > which is what is in my core-site.xml .
> >> >
> >> >   // This fails, resulting (trys to connect over and over again,
> >> eventually
> >> > gives up printing "already tried to connect 20 times"....)
> >> >    public static void main(String[] args)
> >> >    {
> >> >        try {
> >> >            String uri = "hdfs://155.37.101.76:8020/";
> >> >
> >> >            System.out.println( "uri: " + uri );
> >> >            Configuration conf = new Configuration();
> >> >
> >> >            FileSystem fs = FileSystem.get( URI.create( uri ), conf );
> >> >            fs.printStatistics();
> >> >        } catch (Exception e) {
> >> >            // TODO Auto-generated catch block
> >> >            e.printStackTrace();
> >> >        }
> >> >    }
> >> >
> >> > The error message is :
> >> >
> >> > 11/10/28 19:03:38 INFO ipc.Client: Retrying connect to server: /
> >> > 155.37.101.76:8020. Already tried 0 time(s).
> >> > 11/10/28 19:03:39 INFO ipc.Client: Retrying connect to server: /
> >> > 155.37.101.76:8020. Already tried 1 time(s).
> >> > 11/10/28 19:03:40 INFO ipc.Client: Retrying connect to server: /
> >> > 155.37.101.76:8020. Already tried 2 time(s).
> >> > 11/10/28 19:03:41 INFO ipc.Client: Retrying connect to server: /
> >> > 155.37.101.76:8020. Already tried 3 time(s).
> >> >
> >> > Any thoughts on this would be *really* be appreciated  ... Thanks
> guys.
> >> >
> >>
> >
> >
> >
> > --
> > Jay Vyas
> > MMSB/UCHC
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message