hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JAX <jayunit...@gmail.com>
Subject Re: writing to hdfs via java api
Date Sat, 29 Oct 2011 04:16:22 GMT
Yup.... Brutal :-|
but you never regret fixing a bug   ... Unlike -------

Sent from my iPad

On Oct 28, 2011, at 11:43 PM, Alex Gauthier <alexgauthier24@gmail.com> wrote:

> Brutal Friday night.  Coding < pussy.
> 
> :)
> 
> On Fri, Oct 28, 2011 at 8:43 PM, Alex Gauthier <alexgauthier24@gmail.com>wrote:
> 
>> 
>> 
>> On Fri, Oct 28, 2011 at 8:41 PM, Tom Melendez <tom@supertom.com> wrote:
>> 
>>> Hi Jay,
>>> 
>>> Are you able to look at the logs or the web interface?  Can you find
>>> out why it's getting killed?
>>> 
>>> Also, can you verify that these ports are open and a process is
>>> connected to them (maybe with netstat)?
>>> 
>>> http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/
>>> 
>>> Thanks,
>>> 
>>> Tom
>>> 
>>> On Fri, Oct 28, 2011 at 7:57 PM, Jay Vyas <jayunit100@gmail.com> wrote:
>>>> Thanks tom : Thats interesting....
>>>> 
>>>> First, I tried, and it complained that the input directory didnt exist,
>>> so I
>>>> ran
>>>> $> hadoop fs -mkdir /user/cloudera/input
>>>> 
>>>> Then, I tried to do this :
>>>> 
>>>> $> hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar grep input
>>> output2
>>>> 'dfs[a-z.]+'
>>>> 
>>>> And it seemed to start working ...... But then it abruptly printed
>>> "killed"
>>>> somehow at the end of the job [scroll down] ?
>>>> 
>>>> Maybe this is related to why i cant connect ..... ?!
>>>> 
>>>> 1) the hadoop jar 11/10/14 21:34:43 WARN util.NativeCodeLoader: Unable
>>> to
>>>> load native-hadoop library for your platform... using builtin-java
>>> classes
>>>> where applicable
>>>> 11/10/14 21:34:43 WARN snappy.LoadSnappy: Snappy native library not
>>> loaded
>>>> 11/10/14 21:34:43 INFO mapred.FileInputFormat: Total input paths to
>>> process
>>>> : 0
>>>> 11/10/14 21:34:44 INFO mapred.JobClient: Running job:
>>> job_201110142010_0009
>>>> 11/10/14 21:34:45 INFO mapred.JobClient:  map 0% reduce 0%
>>>> 11/10/14 21:34:55 INFO mapred.JobClient:  map 0% reduce 100%
>>>> 11/10/14 21:34:57 INFO mapred.JobClient: Job complete:
>>> job_201110142010_0009
>>>> 11/10/14 21:34:57 INFO mapred.JobClient: Counters: 14
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:   Job Counters
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Launched reduce tasks=1
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5627
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Total time spent by all
>>> reduces
>>>> waiting after reserving slots (ms)=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Total time spent by all
>>> maps
>>>> waiting after reserving slots (ms)=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=5050
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:   FileSystemCounters
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=53452
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=86
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:   Map-Reduce Framework
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce input groups=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Combine output records=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce shuffle bytes=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce output records=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Spilled Records=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Combine input records=0
>>>> 11/10/14 21:34:57 INFO mapred.JobClient:     Reduce input records=0
>>>> 11/10/14 21:34:57 WARN mapred.JobClient: Use GenericOptionsParser for
>>>> parsing the arguments. Applications should implement Tool for the same.
>>>> 11/10/14 21:34:58 INFO mapred.FileInputFormat: Total input paths to
>>> process
>>>> : 1
>>>> 11/10/14 21:34:58 INFO mapred.JobClient: Running job:
>>> job_201110142010_0010
>>>> 11/10/14 21:34:59 INFO mapred.JobClient:  map 0% reduce 0%
>>>> Killed
>>>> 
>>>> 
>>>> On Fri, Oct 28, 2011 at 8:24 PM, Tom Melendez <tom@supertom.com> wrote:
>>>> 
>>>>> Hi Jay,
>>>>> 
>>>>> Some questions for you:
>>>>> 
>>>>> - Does the hadoop client itself work from that same machine?
>>>>> - Are you actually able to run the hadoop example jar (in other words,
>>>>> your setup is valid otherwise)?
>>>>> - Is port 8020 actually available?  (you can telnet or nc to it?)
>>>>> - What does jps show on the namenode?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Tom
>>>>> 
>>>>> On Fri, Oct 28, 2011 at 4:04 PM, Jay Vyas <jayunit100@gmail.com>
>>> wrote:
>>>>>> Hi guys : Made more progress debugging my hadoop connection, but
>>> still
>>>>>> haven't got it working......  It looks like my VM (cloudera hadoop)
>>> won't
>>>>>> let me in.  I find that there is no issue connecting to the name
node
>>> -
>>>>> that
>>>>>> is , using hftp and 50070......
>>>>>> 
>>>>>> via standard HFTP as in here :
>>>>>> 
>>>>>> //This method works fine - connecting directly to hadoop's namenode
>>> and
>>>>>> querying the filesystem
>>>>>> public static void main1(String[] args) throws Exception
>>>>>>   {
>>>>>>       String uri = "hftp://155.37.101.76:50070/";
>>>>>> 
>>>>>>       System.out.println( "uri: " + uri );
>>>>>>       Configuration conf = new Configuration();
>>>>>> 
>>>>>>       FileSystem fs = FileSystem.get( URI.create( uri ), conf );
>>>>>>       fs.printStatistics();
>>>>>>   }
>>>>>> 
>>>>>> 
>>>>>> But unfortunately, I can't get into hdfs ..... Any thoughts on this
?
>>> I
>>>>> am
>>>>>> modifying the uri to access port 8020
>>>>>> which is what is in my core-site.xml .
>>>>>> 
>>>>>>  // This fails, resulting (trys to connect over and over again,
>>>>> eventually
>>>>>> gives up printing "already tried to connect 20 times"....)
>>>>>>   public static void main(String[] args)
>>>>>>   {
>>>>>>       try {
>>>>>>           String uri = "hdfs://155.37.101.76:8020/";
>>>>>> 
>>>>>>           System.out.println( "uri: " + uri );
>>>>>>           Configuration conf = new Configuration();
>>>>>> 
>>>>>>           FileSystem fs = FileSystem.get( URI.create( uri ), conf
);
>>>>>>           fs.printStatistics();
>>>>>>       } catch (Exception e) {
>>>>>>           // TODO Auto-generated catch block
>>>>>>           e.printStackTrace();
>>>>>>       }
>>>>>>   }
>>>>>> 
>>>>>> The error message is :
>>>>>> 
>>>>>> 11/10/28 19:03:38 INFO ipc.Client: Retrying connect to server: /
>>>>>> 155.37.101.76:8020. Already tried 0 time(s).
>>>>>> 11/10/28 19:03:39 INFO ipc.Client: Retrying connect to server: /
>>>>>> 155.37.101.76:8020. Already tried 1 time(s).
>>>>>> 11/10/28 19:03:40 INFO ipc.Client: Retrying connect to server: /
>>>>>> 155.37.101.76:8020. Already tried 2 time(s).
>>>>>> 11/10/28 19:03:41 INFO ipc.Client: Retrying connect to server: /
>>>>>> 155.37.101.76:8020. Already tried 3 time(s).
>>>>>> 
>>>>>> Any thoughts on this would be *really* be appreciated  ... Thanks
>>> guys.
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Jay Vyas
>>>> MMSB/UCHC
>>>> 
>>> 
>> 
>> 

Mime
View raw message