hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From brien colwell <xcolw...@gmail.com>
Subject Re: Random weirdness — anyone explain, please?
Date Thu, 22 Oct 2009 14:56:36 GMT
When I debug Hadoop locally in Eclipse I run jobs with the following 
configurations. I access the HDFS with paths like below . It seems to me 
this should be the same as what you have so I also suspect a classpath 
issue.

        baseConf.set("fs.default.name", "file:///");

Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
FileSystem fs = p.getFileSystem(baseConf) .





Bogdan M. Maryniuk wrote:
> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
>         this.conf = new Configuration();
>         this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
>         this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <bogdan.maryniuk@gmail.com> wrote:
>   
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:247)
>>        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>>        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>>        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>>        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>>      ) throws IOException {
>>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>>    if (clazz == null) {
>>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>>    }
>>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>>    fs.initialize(uri, conf);
>>    return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>     
>
>
>
>   


Mime
View raw message