hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Huy Phan <dac...@gmail.com>
Subject Re: Random weirdness — anyone explain, please?
Date Thu, 22 Oct 2009 04:33:11 GMT
Hi Bogdan,
There's nothing wrong with your code, since you're talking about 
CLASSPATH, I guess you didn't execute your script the right way.
How did you run your script anyway ? With or without hadoop command ?

Bogdan M. Maryniuk wrote:
> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
>         this.conf = new Configuration();
>         this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
>         this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <bogdan.maryniuk@gmail.com> wrote:
>   
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>>        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>        at java.lang.Class.forName0(Native Method)
>>        at java.lang.Class.forName(Class.java:247)
>>        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>>        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>>        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>        at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>>        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>>      ) throws IOException {
>>    Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>>    if (clazz == null) {
>>      throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>>    }
>>    FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>>    fs.initialize(uri, conf);
>>    return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>     
>
>
>
>   


Mime
View raw message