hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dcave555 <dcave...@gmail.com>
Subject Re: writing file
Date Sat, 13 Oct 2007 21:53:05 GMT

Thx again for your reply

Ok maybe you right

I have my ubuntu hadoop node on ip 168.13.51.13

So i write in hadoop-site.xml (windows application)

<configuration>  
 <property>
   <name>hadoop.tmp.dir</name>
   <value>//168.13.51.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
   <description>A base for other temporary directories.</description>
 </property>
 
 <property>
   <name>fs.default.name</name>
   <value>hdfs://168.13.51.13:54310</value>  
 </property>
  
 <property>
   <name>mapred.job.tracker</name>
   <value>168.13.51.13:54311</value>
 </property>
  
   <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>
</configuration>


The same xml i have on ubuntu machine.


after running my application i have exception:

Login failded.

i suspect that i have problem with ssh public key

so i think i need to copy ssh public key to windows machine

but how can i tell to my application to use this key?

Thx again


Ted Dunning-3 wrote:
> 
> 
> Your program may be trying to run locally on your windows box instead of
> talking to the ubuntu micro cluster.
> 
> 
> On 10/13/07 11:43 AM, "dcave555" <dcave555@gmail.com> wrote:
> 
>> 
>> Thx for your reply
>> 
>> But i think i don't understand something.
>> i have installed hadoop single node on ubuntu.
>> my test is running on windows.
>> 
>> i changed property in hadoop-site.xml in my application to
>> 
>> <property>
>>    <name>hadoop.tmp.dir</name>
>>    <value>//172.16.50.13/usr/local/hadoop-datastore/hadoop-hadoop</value>
>>    <description>A base for other temporary directories.</description>
>> </property>
>> 
>> amd i am getting Login failed.
>> 
>> If i want to save file from windows to hadoop on ubuntu i need to import
>> ssh
>> public key?
>> how can i tell my test application on windows to use this pulic key?
>> 
>> Or i need to do something else?
>> 
>> Thx
>> 
>> 
>> 
>> 
>> 
>> Ted Dunning-3 wrote:
>>> 
>>> 
>>> Hadoop is having trouble spawning a process to measure how much disk
>>> space
>>> you have available.
>>> 
>>> It really is much easier to run hadoop on Linux.  In case you can't do
>>> that,
>>> make sure that you really do have cygwin installed in a standard way on
>>> each
>>> hadoop node.
>>> 
>>> 
>>> On 10/13/07 8:05 AM, "dcave555" <dcave555@gmail.com> wrote:
>>> 
>>>> 
>>>> Hello all
>>>> 
>>>> I am new to hadoop .
>>>> 
>>>> I am trying to write file to single cluster and getting this exception
>>>> when
>>>> i am trying to close output stream
>>>> 
>>>> java.io.IOException: CreateProcess: df -k
>>>> C:\usr\local\hadoop-datastore\hadoop-hadoop\dfs\tmp error=2
>>>> at java.lang.ProcessImpl.create(Native Method)
>>>> at java.lang.ProcessImpl.<init>(Unknown Source)
>>>> at java.lang.ProcessImpl.start(Unknown Source)
>>>> at java.lang.ProcessBuilder.start(Unknown Source)
>>>> at java.lang.Runtime.exec(Unknown Source)
>>>> at java.lang.Runtime.exec(Unknown Source)
>>>> at org.apache.hadoop.fs.DF.doDF(DF.java:60)
>>>> at org.apache.hadoop.fs.DF.<init>(DF.java:53)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(Local
>>>> Di
>>>> rAllocator.java:198)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWr
>>>> it
>>>> e(LocalDirAllocator.java:235)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForW
>>>> ri
>>>> te(LocalDirAllocator.java:276)
>>>> at
>>>> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocat
>>>> or
>>>> .java:155)
>>>> at
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java
>>>> :1
>>>> 475)
>>>> at
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.openBackupStream(DFSClient.j
>>>> av
>>>> a:1442)
>>>> at
>>>> 
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:160
>>> 0>
>>> )
>>>> at
>>>> 
>>> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:14
>>> 0>
>>> )
>>>> at
>>>> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:122)
>>>> at
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1739)
>>>> at
>>>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStre
>>>> am
>>>> .java:49)
>>>> at
>>>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:64)
>>>> at Test1.main(Test1.java:23)
>>>> 
>>>> 
>>>> 
>>>> My test is:
>>>> 
>>>>                         Configuration configuration = new
>>>> Configuration();
>>>>                         FileSystem fileSystem =
>>>> FileSystem.get(configuration);
>>>>                         Path path = new Path("/testfile");
>>>> //writing:
>>>> FSDataOutputStream dataOutputStream = fileSystem.create(path);
>>>> dataOutputStream.writeUTF("hello world");
>>>> dataOutputStream.close();
>>>> //reading 
>>>> FSDataInputStream dataInputStream = fileSystem.open(path);
>>>> System.out.println(dataInputStream.readUTF());
>>>> dataInputStream.close();
>>>> fileSystem.close();
>>>> 
>>>> i added hadoop-site.xml to classpath :
>>>> 
>>>> 
>>>> <configuration>
>>>>  
>>>> <property>
>>>>   <name>hadoop.tmp.dir</name>
>>>>   <value>/usr/local/hadoop-datastore/hadoop-hadoop</value>
>>>>   <description>A base for other temporary directories.</description>
>>>> </property>
>>>> 
>>>> <property>
>>>>   <name>fs.default.name</name>
>>>>   <value>hdfs://172.16.50.13:54310</value>
>>>>   <description>The name of the default file system.  A URI whose
>>>>   scheme and authority determine the FileSystem implementation.  The
>>>>   uri's scheme determines the config property (fs.SCHEME.impl) naming
>>>>   the FileSystem implementation class.  The uri's authority is used to
>>>>   determine the host, port, etc. for a filesystem.</description>
>>>> </property>
>>>>  
>>>> <property>
>>>>   <name>mapred.job.tracker</name>
>>>>   <value>172.16.50.13:54311</value>
>>>>   <description>The host and port that the MapReduce job tracker runs
>>>>   at.  If "local", then jobs are run in-process as a single map
>>>>   and reduce task.
>>>>   </description>
>>>> </property>
>>>>  
>>>> <property>
>>>>   <name>dfs.replication</name>
>>>>   <value>1</value>
>>>>   <description>Default block replication.
>>>>   The actual number of replications can be specified when the file is
>>>> created.
>>>>   The default is used if replication is not specified in create time.
>>>>   </description>
>>>> </property>
>>>> </configuration>
>>>> 
>>>> 
>>>> 
>>>> Please help me
>>>> thx
>>> 
>>> 
>>> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/writing-file-tf4618565.html#a13193822
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message