hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandru Calin <alexandrucali...@gmail.com>
Subject Re: File is not written on HDFS after running libhdfs C API
Date Thu, 05 Mar 2015 12:39:44 GMT
DataNode was not starting due to this error : java.io.IOException:
Incompatible clusterIDs in /usr/local/hadoop/hadoop_store/hdfs/datanode:
namenode clusterID = CID-b788c93b-a1d7-4351-bd91-28fdd134e9ba; datanode
clusterID = CID-862f3fad-175e-442d-a06b-d65ac57d64b2

I can't image how this happened, anyway .. I issued this command :
*bin/hdfs namenode -format -clusterId
CID-862f3fad-175e-442d-a06b-d65ac57d64b2*

And that got it started, the file is written correctly.

Thank you very much


On Thu, Mar 5, 2015 at 2:03 PM, Alexandru Calin <alexandrucalin29@gmail.com>
wrote:

> After putting the CLASSPATH initialization in .bashrc it creates the file,
> but it has 0 size and I also get this warnning:
>
> file opened
>
> Wrote 14 bytes
> 15/03/05 14:00:55 WARN hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /tmp/testfile.txt could only be replicated to 0 nodes instead of
> minReplication (=1).  There are 0 datanode(s) running and no node(s) are
> excluded in this operation.
>         at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>         at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> FSDataOutputStream#close error:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /tmp/testfile.txt could only be replicated to 0 nodes instead of
> minReplication (=1).  There are 0 datanode(s) running and no node(s) are
> excluded in this operation.
>         at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>         at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> 15/03/05 14:00:55 ERROR hdfs.DFSClient: Failed to close inode 16393
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /tmp/testfile.txt could only be replicated to 0 nodes instead of
> minReplication (=1).  There are 0 datanode(s) running and no node(s) are
> excluded in this operation.
>         at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>         at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
>         at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
>
>
> *bin/hdfs dfs -ls /tmp* :
> Found 1 items
> -rw-r--r--   1 userpc supergroup          0 2015-03-05 14:00
> /tmp/testfile.txt
>
>
>
> On Thu, Mar 5, 2015 at 11:10 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
>> You don't need to start Yarn if you only want to write HDFS using C API.
>> and you also don't need to restart HDFS.
>>
>>
>>
>> On Thu, Mar 5, 2015 at 4:58 PM, Alexandru Calin <
>> alexandrucalin29@gmail.com> wrote:
>>
>>> Now I've also started YARN ( just for the sake of trying anything), the
>>> config for mapred-site.xml and yarn-site.xml are those on apache website. A *jps
>>> *command shows:
>>>
>>> 11257 NodeManager
>>> 11129 ResourceManager
>>> 11815 Jps
>>> 10620 NameNode
>>> 10966 SecondaryNameNode
>>>
>>> On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>
>>>> Can you share your core-site.xml here?
>>>>
>>>>
>>>> On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin <
>>>> alexandrucalin29@gmail.com> wrote:
>>>>
>>>>> No change at all, I've added them at the start and end of the
>>>>> CLASSPATH, either way it still writes the file on the local fs. I've
also
>>>>> restarted hadoop.
>>>>>
>>>>> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu <azuryyyu@gmail.com>
wrote:
>>>>>
>>>>>> Yes,  you should do it:)
>>>>>>
>>>>>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
>>>>>> alexandrucalin29@gmail.com> wrote:
>>>>>>
>>>>>>> Wow, you are so right! it's on the local filesystem!  Do I have
to
>>>>>>> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH
variable
>>>>>>> ? Like this:
>>>>>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
>>>>>>> ?
>>>>>>>
>>>>>>> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu <azuryyyu@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> you need to include core-site.xml as well. and I think you
can find
>>>>>>>> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>>>>>>>>
>>>>>>>> if so,  My guess is right.  because you don't include
>>>>>>>> core-site.xml, then your Filesystem schema is file:// by
default, not
>>>>>>>> hdfs://.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
>>>>>>>> alexandrucalin29@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I am trying to run the basic libhdfs example, it compiles
ok, and
>>>>>>>>> actually runs ok, and executes the whole program, but
I cannot see the file
>>>>>>>>> on the HDFS.
>>>>>>>>>
>>>>>>>>> It is said  here
>>>>>>>>> <http://hadoop.apache.org/docs/r1.2.1/libhdfs.html>,
that you
>>>>>>>>> have to include *the right configuration directory containing
>>>>>>>>> hdfs-site.xml*
>>>>>>>>>
>>>>>>>>> My hdfs-site.xml:
>>>>>>>>>
>>>>>>>>> <configuration>
>>>>>>>>>     <property>
>>>>>>>>>         <name>dfs.replication</name>
>>>>>>>>>         <value>1</value>
>>>>>>>>>     </property>
>>>>>>>>>     <property>
>>>>>>>>>       <name>dfs.namenode.name.dir</name>
>>>>>>>>>       <value>file:///usr/local/hadoop/hadoop_data/hdfs/namenode</value>
>>>>>>>>>     </property>
>>>>>>>>>     <property>
>>>>>>>>>       <name>dfs.datanode.data.dir</name>
>>>>>>>>>       <value>file:///usr/local/hadoop/hadoop_store/hdfs/datanode</value>
>>>>>>>>>     </property></configuration>
>>>>>>>>>
>>>>>>>>> I generate my classpath with this:
>>>>>>>>>
>>>>>>>>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>>>>>>>>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for
subdir in "${subdirs[@]}"do
>>>>>>>>>         for file in $(find /usr/local/hadoop/share/hadoop/$subdir
-name *.jar)
>>>>>>>>>         do
>>>>>>>>>                 export CLASSPATH=$CLASSPATH:$file
>>>>>>>>>         donedone
>>>>>>>>>
>>>>>>>>> and I also add export
>>>>>>>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where
my
>>>>>>>>> *hdfs-site.xml* reside.
>>>>>>>>>
>>>>>>>>> MY LD_LIBRARY_PATH =
>>>>>>>>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>>>>>>>>> Code:
>>>>>>>>>
>>>>>>>>> #include "hdfs.h"#include <stdio.h>#include <string.h>#include
<stdio.h>#include <stdlib.h>
>>>>>>>>> int main(int argc, char **argv) {
>>>>>>>>>
>>>>>>>>>     hdfsFS fs = hdfsConnect("default", 0);
>>>>>>>>>     const char* writePath = "/tmp/testfile.txt";
>>>>>>>>>     hdfsFile writeFile = hdfsOpenFile(fs, writePath,
O_WRONLY|O_CREAT, 0, 0, 0);
>>>>>>>>>     if(!writeFile) {
>>>>>>>>>           printf("Failed to open %s for writing!\n",
writePath);
>>>>>>>>>           exit(-1);
>>>>>>>>>     }
>>>>>>>>>     printf("\nfile opened\n");
>>>>>>>>>     char* buffer = "Hello, World!";
>>>>>>>>>     tSize num_written_bytes = hdfsWrite(fs, writeFile,
(void*)buffer, strlen(buffer)+1);
>>>>>>>>>     printf("\nWrote %d bytes\n", (int)num_written_bytes);
>>>>>>>>>     if (hdfsFlush(fs, writeFile)) {
>>>>>>>>>            printf("Failed to 'flush' %s\n", writePath);
>>>>>>>>>           exit(-1);
>>>>>>>>>     }
>>>>>>>>>    hdfsCloseFile(fs, writeFile);
>>>>>>>>>    hdfsDisconnect(fs);
>>>>>>>>>    return 0;}
>>>>>>>>>
>>>>>>>>> It compiles and runs without error, but I cannot see
the file on
>>>>>>>>> HDFS.
>>>>>>>>>
>>>>>>>>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>>>>>>>>
>>>>>>>>> Any ideas on this ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message