hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guido Serra <z...@fsfe.org>
Subject Re: listing a 530k files directory
Date Fri, 30 May 2014 13:08:28 GMT
guido@hd11 ~ $ export HADOOP_OPTS=-Xmx3g;hdfs dfs -ls /logs/2014-05-28/
14/05/30 13:05:44 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB. Trying 
to fail over immediately.
14/05/30 13:05:45 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 1 
fail over attempts. Trying to fail over after sleeping for 935ms.
14/05/30 13:05:48 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 2 
fail over attempts. Trying to fail over immediately.
14/05/30 13:05:48 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 3 
fail over attempts. Trying to fail over after sleeping for 5408ms.
14/05/30 13:05:55 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 4 
fail over attempts. Trying to fail over immediately.
14/05/30 13:05:55 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 5 
fail over attempts. Trying to fail over after sleeping for 14316ms.
14/05/30 13:06:12 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 6 
fail over attempts. Trying to fail over immediately.
14/05/30 13:06:12 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 7 
fail over attempts. Trying to fail over after sleeping for 8216ms.
14/05/30 13:06:22 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 8 
fail over attempts. Trying to fail over immediately.
14/05/30 13:06:23 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 9 
fail over attempts. Trying to fail over after sleeping for 18917ms.
14/05/30 13:06:44 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 10 
fail over attempts. Trying to fail over immediately.
14/05/30 13:06:44 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 11 
fail over attempts. Trying to fail over after sleeping for 16386ms.
14/05/30 13:07:03 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 12 
fail over attempts. Trying to fail over immediately.
14/05/30 13:07:03 WARN retry.RetryInvocationHandler: Exception while 
invoking getListing of class ClientNamenodeProtocolTranslatorPB after 13 
fail over attempts. Trying to fail over after sleeping for 20387ms.
^[[B^[[B^[[B^[[B14/05/30 13:07:26 WARN retry.RetryInvocationHandler: 
Exception while invoking getListing of class 
ClientNamenodeProtocolTranslatorPB after 14 fail over attempts. Trying 
to fail over immediately.
14/05/30 13:07:26 WARN retry.RetryInvocationHandler: Exception while 
invoking class 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing. 
Not retrying because failovers (15) exceeded maximum allowed (15)
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): 
Operation category READ is not supported in state standby
         at 
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
         at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1416)
         at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:969)
         at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:3542)
         at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:3530)
         at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:682)
         at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:433)
         at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44972)
         at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)
         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:415)
         at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

         at org.apache.hadoop.ipc.Client.call(Client.java:1238)
         at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
         at com.sun.proxy.$Proxy9.getListing(Unknown Source)
         at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:441)
         at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:606)
         at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
         at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
         at com.sun.proxy.$Proxy10.getListing(Unknown Source)
         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1526)
         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1509)
         at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:437)
         at 
org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:213)
         at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:337)
         at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:89)
         at 
org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
         at 
org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
         at 
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
         at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
         at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
         at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)
ls: Operation category READ is not supported in state standby

On 05/30/2014 03:03 PM, Suresh Srinivas wrote:
> Listing such a directory should not be a big problem. Can you cut and 
> paste the command output.
>
> Which release are you using?
>
> Sent from phone
>
> On May 30, 2014, at 5:49 AM, Guido Serra <zeph@fsfe.org 
> <mailto:zeph@fsfe.org>> wrote:
>
>> already tried, didn't work (24cores at 100% and a-lot-memory, stilll 
>> ... "GC overhead limit exceed")
>>
>> thanks anyhow
>>
>> On 05/30/2014 02:43 PM, bharath vissapragada wrote:
>>> Hi Guido,
>>>
>>> You can set client side heap in HADOOP_OPTS variable before running 
>>> the ls command.
>>>
>>> export HADOOP_OPTS="-Xmx3g"; hadoop fs -ls /
>>>
>>> - Bharath
>>>
>>>
>>> On Fri, May 30, 2014 at 5:22 PM, Guido Serra <zeph@fsfe.org 
>>> <mailto:zeph@fsfe.org>> wrote:
>>>
>>>     Hi,
>>>     do you have an idea on how to look at the content of a
>>>     530k-files HDFS folder?
>>>     (yes, I know it is a bad idea to have such setup, but that’s the
>>>     status and I’d like to debug it)
>>>     and the only tool that doesn’t go out of memory is "hdfs dfs
>>>     -count folder/“
>>>
>>>     -ls goes out of memory, -count with the folder/* goes out of
>>>     memory …
>>>     I’d like at least at the first 10 file names, see the size,
>>>     maybe open one
>>>
>>>     thanks,
>>>     G.
>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. 
> Thank You. 


Mime
View raw message