Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
From: "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com>
To: "dev@apex.incubator.apache.org" <dev@apex.incubator.apache.org>
Subject: RE: Long-running HDFS Write errors
Thread-Topic: Long-running HDFS Write errors
Thread-Index: AQHRe2MlMch59W52AkK5Sa3PZkXjYp9T0RmAgACJH+k=
Date: Fri, 11 Mar 2016 15:12:59 +0000
Message-ID: 
 <BN3P103MB00034A6C4EA8C9926778C7CDE6B50@BN3P103MB0003.NAMP103.PROD.OUTLOOK.COM>
References: 
 <ACB863D8-4175-4E37-9753-C9712B4ED235@capitalone.com>,<CAKJfLDOTwsHAG7S1GqmgcSkWG-_Gz9FZiBFXkH2xuR7-LmSC1Q@mail.gmail.com>
In-Reply-To: 
 <CAKJfLDOTwsHAG7S1GqmgcSkWG-_Gz9FZiBFXkH2xuR7-LmSC1Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BN3P103MB00034A6C4EA8C9926778C7CDE6B50BN3P103MB0003NAMP_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Mar 2016 15:12:59.4817
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: c4bbfe29-1e89-4a06-9144-923b36afc949
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3P103MB0004
X-OriginatorOrg: capitalone.com

--_000_BN3P103MB00034A6C4EA8C9926778C7CDE6B50BN3P103MB0003NAMP_
MIME-Version: 1.0
Content-Type: text/plain; charset="windows-1252"
Content-Transfer-Encoding: quoted-printable

This is 3.0.0.


Sent with Good (www.good.com)
________________________________
From: Thomas Weise <thomas@datatorrent.com>
Sent: Friday, March 11, 2016 2:02:13 AM
To: dev@apex.incubator.apache.org
Subject: Re: Long-running HDFS Write errors

Which version of Malhar is this?


On Thu, Mar 10, 2016 at 10:56 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> wrote:

> Hello =96 I have a long-running job which simultaneously writes to multip=
le
> files on HDFS. I am seeing the following error come up:
>
> I would appreciate any insight into what=92s going on here.
>
>
> Stopped running due to an exception.
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.Alr=
eadyBeingCreatedException):
> Failed to create file
> [/user/vault8/citadel_out/2016_03_20_17_34_339/records/records_I@633.txt.=
1457678005848.tmp]
> for [DFSClient_NONMAPREDUCE_232430238_1207] for client [10.24.28.64],
> because this file is already being created by
> [DFSClient_NONMAPREDUCE_-1482819983_1172] on [10.24.28.64]
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(=
FSNamesystem.java:3122)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FS=
Namesystem.java:2905)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNames=
ystem.java:3186)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesyst=
em.java:3149)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeR=
pcServer.java:611)
>         at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientPr=
otocol.append(AuthorizationProviderProxyClientProtocol.java:124)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTransla=
torPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:416)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Client=
NamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Pr=
otobufRpcEngine.java:617)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.java:1671)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
>
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
>         at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
>         at
> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
>         at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4=
829)
>         at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.processTuple(Abstrac=
tFileOutputOperator.java:667)
>         at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator$1.process(AbstractFi=
leOutputOperator.java:236)
>         at
> com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:67)
>         at
> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep=
(BufferServerSubscriber.java:244)
>         at
> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:226)
>         at
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.=
java:1365)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.Alr=
eadyBeingCreatedException):
> Failed to create file
> [/user/vault8/citadel_out/2016_03_20_17_34_339/records/records_I@633.txt.=
1457678005848.tmp]
> for [DFSClient_NONMAPREDUCE_232430238_1207] for client [10.24.28.64],
> because this file is already being created by
> [DFSClient_NONMAPREDUCE_-1482819983_1172] on [10.24.28.64]
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(=
FSNamesystem.java:3122)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FS=
Namesystem.java:2905)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNames=
ystem.java:3186)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesyst=
em.java:3149)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeR=
pcServer.java:611)
>         at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientPr=
otocol.append(AuthorizationProviderProxyClientProtocol.java:124)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTransla=
torPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:416)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Client=
NamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Pr=
otobufRpcEngine.java:617)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.java:1671)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
>
>         at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator$3.load(AbstractFileO=
utputOperator.java:414)
>         at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator$3.load(AbstractFileO=
utputOperator.java:334)
>         at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(Local=
Cache.java:3568)
>         at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
>         at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.jav=
a:2313)
>         at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
>         ... 9 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.Alr=
eadyBeingCreatedException):
> Failed to create file
> [/user/vault8/citadel_out/2016_03_20_17_34_339/records/records_I@633.txt.=
1457678005848.tmp]
> for [DFSClient_NONMAPREDUCE_232430238_1207] for client [10.24.28.64],
> because this file is already being created by
> [DFSClient_NONMAPREDUCE_-1482819983_1172] on [10.24.28.64]
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(=
FSNamesystem.java:3122)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FS=
Namesystem.java:2905)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNames=
ystem.java:3186)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesyst=
em.java:3149)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeR=
pcServer.java:611)
>         at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientPr=
otocol.append(AuthorizationProviderProxyClientProtocol.java:124)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTransla=
torPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:416)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Client=
NamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Pr=
otobufRpcEngine.java:617)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.java:1671)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1472)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1403)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.=
java:230)
>         at com.sun.proxy.$Proxy14.append(Unknown Source)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.appe=
nd(ClientNamenodeProtocolTranslatorPB.java:313)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc=
ationHandler.java:252)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH=
andler.java:104)
>         at com.sun.proxy.$Proxy15.append(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:184=
2)
>         at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1878)
>         at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1871)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSyst=
em.java:329)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSyst=
em.java:325)
>         at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolve=
r.java:81)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem=
.java:325)
>         at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1172)
>         at
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator$3.load(AbstractFileO=
utputOperator.java:371)
>         ... 14 more
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intend=
ed
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary=
 to Capital One and/or its affiliates and may only be used solely in perfor=
mance of work or services for Capital One. The information transmitted here=
with is intended only for use by the individual or entity to which it is ad=
dressed. If the reader of this message is not the intended recipient, you a=
re hereby notified that any review, retransmission, dissemination, distribu=
tion, copying or other use of, or taking of any action in reliance upon thi=
s information is strictly prohibited. If you have received this communicati=
on in error, please contact the sender and delete the material from your co=
mputer.

--_000_BN3P103MB00034A6C4EA8C9926778C7CDE6B50BN3P103MB0003NAMP_--