hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Robison <david.robi...@psgglobal.net>
Subject Problem appending to file with WebHDFS
Date Fri, 13 May 2016 16:20:31 GMT
I am trying to use WebHDFS to write two files to Hadoop. One is an image file and the other
is an index file. I am able to create both files, append twice to both files, but when I try
to append a 3rd time I get the following error:

"{"RemoteException":{"exception":"AlreadyBeingCreatedException","javaClassName":"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException","message":"Failed
to APPEND_FILE /PSG/VideoVault/VideoStorage/VICADS-100/20160513.img for DFSClient_NONMAPREDUCE_1168342942_28
on 192.168.1.211 because this file lease is currently owned by DFSClient_NONMAPREDUCE_-2117428583_27
on 192.168.1.211\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2684)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2991)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2960)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:719)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:421)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)\n\tat java.security.AccessController.doPrivileged(Native
Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)\n"}}"
WARN  [net.psgglobal.videovault.recorder.JpegWriter] (WriterThread) Error Writting JPeg: Error
appending to file at Data Node http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden

Any idea what I may be doing wrong? Is there something I need to know about the file leases
when using WebHDFS?

Here is the HTTP communication between the client and the WebHDFS server with some annotations
added.

--------------- CREATE IMG FILE ----------------

>> "PUT /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?user.name=hdfs&op=CREATE&overwrite=false
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location: http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"

>> "PUT /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 201 Created[\r][\n]"
<< "Location: hdfs://vb1.localdomain:8020/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IMG FILE ----------------

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?user.name=hdfs&op=APPEND
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location: http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
HTTP/1.1[\r][\n]"
>> "Content-Length: 52685[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IDX FILE ----------------

>> "PUT /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?user.name=hdfs&op=CREATE&overwrite=false
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location: http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"

>> "PUT /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 201 Created[\r][\n]"
<< "Location: hdfs://vb1.localdomain:8020/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IDX FILE ----------------

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?user.name=hdfs&op=APPEND
HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie: hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"

<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location: http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
HTTP/1.1[\r][\n]"
>> "Content-Length: 24[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IMG FILE ----------------

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
HTTP/1.1[\r][\n]"
>> "Content-Length: 52685[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IDX FILE ----------------

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
HTTP/1.1[\r][\n]"
>> "Content-Length: 24[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"

--------------- APPEND IMG FILE ----------------

>> "POST /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
HTTP/1.1[\r][\n]"
>> "Content-Length: 54623[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>

<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 403 Forbidden[\r][\n]"
<< "Content-Type: application/json; charset=utf-8[\r][\n]"
<< "Content-Length: 1756[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
<< "{"RemoteException":{"exception":"AlreadyBeingCreatedException","javaClassName":"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException","message":"Failed
to APPEND_FILE /PSG/VideoVault/VideoStorage/VICADS-100/20160513.img for DFSClient_NONMAPREDUCE_1168342942_28
on 192.168.1.211 because this file lease is currently owned by DFSClient_NONMAPREDUCE_-2117428583_27
on 192.168.1.211\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2684)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2991)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2960)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:719)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:421)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)\n\tat java.security.AccessController.doPrivileged(Native
Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)\n"}}"
WARN  [net.psgglobal.videovault.recorder.JpegWriter] (WriterThread) Error Writting JPeg: Error
appending to file at Data Node http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden
ERROR [stderr] (WriterThread) java.io.IOException: Error appending to file at Data Node http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden
ERROR [stderr] (WriterThread)      at net.psgglobal.hdfs.videovault.videostorage.WebHdfsFile.appendToFile(WebHdfsFile.java:252)
ERROR [stderr] (WriterThread)      at net.psgglobal.hdfs.videovault.videostorage.JpegImageFile.appendJpeg(JpegImageFile.java:43)
ERROR [stderr] (WriterThread)      at net.psgglobal.hdfs.videovault.videostorage.VideoVaultRecordFile.writeToFile(VideoVaultRecordFile.java:40)
ERROR [stderr] (WriterThread)      at net.psgglobal.videovault.recorder.JpegWriter$WriterThread.run(JpegWriter.java:95)

Thanks, David

David R Robison
Senior Systems Engineer
O. +1 512 247 3700
M. +1 512 608 3173
david.robison@psgglobal.net<mailto:david.robison@psgglobal.net>
www.psgglobal.net<http://www.psgglobal.net/>

Prometheus Security Group Global, Inc.
3019 Alvin Devane Boulevard
Building 4, Suite 450
Austin, TX 78741



Mime
View raw message