hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yoann Moulin <yoann.mou...@epfl.ch>
Subject hadoop with swift storage
Date Wed, 29 Jun 2016 08:19:14 GMT
Hello,

I'd like to use swift storage provided by a ceph cluster (not under Openstack
and no keystone installed) with hadoop (Ambari).

Version :

HDP   2.4.0.0-169
HDFS  2.7.1.2.
YARN  2.7.1.2.4
Ceph  10.2.2 (jewel)

I'm able to connect to my swift bucket with rclone but I can't connect with
hadoop, Id' like to know where I am wrong in my configuration.

The command I run :

> $ hadoop distcp \
> -D fs.swift.service.ceph.username=ymoulin:swift \
> -D fs.swift.service.ceph.password=<snip> \
> -D fs.swift.service.ceph.auth.url=http://node018.ceph.local:8080/auth/v1.0/ \
> -D fs.swift.service.ceph.http.port=80 \
> dcu.txt swift://mybigdatacontainer.ceph/dcu.txt
> 16/06/29 09:58:35 ERROR tools.DistCp: Invalid arguments: 
> Invalid Response: Method POST on http://node018.ceph.local:8080/auth/v1.0/ failed, status
code: 405, status line: HTTP/1.1 405 Method Not Allowed  POST http://node018.ceph.local:8080/auth/v1.0/
=> 405 : {"Code":"MethodNotAllowed","RequestId":"tx0000000000000000e4505-0057737faa-19702-default","HostId":"19702-default-default"}
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1501)
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402)
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1079)
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.authIfNeeded(SwiftRestClient.java:1298)
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.preRemoteCommand(SwiftRestClient.java:1314)
> 	at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1015)
> 	at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:258)
> 	at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:213)
> 	at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:182)
> 	at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> 	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> 	at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
> 	at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> 	at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
> Invalid arguments: Method POST on http://node018.ceph.local:8080/auth/v1.0/ failed, status
code: 405, status line: HTTP/1.1 405 Method Not Allowed

HTTP Headers from hadoop are :

> POST /auth/v1.0/ HTTP/1.1
> User-Agent: Apache Hadoop Swift Client 2.7.1.2.4.0.0-169 from 26104d8ac833884c8776473823007f176854f2eb
by jenkins source checksum cf48a4c63aaec76a714c1897e2ba8be6
> Host: node018.ceph.local:8080
> Content-Length: 133
> Content-Type: application/json; charset=UTF-8
> 
> {"auth":{"tenantName":null,"passwordCredentials":{"username":"ymoulin:swift","password":"<snip>"}}}HTTP/1.1
405 Method Not Allowed
> x-amz-request-id: tx0000000000000000e4500-005772967d-19702-default
> Content-Length: 123
> Accept-Ranges: bytes
> Content-Type: application/json
> Date: Tue, 28 Jun 2016 15:23:41 GMT
> 
> {"Code":"MethodNotAllowed","RequestId":"tx0000000000000000e4500-005772967d-19702-default","HostId":"19702-default-default"}

HTTP Headers from rclone are :

> GET /auth/v1.0/ HTTP/1.1
> Host: node018.ceph.local:8080
> User-Agent: rclone/v1.29
> X-Auth-Key: <snip>
> X-Auth-User: ymoulin:swift
> Accept-Encoding: gzip
> 
> HTTP/1.1 204 No Content
> X-Storage-Url: http://node018.ceph.local:8080/swift/v1
> X-Storage-Token: AUTH_rgwtk0d000000796d6f756c696e3a7377696674113cf055a759d9ec04e87357f1a0202e53c2b24a93c90be7eb58def247434d259c1c512b
> X-Auth-Token: AUTH_rgwtk0d000000796d6f756c696e3a7377696674113cf055a759d9ec04e87357f1a0202e53c2b24a93c90be7eb58def247434d259c1c512b
> x-amz-request-id: tx0000000000000000e4501-0057729684-19702-default
> Date: Tue, 28 Jun 2016 15:23:48 GMT

Is there other parameter I should add to the hadoop command line ?

Thanks for you help

-- 
Yoann Moulin
EPFL IC-IT

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message