Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D696C107F8 for ; Sat, 12 Apr 2014 18:41:39 +0000 (UTC) Received: (qmail 14496 invoked by uid 500); 12 Apr 2014 18:41:25 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 14036 invoked by uid 500); 12 Apr 2014 18:41:24 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 14009 invoked by uid 99); 12 Apr 2014 18:41:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Apr 2014 18:41:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of xeonmailinglist@gmail.com designates 209.85.212.170 as permitted sender) Received: from [209.85.212.170] (HELO mail-wi0-f170.google.com) (209.85.212.170) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Apr 2014 18:41:16 +0000 Received: by mail-wi0-f170.google.com with SMTP id bs8so3852875wib.5 for ; Sat, 12 Apr 2014 11:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; bh=2mPtpF72MewS0c2NA5+CqaDQrrdvGFJ8oIQX4uW+RE4=; b=QB2/j2wAJFL9tjglLouSuxsEIPjLvGGZw3FC3HBL0U2DABIBylJ5a8SbDq4UiA/jVZ vT3n1wi7GseiPxaeLWSq9EF74SGIqqrPQe43h0hXV69ulNrKwwz5LMosffvSSl6iAkah zGhVuW+Qn0hqO7PoK4xwAPe3+bSWvRQeUpdBLfugSQFfEL+A3chFVYPbOpsnbwgupUqV e9tGyhfviMvvCyEJFC0nCeIwsFjq/o+6W3OVCt2r6aqV2p9ResLrOuxlxjub1bU5IDaf h8kd6vvJVPdhaCgMXW+2sOQvdEqWDHUidLbIEEPg/PSd2axwSJ8BlRmqKC37QHDHqSXa 3lFg== X-Received: by 10.180.106.168 with SMTP id gv8mr3238484wib.19.1397328054427; Sat, 12 Apr 2014 11:40:54 -0700 (PDT) Received: from [10.101.227.100] ([194.117.18.101]) by mx.google.com with ESMTPSA id kp5sm16820294wjb.30.2014.04.12.11.40.52 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 12 Apr 2014 11:40:53 -0700 (PDT) Message-ID: <534988B4.3040207@gmail.com> Date: Sat, 12 Apr 2014 19:40:52 +0100 From: xeon User-Agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Sergey Murylev CC: user@hadoop.apache.org, common-user@hadoop.apache.org, hdfs-dev@hadoop.apache.org Subject: Re: UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) References: <53496F63.5080109@gmail.com> <534973BB.8080708@gmail.com> In-Reply-To: <534973BB.8080708@gmail.com> Content-Type: multipart/alternative; boundary="------------040608090701090008060103" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------040608090701090008060103 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit 1 - Even with the dfs.permissions set to false in the core-site.xml, I still get the same error when I try to copy with webhdfs. You can see here the conf files in the destination host. I have similar configuration in the source host. WorkGroup0010 # cat ../hadoop/etc/hadoop/core-site.xml fs.default.name hdfs://172.16.100.5:9000 hadoop.tmp.dir /tmp/hadoop-temp hadoop.proxyuser.root.hosts* hadoop.proxyuser.root.groups* dfs.permissions false root@WorkGroup0010 # cat ../hadoop/etc/hadoop/hdfs-site.xml dfs.replication 1 dfs.permissions false dfs.name.dir /tmp/data/dfs/name/ dfs.data.dir /tmp/data/dfs/data/ dfs.webhdfs.enabled true 2 - So, I tried to copy data with hdfs, and I also can't copy data, but with a different error. Any help to solve this either way? WorkGroup0000 # /root/Programs/hadoop/n/hadoop distcp hdfs://WorkGroup0000:9000/wiki hdfs://WorkGroup0010:9000/wiki And I get this error in the namenode, and datanode: (Namonode) 2014-04-12 18:37:26,284 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 31 Total time for transactions(ms): 7 Number of transactions batched in Syncs: 0 Number of syncs: 17 SyncTimes(ms): 4 2014-04-12 18:37:26,354 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /wiki/.distcp.tmp.attempt_1397327306299_0003_m_000000_0. BP-862979082-172.16.100.5-1397327020274 blk_-3251037992946253643_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.16.100.8:50010|RBW]]} 2014-04-12 18:39:02,298 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 34 Total time for transactions(ms): 8 Number of transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 5 (Datanode) 2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted. 2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 remote=/172.16.100.14:47189] 2014-04-12 18:31:41,931 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: WorkGroup0011:50010:DataXceiver error processing WRITE_BLOCK operation src: /172.16.100.14:47189 dest: /172.16.100.6:50010 java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 remote=/172.16.100.14:47189] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.FilterInputStream.read(FilterInputStream.java:116) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) On 04/12/2014 06:11 PM, Sergey Murylev wrote: > Hi, > >> 1 - I don't understand why this is happening. I should have been able >> to copy data. Why I can't copy data between hdfs? > I think you should check permissions not only for root ("/"), you need > to make sure that whole path is accessible for root. You can do this > using following command: >> /root/Programs/hadoop/bin/hadoop fs -cat >> webhdfs://WorkGroup0000:50070/wiki/wiki/wv92.txt > Make sure that folders /wiki and /wiki/wiki can be listed by root. > Actually I've confused by your use case, I don't understand why you > use webhdfs instead of hdfs. Native hdfs protocol is more faster than > webhdfs. >> 2 - Is it possible to turn off the privilege rules? > Yes, it is possible, you should add flowing configuration to > core-site.xml and restart mapreduce & hdfs daemons >> >> dfs.permissions >> false >> > > -- > Thanks, > Sergey > > On 12/04/14 20:52, xeon wrote: >> Hi, >> >> I want to copy data between hdfs in different hosts using webhdfs in >> hadoop MRv2. The destination host has the user and group as root and >> supergroup. >> >> WorkGroup0010:~/# hdfs dfs -ls / >> 14/04/12 16:42:18 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes >> where applicable >> Found 3 items >> drwxrwx--- - root supergroup 0 2014-04-12 16:25 /root >> drwxr-xr-x - root supergroup 0 2014-04-12 16:25 /tmp >> drwxrwxrwx - root supergroup 0 2014-04-12 16:39 /wiki >> >> I defined in the core-site.xml these params for the user and the group. >> >> WorkGroup0010:~/# cat ../hadoop/etc/hadoop/core-site.xml >> >> >> >> >> >> fs.default.name >> hdfs://172.16.100.5:9000 >> hadoop.tmp.dir >> /tmp/hadoop-temp >> hadoop.proxyuser.root.hosts* >> >> hadoop.proxyuser.supergroup.groups* >> <-- I am trying to use with thisgroup definition >> <-- or with this group >> >> >> When I copy data >> >> WorkGroup0000:~/# /root/Programs/hadoop/bin/hadoop distcp >> webhdfs://WorkGroup0000:50070/wiki webhdfs://WorkGroup0010:50070/wiki >> >> I get the error on the destination as: >> 2014-04-12 16:37:06,720 ERROR >> org.apache.hadoop.security.UserGroupInformation: >> PriviledgedActionException as:root (auth:SIMPLE) >> cause:java.io.FileNotFoundException: File does not exist: >> /wiki/wiki/wv92.txt >> >> 1 - I don't understand why this is happening. I should have been able >> to copy data. Why I can't copy data between hdfs? >> 2 - Is it possible to turn off the privilege rules? >> > -- Thanks, --------------040608090701090008060103 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit 1 - Even with the dfs.permissions set to false in the core-site.xml, I still get the same error when I try to copy with webhdfs. You can see here the conf files in the destination host. I have similar configuration in the source host.

WorkGroup0010 # cat ../hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property> <name>fs.default.name</name> <value>hdfs://172.16.100.5:9000</value> </property>
  <property> <name>hadoop.tmp.dir</name> <value>/tmp/hadoop-temp</value> </property>
  <property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
  <property><name>hadoop.proxyuser.root.groups</name><value>*</value></property>
  <property> <name>dfs.permissions</name> <value>false</value> </property>
</configuration>

root@WorkGroup0010 # cat ../hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
        <property> <name>dfs.replication</name> <value>1</value> </property>
        <property> <name>dfs.permissions</name> <value>false</value> </property>
        <property> <name>dfs.name.dir</name> <value>/tmp/data/dfs/name/</value> </property>
        <property> <name>dfs.data.dir</name> <value>/tmp/data/dfs/data/</value> </property>
    <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property>
</configuration>


2 - So, I tried to copy data with hdfs, and I also can't copy data, but with a different error. Any help to solve this either way?
WorkGroup0000 # /root/Programs/hadoop/n/hadoop distcp hdfs://WorkGroup0000:9000/wiki hdfs://WorkGroup0010:9000/wiki

And I get this error in the namenode, and datanode:

(Namonode)

2014-04-12 18:37:26,284 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 31 Total time for transactions(ms): 7 Number of transactions batched in Syncs: 0 Number of syncs: 17 SyncTimes(ms): 4
2014-04-12 18:37:26,354 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /wiki/.distcp.tmp.attempt_1397327306299_0003_m_000000_0. BP-862979082-172.16.100.5-1397327020274 blk_-3251037992946253643_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[172.16.100.8:50010|RBW]]}
2014-04-12 18:39:02,298 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 34 Total time for transactions(ms): 8 Number of transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 5

(Datanode)

2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2014-04-12 18:31:41,931 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005 received exception java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 remote=/172.16.100.14:47189]
2014-04-12 18:31:41,931 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: WorkGroup0011:50010:DataXceiver error processing WRITE_BLOCK operation  src: /172.16.100.14:47189 dest: /172.16.100.6:50010
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 remote=/172.16.100.14:47189]
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.FilterInputStream.read(FilterInputStream.java:116)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)



On 04/12/2014 06:11 PM, Sergey Murylev wrote:
Hi,

1 - I don't understand why this is happening. I should have been able to copy data. Why I can't copy data between hdfs?
I think you should check permissions not only for root ("/"), you need to make sure that whole path is accessible for root. You can do this using following command:
/root/Programs/hadoop/bin/hadoop fs -cat webhdfs://WorkGroup0000:50070/wiki/wiki/wv92.txt
Make sure that folders /wiki and /wiki/wiki can be listed by root. Actually I've confused by your use case, I don't understand why you use webhdfs instead of hdfs. Native hdfs protocol is more faster than webhdfs.
2 - Is it possible to turn off the privilege rules?
Yes, it is possible, you should add flowing configuration to core-site.xml and restart mapreduce & hdfs daemons
  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>

--
Thanks,
Sergey

On 12/04/14 20:52, xeon wrote:
Hi,

I want to copy data between hdfs in different hosts using webhdfs in hadoop MRv2. The destination host has the user and group as root and supergroup.

WorkGroup0010:~/# hdfs dfs -ls /
14/04/12 16:42:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
drwxrwx---   - root supergroup          0 2014-04-12 16:25 /root
drwxr-xr-x   - root supergroup          0 2014-04-12 16:25 /tmp
drwxrwxrwx   - root supergroup          0 2014-04-12 16:39 /wiki

I defined in the core-site.xml these params for the user and the group.

WorkGroup0010:~/# cat ../hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property> <name>fs.default.name</name> <value>hdfs://172.16.100.5:9000</value> </property>
  <property> <name>hadoop.tmp.dir</name> <value>/tmp/hadoop-temp</value> </property>
<property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
<property><name>hadoop.proxyuser.supergroup.groups</name><value>*</value></property> <-- I am trying to use with thisgroup definition
  <!-- property><name>hadoop.proxyuser.root.groups</name><value>*</value></property --> <-- or with this group
</configuration>

When I copy data

WorkGroup0000:~/# /root/Programs/hadoop/bin/hadoop distcp webhdfs://WorkGroup0000:50070/wiki webhdfs://WorkGroup0010:50070/wiki

I get the error on the destination as:
2014-04-12 16:37:06,720 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /wiki/wiki/wv92.txt

1 - I don't understand why this is happening. I should have been able to copy data. Why I can't copy data between hdfs?
2 - Is it possible to turn off the privilege rules? 



-- 
Thanks,
--------------040608090701090008060103--