hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeon <xeonmailingl...@gmail.com>
Subject Re: UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE)
Date Sat, 12 Apr 2014 18:40:52 GMT
1 - Even with the dfs.permissions set to false in the core-site.xml, I 
still get the same error when I try to copy with webhdfs. You can see 
here the conf files in the destination host. I have similar 
configuration in the source host.

WorkGroup0010 # cat ../hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
   <property> <name>fs.default.name</name> 
<value>hdfs://172.16.100.5:9000</value> </property>
   <property> <name>hadoop.tmp.dir</name> 
<value>/tmp/hadoop-temp</value> </property>
<property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
<property><name>hadoop.proxyuser.root.groups</name><value>*</value></property>
   <property> <name>dfs.permissions</name> <value>false</value>
</property>
</configuration>

root@WorkGroup0010 # cat ../hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
         <property> <name>dfs.replication</name> <value>1</value>

</property>
         <property> <name>dfs.permissions</name> <value>false</value>

</property>
         <property> <name>dfs.name.dir</name> 
<value>/tmp/data/dfs/name/</value> </property>
         <property> <name>dfs.data.dir</name> 
<value>/tmp/data/dfs/data/</value> </property>
     <property> <name>dfs.webhdfs.enabled</name> <value>true</value>

</property>
</configuration>


2 - So, I tried to copy data with hdfs, and I also can't copy data, but 
with a different error. Any help to solve this either way?
WorkGroup0000 # /root/Programs/hadoop/n/hadoop distcp 
hdfs://WorkGroup0000:9000/wiki hdfs://WorkGroup0010:9000/wiki

And I get this error in the namenode, and datanode:

(Namonode)

2014-04-12 18:37:26,284 INFO 
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of 
transactions: 31 Total time for transactions(ms): 7 Number of 
transactions batched in Syncs: 0 Number of syncs: 17 SyncTimes(ms): 4
2014-04-12 18:37:26,354 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
allocateBlock: /wiki/.distcp.tmp.attempt_1397327306299_0003_m_000000_0. 
BP-862979082-172.16.100.5-1397327020274 
blk_-3251037992946253643_1007{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[172.16.100.8:50010|RBW]]}
2014-04-12 18:39:02,298 INFO 
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of 
transactions: 34 Total time for transactions(ms): 8 Number of 
transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 5

(Datanode)

2014-04-12 18:31:41,931 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: 
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, 
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2014-04-12 18:31:41,931 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: 
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005, 
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2014-04-12 18:31:41,931 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock 
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005 
received exception java.net.SocketTimeoutException: 60000 millis timeout 
while waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 
remote=/172.16.100.14:47189]
2014-04-12 18:31:41,931 ERROR 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
WorkGroup0011:50010:DataXceiver error processing WRITE_BLOCK operation  
src: /172.16.100.14:47189 dest: /172.16.100.6:50010
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010 
remote=/172.16.100.14:47189]
     at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
     at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
     at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.FilterInputStream.read(FilterInputStream.java:116)
     at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
     at java.io.DataInputStream.read(DataInputStream.java:132)
     at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)



On 04/12/2014 06:11 PM, Sergey Murylev wrote:
> Hi,
>
>> 1 - I don't understand why this is happening. I should have been able 
>> to copy data. Why I can't copy data between hdfs? 
> I think you should check permissions not only for root ("/"), you need 
> to make sure that whole path is accessible for root. You can do this 
> using following command:
>> /root/Programs/hadoop/bin/hadoop fs -cat 
>> webhdfs://WorkGroup0000:50070/wiki/wiki/wv92.txt 
> Make sure that folders /wiki and /wiki/wiki can be listed by root. 
> Actually I've confused by your use case, I don't understand why you 
> use webhdfs instead of hdfs. Native hdfs protocol is more faster than 
> webhdfs.
>> 2 - Is it possible to turn off the privilege rules? 
> Yes, it is possible, you should add flowing configuration to 
> core-site.xml and restart mapreduce & hdfs daemons
>>   <property>
>>     <name>dfs.permissions</name>
>>     <value>false</value>
>>   </property>
>
> --
> Thanks,
> Sergey
>
> On 12/04/14 20:52, xeon wrote:
>> Hi,
>>
>> I want to copy data between hdfs in different hosts using webhdfs in 
>> hadoop MRv2. The destination host has the user and group as root and 
>> supergroup.
>>
>> WorkGroup0010:~/# hdfs dfs -ls /
>> 14/04/12 16:42:18 WARN util.NativeCodeLoader: Unable to load 
>> native-hadoop library for your platform... using builtin-java classes 
>> where applicable
>> Found 3 items
>> drwxrwx---   - root supergroup          0 2014-04-12 16:25 /root
>> drwxr-xr-x   - root supergroup          0 2014-04-12 16:25 /tmp
>> drwxrwxrwx   - root supergroup          0 2014-04-12 16:39 /wiki
>>
>> I defined in the core-site.xml these params for the user and the group.
>>
>> WorkGroup0010:~/# cat ../hadoop/etc/hadoop/core-site.xml
>> <?xml version="1.0" encoding="UTF-8"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>> <configuration>
>>   <property> <name>fs.default.name</name> 
>> <value>hdfs://172.16.100.5:9000</value> </property>
>>   <property> <name>hadoop.tmp.dir</name> 
>> <value>/tmp/hadoop-temp</value> </property>
>> <property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>

>>
>> <property><name>hadoop.proxyuser.supergroup.groups</name><value>*</value></property>

>> <-- I am trying to use with thisgroup definition
>>   <!-- 
>> property><name>hadoop.proxyuser.root.groups</name><value>*</value></property

>> --> <-- or with this group
>> </configuration>
>>
>> When I copy data
>>
>> WorkGroup0000:~/# /root/Programs/hadoop/bin/hadoop distcp 
>> webhdfs://WorkGroup0000:50070/wiki webhdfs://WorkGroup0010:50070/wiki
>>
>> I get the error on the destination as:
>> 2014-04-12 16:37:06,720 ERROR 
>> org.apache.hadoop.security.UserGroupInformation: 
>> PriviledgedActionException as:root (auth:SIMPLE) 
>> cause:java.io.FileNotFoundException: File does not exist: 
>> /wiki/wiki/wv92.txt
>>
>> 1 - I don't understand why this is happening. I should have been able 
>> to copy data. Why I can't copy data between hdfs?
>> 2 - Is it possible to turn off the privilege rules?
>>
>

-- 
Thanks,


Mime
View raw message