Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of yaboulna@uwaterloo.ca
 designates 129.97.50.127 as permitted sender)
Message-ID: <20121126110717.43033xq2imt8g3uo@www.nexusmail.uwaterloo.ca>
Date: Mon, 26 Nov 2012 11:07:17 -0500
From: yaboulna@uwaterloo.ca
Reply-to: younos.aboulnaga@uwaterloo.ca
To: user@hive.apache.org
Subject: Permissions preventing me from inserting data into table I have
 just created
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) H3 (4.3.5) / FreeBSD-7.2

Hello,

I'm using Cloudera's CDH4 with Hive 0.9 and Hive Server 2. I am trying  
to load data into hive using the JDBC driver (the one distributed with  
Cloudera CDH4 "org.apache.hive.jdbc.HiveDriver". I can create the  
staging table and LOAD LOCAL into it. However when I try to insert  
data into a table with Columnar SerDe Stored As RCFILE I get an error  
caused by file permissions. I don't think that the SerDE or the Stored  
as parameters have anything to do with the problem but I mentioned  
them for completeness. The problem is that hive creates a temporary  
file in its scratch folder (local) owned by hive:hive with permissions  
755, then pass it as an input to a mapper running as the user  
mapred:mapred. Now the mapper tries to create something inside the  
input folder (probably can do this elsewhere), and the following  
exception is thrown:

org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:  
Mkdirs failed to create  
file:/home/yaboulnaga/tmp/hive-scratch/hive_2012-11-26_10-46-44_887_2004468370569495405/_task_tmp.-ext-10002
	at  
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237)
	at  
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477)
	at  
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:709)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)


As you might have noticed, I moved the scrach folder to a directory  
under my home dir so that I can give this directory 777 permissions.  
The idea was to use hive.files.umask.value of 0000 to cause  
subdirectories to inherit the same open permission (not the best  
workaround, but wouldn't hurt on my local machine). Unfortunately this  
didn't work even when I added a umask 0000 to /etc/init.d/hiveserver2.  
Can someone please tell me what's the right way to do this? I mean  
create a table and then insert values into it! The Hive QL statements  
I use are very similar to the ones in the tutorials about loading data.

Cheers!
-- Younos