hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Bamford <nathan.bamf...@redpointglobal.com>
Subject Re: unable to write partitions with HCatWriter on Cloudera with Sentry and HDFS ACL plugin
Date Tue, 25 Sep 2018 15:07:02 GMT
Alex,

  Thank you for the response.

  I do see that all of the directories in question are owned by hive:hive, which makes sense,
and indeed, writing to them from HCatWriter seems to be no problem.

  The problem arises in the org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.constructPartition
function when applyGroupAndPerms is called. This generates the "user X is not the owner of
inode 'partition'" error.

  That's why I'm thinking HCatWriter simply cannot write partitions under Sentry.

  I guess a potential fix would be to have the constructPartition call check for appropriate
permissions before blindly trying to set them. In my case, at least, the directory can be
created no problem and has the hive:hive permissions the same as any other table directory.


Nathan

________________________________
From: Alexander Kolbasov <akolb@cloudera.com>
Sent: Monday, September 24, 2018 1:30:58 PM
To: user@hive.apache.org
Subject: Re: unable to write partitions with HCatWriter on Cloudera with Sentry and HDFS ACL
plugin

Nathan,

when Sentry is enabled, all files are supposed to be owned by hive:hive or impala:impala and
permissions are managed via HDFS ACLs which are coordinated with Sentry.

- Alex

On Mon, Sep 24, 2018 at 1:15 PM Nathan Bamford <nathan.bamford@redpointglobal.com<mailto:nathan.bamford@redpointglobal.com>>
wrote:

Hi,

  We use HCatWriter to write records to Hive, and I've recently run into a problem with HCatWriter
that seems intractable.

  We can write tables without partitions all the live-long day, but any attempt to write to
a partition results in the following error:


"net/redpoint/hiveclient/DMHCatWriter.closeWriter:org.apache.hive.hcatalog.common.HCatException
: 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException
: 2006 : Error adding partition to metastore. Cause : org.apache.hadoop.security.AccessControlException:
Permission denied. user=nbamford is not the owner of inode=state=CO"

  Digging into the source for org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.constructPartition,
I find the following lines:
for (FieldSchema partKey : table.getPartitionKeys()) {
if (i++ != 0) {
fs.mkdirs(partPath); // Attempt to make the path in case it does not exist before we check
applyGroupAndPerms(fs, partPath, perms, grpName, false);
}
partPath = constructPartialPartPath(partPath, partKey.getName().toLowerCase(), partKVs);
}
}

  The error is thrown from the applyGroupAndPerms function, which you will note does not check
for the directory existing, and having the right permissions (in this case, it does).


  I am at a complete loss for how to proceed. I can't even think of a workaround. It seems
to me HCatWriter simply cannot write partitions when Sentry and the HDFS ACL plugin are in
force.

Mime
View raw message