hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Murumkar <manoj.murum...@gmail.com>
Subject Re: Hive transactions not working
Date Thu, 17 Nov 2016 02:16:34 GMT
By default, I have these:

hadoop.proxyuser.hive.hosts=*
hadoop.proxyuser.hive.groups=*

Also, I have impersonation enabled. These settings should have made the
compaction to run without issues, but does not. What am I missing?

On Wed, Nov 16, 2016 at 6:00 PM, Manoj Murumkar <manoj.murumkar@gmail.com>
wrote:

> Okay, made the cleaner work as well :)
>
> When the files are owned by a user that is different from the user that
> runs metastore ("hive"), this doesn't work. Just to test this, I changed
> ownership of the files for the table to "hive" and ran a major compaction
> and it does the job.
>
> drwxrwxrwt   - hive hive          0 2016-11-17 01:35
> /user/hive/warehouse/txntest.db/txntab3/base_0000024
> -rw-r--r--   3 hive hive     451008 2016-11-17 01:35
> /user/hive/warehouse/txntest.db/txntab3/base_0000024/bucket_00000
>
> I am going to follow this piece of documentation, so it works:
> Configuration Values to Set for Compaction
>
> If the data in your system is not owned by the Hive user (i.e., the user
> that the Hive metastore runs as), then Hive will need permission to run as
> the user who owns the data in order to perform compactions.  If you have
> already set up HiveServer2 to impersonate users, then the only additional
> work to do is assure that Hive has the right to impersonate users from the
> host running the Hive metastore.  This is done by adding the hostname to
> hadoop.proxyuser.hive.hosts in Hadoop's core-site.xml file.  If you have
> not already done this, then you will need to configure Hive to act as a
> proxy user.  This requires you to set up keytabs for the user running the
> Hive metastore and add hadoop.proxyuser.hive.hosts and hadoop.proxyuser.
> hive.groups to Hadoop's core-site.xml file.  See the Hadoop documentation
> on secure mode for your version of Hadoop (e.g., for Hadoop 2.5.1 it is at Hadoop
> in Secure Mode
> <http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/SecureMode.html>
> ).
>
>
> On Wed, Nov 16, 2016 at 5:43 PM, Manoj Murumkar <manoj.murumkar@gmail.com>
> wrote:
>
>> Further update:
>>
>> I see this:
>>
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>> |  dbname   | tabname  |  partname  |  type  |        state        |
>> workerid  |   starttime    |
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>> | Database  | Table    | Partition  | Type   | State               |
>> Worker    | Start Time     |
>> | txntest   | txntab3  | NULL       | MAJOR  | ready for cleaning  |
>> NULL      | 1479346924000  |
>> +-----------+----------+------------+--------+--------------
>> -------+-----------+----------------+--+
>>
>> However, I don't see cleaner actually run in hive logs (I am looking for
>> a string like "compactor.Cleaner").
>>
>>
>> On Wed, Nov 16, 2016 at 5:30 PM, Manoj Murumkar <manoj.murumkar@gmail.com
>> > wrote:
>>
>>> Quick update:
>>>
>>> After each compaction, files under base directory (for the buckets) have
>>> latest data. However, I am expecting to see all delta files (and
>>> directories) gone, as they should be merged in the base directory.
>>> Otherwise, we'll start seeing too many small files on HDFS which is a
>>> problem. Am I understanding this feature correctly in assuming so?
>>>
>>> On Wed, Nov 16, 2016 at 5:24 PM, Manoj Murumkar <
>>> manoj.murumkar@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to implement transaction feature in hive. I created
>>>> following table:
>>>>
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>> |                                     createtab_stmt
>>>> |
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>> | CREATE TABLE `txntest.txntab3`(
>>>>                                             |
>>>> |   `id` int,
>>>> |
>>>> |   `name` string)
>>>> |
>>>> | CLUSTERED BY (
>>>>                                             |
>>>> |   id)
>>>> |
>>>> | INTO 2 BUCKETS
>>>> |
>>>> | ROW FORMAT SERDE
>>>>                                               |
>>>> |   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
>>>> |
>>>> | STORED AS INPUTFORMAT
>>>>                                                |
>>>> |   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
>>>> |
>>>> | OUTPUTFORMAT
>>>> |
>>>> |   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
>>>> |
>>>> | LOCATION
>>>> |
>>>> |   'hdfs://or1010051016175.corp.adobe.com:8020/user/hive/wareho
>>>> use/txntest.db/txntab3'  |
>>>> | TBLPROPERTIES (
>>>>                                            |
>>>> |   'COLUMN_STATS_ACCURATE'='true',
>>>> |
>>>> |   'numFiles'='22',
>>>> |
>>>> |   'numRows'='90000',
>>>> |
>>>> |   'rawDataSize'='0',
>>>> |
>>>> |   'totalSize'='3564019',
>>>> |
>>>> |   'transactional'='true',
>>>> |
>>>> |   'transient_lastDdlTime'='1479329198')
>>>> |
>>>> +-----------------------------------------------------------
>>>> -----------------------------+--+
>>>>
>>>> I inserted 90000 rows in it in multiple iterations, so it created 22
>>>> files (as is visible above). I have run multiple compactions (major and
>>>> minor), but nothing seems to happen on HDFS. What am I missing?
>>>>
>>>> I have following configuration:
>>>>
>>>> Metastore:
>>>>
>>>> hive.compactor.initiator.on = true;
>>>> hive.compactor.worker.threads = 2;
>>>>
>>>> Client:
>>>>
>>>> hive.support.concurrency = true;
>>>> hive.exec.dynamic.partition.mode = nonstrict;
>>>> hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>>>>
>>>> Can someone point me in right direction? Compaction process did run
>>>> (verified via SHOW COMPACTIONS) and I also see we have base directory
>>>> created on HDFS.
>>>> I was expecting all the delta directories gone when major compation
>>>> runs.
>>>>
>>>> drwxrwxrwt   - admin    hive          0 2016-11-16 20:47
>>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021
>>>> -rw-r--r--   3 admin    hive     227916 2016-11-16 20:47
>>>> /user/hive/warehouse/txntest.db/txntab3/base_0000021/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_0000003
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000003_000000
>>>> 3/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_0000004
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000004_000000
>>>> 4/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_0000005
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:33
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000005_000000
>>>> 5/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_0000006
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000006_000000
>>>> 6/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_0000007
>>>> -rw-r--r--   3 nex37045 hive        636 2016-11-16 01:34
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000007_000000
>>>> 7/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_0000008
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000008_000000
>>>> 8/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_0000009
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:36
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000009_000000
>>>> 9/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_0000010
>>>> -rw-r--r--   3 nex37045 hive        640 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000010_000001
>>>> 0/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_0000011
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000011_000001
>>>> 1/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_0000012
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 01:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000012_000001
>>>> 2/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:44
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_0000013
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 01:44
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000013_000001
>>>> 3/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 01:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_0000014
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 01:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000014_000001
>>>> 4/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 02:02
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000015
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 02:02
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>>> 5/bucket_00000
>>>> drwxrwxrwt   - admin    hive          0 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_0000016
>>>> -rw-r--r--   3 admin    hive        531 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000015_000001
>>>> 6/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_0000016
>>>> -rw-r--r--   3 nex37045 hive        644 2016-11-16 02:03
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000016_000001
>>>> 6/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 20:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000017
>>>> -rw-r--r--   3 nex37045 hive     156395 2016-11-16 20:37
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>>> 7/bucket_00000
>>>> drwxrwxrwt   - admin    hive          0 2016-11-16 20:40
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_0000019
>>>> -rw-r--r--   3 admin    hive    2598250 2016-11-16 20:40
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000017_000001
>>>> 9/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_0000018
>>>> -rw-r--r--   3 nex37045 hive       4737 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000018_000001
>>>> 8/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_0000019
>>>> -rw-r--r--   3 nex37045 hive     192658 2016-11-16 20:39
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000019_000001
>>>> 9/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 20:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_0000020
>>>> -rw-r--r--   3 nex37045 hive     192835 2016-11-16 20:45
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000020_000002
>>>> 0/bucket_00000
>>>> drwxr-xr-x   - nex37045 hive          0 2016-11-16 20:46
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_0000021
>>>> -rw-r--r--   3 nex37045 hive     201206 2016-11-16 20:46
>>>> /user/hive/warehouse/txntest.db/txntab3/delta_0000021_000002
>>>> 1/bucket_00000
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Manoj
>>>>
>>>>
>>>
>>
>

Mime
View raw message