hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Lin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-14482) Add and drop table partition is not audit logged in HMS
Date Fri, 21 Oct 2016 02:11:58 GMT

     [ https://issues.apache.org/jira/browse/HIVE-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eric Lin updated HIVE-14482:
----------------------------
    Attachment: Screen Shot 2016-10-21 at 1.06.33 pm.png

The error seems to be unrelated and there is no existing test coverage for HMS audit logs,
also it is pretty hard to test it.

Please advise if the patch can be accepted.

Thanks

> Add and drop table partition is not audit logged in HMS
> -------------------------------------------------------
>
>                 Key: HIVE-14482
>                 URL: https://issues.apache.org/jira/browse/HIVE-14482
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 2.1.0
>            Reporter: Eric Lin
>            Assignee: Eric Lin
>            Priority: Minor
>         Attachments: HIVE-14482.2.patch, HIVE-14482.patch, Screen Shot 2016-10-21 at
1.06.33 pm.png
>
>
> When running:
> {code}
> ALTER TABLE test DROP PARTITION (b=140);
> {code}
> I only see the following in the HMS log:
> {code}
> 2016-08-08 23:12:34,081 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
<PERFLOG method=get_table from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,082 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-2]:
2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,082 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
ugi=hive ip=xx.xx.xxx.xxx    cmd=source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,094 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
</PERFLOG method=get_table start=1470723154081 end=1470723154094 duration=13 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler
threadId=2 retryCount=0 error=false>
> 2016-08-08 23:12:34,095 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
<PERFLOG method=get_partitions_by_expr from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,095 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-2]:
2: source:xx.xx.xxx.xxx get_partitions_by_expr : db=default tbl=test
> 2016-08-08 23:12:34,096 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
ugi=hive ip=xx.xx.xxx.xxx    cmd=source:xx.xx.xxx.xxx get_partitions_by_expr : db=default
tbl=test
> 2016-08-08 23:12:34,112 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
</PERFLOG method=get_partitions_by_expr start=1470723154095 end=1470723154112 duration=17
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2 retryCount=0 error=false>
> 2016-08-08 23:12:34,172 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
<PERFLOG method=get_table from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,173 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-2]:
2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,173 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
ugi=hive ip=xx.xx.xxx.xxx    cmd=source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,186 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
</PERFLOG method=get_table start=1470723154172 end=1470723154186 duration=14 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler
threadId=2 retryCount=0 error=false>
> 2016-08-08 23:12:34,186 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
<PERFLOG method=get_table from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,187 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-2]:
2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,187 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
ugi=hive ip=xx.xx.xxx.xxx    cmd=source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,199 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
</PERFLOG method=get_table start=1470723154186 end=1470723154199 duration=13 from=org.apache.hadoop.hive.metastore.RetryingHMSHandler
threadId=2 retryCount=0 error=false>
> 2016-08-08 23:12:34,203 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
<PERFLOG method=drop_partitions_req from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,215 INFO  org.apache.hadoop.hive.metastore.ObjectStore: [pool-4-thread-2]:
JDO filter pushdown cannot be used: Filtering is supported only on partition keys of type
string
> 2016-08-08 23:12:34,226 ERROR org.apache.hadoop.hdfs.KeyProviderCache: [pool-4-thread-2]:
Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
> 2016-08-08 23:12:34,239 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-2]:
dropPartition() will move partition-directories to trash-directory.
> 2016-08-08 23:12:34,239 INFO  hive.metastore.hivemetastoressimpl: [pool-4-thread-2]:
deleting  hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO  org.apache.hadoop.fs.TrashPolicyDefault: [pool-4-thread-2]:
Moved: 'hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140' to trash at:
hdfs://<host-name>:8020/user/hive/.Trash/Current/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO  hive.metastore.hivemetastoressimpl: [pool-4-thread-2]:
Moved to trash: hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [pool-4-thread-2]:
</PERFLOG method=drop_partitions_req start=1470723154203 end=1470723154247 duration=44
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2 retryCount=0 error=false>
> {code}
> There is no entry in the "HiveMetaStore.audit" to show that partition b=140 was dropped.
> When we add a new partition, we can see the following:
> {code}
> 2016-08-08 23:04:48,534 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
ugi=hive ip=xx.xx.xxx.xxx        cmd=source:xx.xx.xxx.xxx append_partition : db=default tbl=test[130]
> {code}
> Ideally we should see the similar message when dropping partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message