drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4143) REFRESH TABLE METADATA - Permission Issues with metadata files
Date Fri, 20 May 2016 22:13:13 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294362#comment-15294362
] 

ASF GitHub Bot commented on DRILL-4143:
---------------------------------------

Github user amansinha100 commented on the pull request:

    https://github.com/apache/drill/pull/470#issuecomment-220730748
  
    Committed in 3d92d2829


> REFRESH TABLE METADATA - Permission Issues with metadata files
> --------------------------------------------------------------
>
>                 Key: DRILL-4143
>                 URL: https://issues.apache.org/jira/browse/DRILL-4143
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Parquet
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: John Omernik
>            Assignee: Chunhui Shi
>              Labels: Metadata, Parquet, Permissions
>             Fix For: Future
>
>
> Summary of Refresh Metadata Issues confirmed by two different users on Drill User Mailing
list. (Title: REFRESH TABLE METADATA - Access Denied)
> This issue pertains to table METADATA and revolves around user authentication. 
> Basically, when the drill bits are running as one user, and the data is owned by another
user, there can be access denied issues on subsequent queries after issuing a REFRESH TABLE
METADATA command. 
> To troubleshoot what is actually happening, I turned on MapR Auditing (This is a handy
feature) and found that when I run a query (that is giving me access denied.. my query is
select count(1) from testtable ) Per MapR the user I am logged in as (dataowner) is trying
to do a create operation on the .drill.parquet_metadata file and it's failing with status:
17. Per Keys at MapR, "status 17 means errno 17 which means EEXIST. Looks like Drill is trying
to create a file that already exists." This seems to indicate that drill is perhaps trying
to create the .drill.parquet_metadata on each select as the dataowner user, but the permissions
(as seen below) don't allow it. 
> Here are the steps to reproduce:
> Enable Authentication. 
> Run all drill bits in the cluster as "drillbituser", then have the files owned by "dataowner".
Note the root of the table permissions are drwxrwxr-x but as Drill loads each partition it
loads them as drwxr-xr-x (all with dataowner:dataowner ownership). That may be something too,
the default permissions when creating a table?  Another note, in my setup, drillbituser is
in the group for dataowner.  Thus, they should always have read access. 
> # Authenticated as dataowner (this should have full permissions to all the data)
> Enter username for jdbc:drill:zk=zknode1:5181: dataowner
> Enter password for jdbc:drill:zk=zknode1:5181: **********
> 0: jdbc:drill:zk=zknode1> use dfs.dev;
> +-------+--------------------------------------+
> |  ok   |               summary                |
> +-------+--------------------------------------+
> | true  | Default schema changed to [dfs.dev]  |
> +-------+--------------------------------------+
> 1 row selected (0.307 seconds)
> # The query works fine with no table metadata
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> +-----------+
> |  EXPR$0   |
> +-----------+
> | 24565203  |
> +-----------+
> 1 row selected (3.392 seconds)
> # Refresh of metadata works under with no errors
> 0: jdbc:drill:zk=zknode1> refresh table metadata `testtable`;
> +-------+-------------------------------------------------------+
> |  ok   |                        summary                        |
> +-------+-------------------------------------------------------+
> | true  | Successfully updated metadata for table testtable.  |
> +-------+-------------------------------------------------------+
> 1 row selected (5.767 seconds)
>  
> # Trying to run the same query, it returns a access denied issue. 
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> Error: SYSTEM ERROR: IOException: 2127.7646.2950962 /data/dev/testtable/2015-11-12/.drill.parquet_metadata
(Permission denied)
>  
>  
> [Error Id: 7bfce2e7-f78d-4fba-b047-f4c85b471de4 on node1:31010] (state=,code=0)
>  
>  
> # Note how all the files are owned by the drillbituser. Per discussion on list, this
is normal 
>  
> $ find ./ -type f -name ".drill.parquet_metadata" -exec ls -ls {} \;
> 726 -rwxr-xr-x 1 drillbituser drillbituser 742837 Nov 30 14:27 ./2015-11-12/.drill.parquet_metadata
> 583 -rwxr-xr-x 1 drillbituser drillbituser 596146 Nov 30 14:27 ./2015-11-29/.drill.parquet_metadata
> 756 -rwxr-xr-x 1 drillbituser drillbituser 773811 Nov 30 14:27 ./2015-11-11/.drill.parquet_metadata
> 763 -rwxr-xr-x 1 drillbituser drillbituser 780829 Nov 30 14:27 ./2015-11-04/.drill.parquet_metadata
> 632 -rwxr-xr-x 1 drillbituser drillbituser 646851 Nov 30 14:27 ./2015-11-08/.drill.parquet_metadata
> 845 -rwxr-xr-x 1 drillbituser drillbituser 864421 Nov 30 14:27 ./2015-11-05/.drill.parquet_metadata
> 771 -rwxr-xr-x 1 drillbituser drillbituser 788823 Nov 30 14:27 ./2015-11-28/.drill.parquet_metadata
> 1273 -rwxr-xr-x 1 drillbituser drillbituser 1303168 Nov 30 14:27 ./2015-11-10/.drill.parquet_metadata
> 645 -rwxr-xr-x 1 drillbituser drillbituser 660028 Nov 30 14:27 ./2015-11-22/.drill.parquet_metadata
> 1017 -rwxr-xr-x 1 drillbituser drillbituser 1040469 Nov 30 14:27 ./2015-11-13/.drill.parquet_metadata
> 1280 -rwxr-xr-x 1 drillbituser drillbituser 1310552 Nov 30 14:27 ./2015-11-03/.drill.parquet_metadata
> 585 -rwxr-xr-x 1 drillbituser drillbituser 598973 Nov 30 14:27 ./2015-11-07/.drill.parquet_metadata
> 737 -rwxr-xr-x 1 drillbituser drillbituser 754103 Nov 30 14:27 ./2015-11-20/.drill.parquet_metadata
> 836 -rwxr-xr-x 1 drillbituser drillbituser 855363 Nov 30 14:27 ./2015-11-16/.drill.parquet_metadata
> 646 -rwxr-xr-x 1 drillbituser drillbituser 661134 Nov 30 14:27 ./2015-11-27/.drill.parquet_metadata
> 1156 -rwxr-xr-x 1 drillbituser drillbituser 1183378 Nov 30 14:27 ./2015-11-21/.drill.parquet_metadata
> 679 -rwxr-xr-x 1 drillbituser drillbituser 694551 Nov 30 14:27 ./2015-11-09/.drill.parquet_metadata
> 631 -rwxr-xr-x 1 drillbituser drillbituser 645500 Nov 30 14:27 ./2015-11-26/.drill.parquet_metadata
> 190 -rwxr-xr-x 1 drillbituser drillbituser 193798 Nov 30 14:27 ./2015-11-18/.drill.parquet_metadata
> 221 -rwxr-xr-x 1 drillbituser drillbituser 225342 Nov 30 14:27 ./2015-11-23/.drill.parquet_metadata
> 269 -rwxr-xr-x 1 drillbituser drillbituser 274988 Nov 30 14:27 ./2015-11-25/.drill.parquet_metadata
> 845 -rwxr-xr-x 1 drillbituser drillbituser 864615 Nov 30 14:27 ./2015-11-06/.drill.parquet_metadata
> 32 -rwxr-xr-x 1 drillbituser drillbituser 32221 Nov 30 14:27 ./2015-11-19/.drill.parquet_metadata
> 16468 -rwxr-xr-x 1 drillbituser drillbituser 16862958 Nov 30 14:27 ./.drill.parquet_metadata
>  
> # Once the files are deleted... 
> $ sudo find ./ -type f -name ".drill.parquet_metadata" -exec rm {} \;
> # Then the query works again 
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> +-----------+
> |  EXPR$0   |
> +-----------+
> | 24567700  |
> +-----------+
> 1 row selected (1.353 seconds)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message