kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-3626) Allow customization for Parquet path
Date Wed, 28 Nov 2018 07:42:00 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shaofeng SHI updated KYLIN-3626:
--------------------------------
    Fix Version/s: v2.6.0

> Allow customization for Parquet path
> ------------------------------------
>
>                 Key: KYLIN-3626
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3626
>             Project: Kylin
>          Issue Type: Sub-task
>          Components: Metadata
>            Reporter: Shaofeng SHI
>            Assignee: Chao Long
>            Priority: Major
>             Fix For: v2.6.0
>
>
> Due to the difference of HDFS and object store like S3/blob store, Kylin needs to allow
different path patterns for the files when putting to the underlying storage. 
>  
> For example, to HDFS, this pattern is good:
> /kylin/metadata_name/cube_name/segment_name/file1.parquet
>  
> But, the above pattern is not good for S3, since S3 uses the first couple bytes of the
path to do the hashing, so we'd better add salt ahead of the original pathname:
> /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet
> or do a reverse order:
> /segment_name/cube_name/metadata_name/kylin/file1.parquet
>  
> To get the flexibility, we should make this extensible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message