hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Re: Re: where kylin global dict store?
Date Fri, 06 Jan 2017 03:10:55 GMT
I find the Globel Dictionary infomation in HBase table ,sorry for disturb
Date: 2017-01-06 10:48
To: user; dev
CC: user
Subject: Re: Re: where kylin global dict store?
I am sorry for not making a point for my question

I  used a Global Dictionary cloumn when I build a cube .but I can't find anything with this
Global Dictionary  infomation related with my cube in HBase table 'kylin_metadata' 
I searched all rowkeys in HBase table 'kylin_metadata' and filter out  rowkeys prefix with
'/cube' ,'/model_desc' ,'/table' related with my cube name or my model name 
Nothing with Global Dictionary infomation can be found.

than I searched HDFS files in  directory /kylin/kylin_metadata  and find nothing sadly.

I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build
this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted
by me.

finally I accoding to the stack infomation and find code in engine-mr module which is source
code in kylin 1.6.0
in the class AbstracHadoopJob have a method
attachKylinPropsAndMatedata(CubeInstance  cube,Configuration conf) throw Exception{

MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());

        // write cube / model_desc / cube_desc / dict / table
        ArrayList<String> dumpList = new ArrayList<String>();

        for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
            TableDesc table = metaMgr.getTableDesc(tableName);
            List<String> dependentResources = SourceFactory.getMRDependentResources(table);
        for (CubeSegment segment : cube.getSegments()) {

        attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);


the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand
how dumpList be filled with data. thanks to your  annotations .I searched all rowkeys in HBase
table 'kylin_metadata'and filter out that which starts with  prefix '/cube' or '/model_desc'
or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my
cube used Global Dictionary
I don't kown whether or not it clearly with my question?
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved
in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.

2017-01-06 8:52 GMT+08:00 zy <>:
I think you could find the more content about kylin from the internet..i give you as follow..

------------------ Original ------------------
From: ""<>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<>; "user"<>;
Subject: where kylin global dict store?

Hi all :
   I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:

in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
    "uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
    "last_modified": 1483607960443,
    "version": "1.6.0",
    "name": "dmp_user_info",
    "owner": "ADMIN",
    "descriptor": "dmp_user_info",
    "cost": 50,
    "status": "DISABLED",
    "segments": [
            "uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
            "name": "20161001000000_20161220000000",
            "storage_location_identifier": "KYLIN_3WV3VKHD99",
            "date_range_start": 1475280000000,
            "date_range_end": 1482192000000,
            "source_offset_start": 0,
            "source_offset_end": 0,
            "status": "NEW",
            "size_kb": 0,
            "input_records": 0,
            "input_records_size": 0,
            "last_build_time": 0,
            "last_build_job_id": null,
            "create_time_utc": 1483607736083,
            "cuboid_shard_nums": { },
            "total_shards": 0,
            "blackout_cuboids": [ ],
            "binary_signature": null,
            "dictionaries": {
                "KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
                "KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
                "KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
                "KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c-
            "snapshots": null,
            "index_path": null,
            "rowkey_stats": [
    "create_time_utc": 1483605652669,
    "size_kb": 0,
    "input_records_count": 0,
    "input_records_size": 0
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related
to my cube and I can't find anything with Global Dictionary with my cube
Help me !

Best regards,

Shaofeng Shi 史少锋

View raw message