hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "446463844@qq.com" <446463...@qq.com>
Subject Re: Re: where kylin global dict store?
Date Fri, 06 Jan 2017 02:48:47 GMT
I am sorry for not making a point for my question

I  used a Global Dictionary cloumn when I build a cube .but I can't find anything with this
Global Dictionary  infomation related with my cube in HBase table 'kylin_metadata' 
I searched all rowkeys in HBase table 'kylin_metadata' and filter out  rowkeys prefix with
'/cube' ,'/model_desc' ,'/table' related with my cube name or my model name 
Nothing with Global Dictionary infomation can be found.

than I searched HDFS files in  directory /kylin/kylin_metadata  and find nothing sadly.


I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build
this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted
by me.

finally I accoding to the stack infomation and find code in engine-mr module which is source
code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance  cube,Configuration conf) throw Exception{

MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());

        // write cube / model_desc / cube_desc / dict / table
        ArrayList<String> dumpList = new ArrayList<String>();
        dumpList.add(cube.getResourcePath());
        dumpList.add(cube.getDescriptor().getModel().getResourcePath());
        dumpList.add(cube.getDescriptor().getResourcePath());

        for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
            TableDesc table = metaMgr.getTableDesc(tableName);
            dumpList.add(table.getResourcePath());
            List<String> dependentResources = SourceFactory.getMRDependentResources(table);
            dumpList.addAll(dependentResources);
        }
        for (CubeSegment segment : cube.getSegments()) {
            dumpList.addAll(segment.getDictionaryPaths());
        }

        attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}

```


the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand
how dumpList be filled with data. thanks to your  annotations .I searched all rowkeys in HBase
table 'kylin_metadata'and filter out that which starts with  prefix '/cube' or '/model_desc'
or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my
cube used Global Dictionary
I don't kown whether or not it clearly with my question?




446463844@qq.com
 
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved
in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.

2017-01-06 8:52 GMT+08:00 zy <346453003@qq.com>:
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin




------------------ Original ------------------
From: "446463844@qq.com"<446463844@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<dev@kylin.apache.org>; "user"<user@hive.apache.org>;
Subject: where kylin global dict store?



Hi all :
   I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:

in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
    "uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
    "last_modified": 1483607960443,
    "version": "1.6.0",
    "name": "dmp_user_info",
    "owner": "ADMIN",
    "descriptor": "dmp_user_info",
    "cost": 50,
    "status": "DISABLED",
    "segments": [
        {
            "uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
            "name": "20161001000000_20161220000000",
            "storage_location_identifier": "KYLIN_3WV3VKHD99",
            "date_range_start": 1475280000000,
            "date_range_end": 1482192000000,
            "source_offset_start": 0,
            "source_offset_end": 0,
            "status": "NEW",
            "size_kb": 0,
            "input_records": 0,
            "input_records_size": 0,
            "last_build_time": 0,
            "last_build_job_id": null,
            "create_time_utc": 1483607736083,
            "cuboid_shard_nums": { },
            "total_shards": 0,
            "blackout_cuboids": [ ],
            "binary_signature": null,
            "dictionaries": {
                "KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
                "KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
                "KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
                "KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c-
.dict"
            },
            "snapshots": null,
            "index_path": null,
            "rowkey_stats": [
                [
                    "SEX",
                    3,
                    1
                ],
                [
                    "BIRTHDAY",
                    9193,
                    2
                ],
                [
                    "NATIVE_PLACE",
                    2268,
                    2
                ],
                [
                    "LOCATION",
                    15,
                    1
                ]
            ]
        }
    ],
    "create_time_utc": 1483605652669,
    "size_kb": 0,
    "input_records_count": 0,
    "input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related
to my cube and I can't find anything with Global Dictionary with my cube
Help me !



446463844@qq.com



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message