kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy Liu <billy...@apache.org>
Subject Re: [外部邮件] Re: [外部邮件] 回复:kylin drop cube segment Htable 仍然存在
Date Wed, 18 Oct 2017 16:40:35 GMT
Agree. A new patch is needed in KYLIN-2846. It's not a bug still, but an
unfinished improvement I think. Let's continue to improve this issue.

2017-10-18 21:05 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org>:

> Hi Yaowu,
>
> Could you please update this patch in KYLIN-2846? Storage cleanup should
> also support customized namespace. Thanks.
>
> 2017-10-18 20:37 GMT+08:00 曾耀武 <zeng.yaowu@immomo.com>:
>
>>
>> Yeah I have checked it was a bug , because I  used hbase namespace
>> “kylin” with the patch "https://issues.apache.org/jir
>> a/secure/attachment/12885320/KYLIN-2846-001.patch”
>>
>> And I modified the code as  below and it works  well:
>>
>>
>> MOMO@MOMOdeMacBook-Pro-5:~/gitworkspace/kylin2.0/kylin/dist/apache-kylin-2.1.0-bin/tool$
>> git diff ../../../server-base/src/main/java/org/apache/kylin/rest/job
>> /StorageCleanJobHbaseUtil.java
>> diff --git a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
>> b/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> index 3728ea1..937d02d 100644
>> --- a/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> +++ b/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> @@ -46,15 +46,28 @@ public class StorageCleanJobHbaseUtil {
>>
>>      public static void cleanUnusedHBaseTables(boolean delete, int
>> deleteTimeout) throws IOException {
>>          Configuration conf = HBaseConfiguration.create();
>> -        CubeManager cubeMgr = CubeManager.getInstance(KylinC
>> onfig.getInstanceFromEnv());
>> +        KylinConfig config = KylinConfig.getInstanceFromEnv();
>> +        CubeManager cubeMgr = CubeManager.getInstance(config);
>>          // get all kylin hbase tables
>>          try (HBaseAdmin hbaseAdmin = new HBaseAdmin(conf)) {
>> -            String tableNamePrefix = IRealizationConstants.SharedHb
>> aseStorageLocationPrefix;
>> +            String namespace = config.getHBaseStorageNameSpace();
>> +            StringBuffer sb = new StringBuffer();
>> +            String tableNamePrefix = null;
>> +            if(namespace.equals("default")){
>> +                tableNamePrefix = IRealizationConstants.SharedHb
>> aseStorageLocationPrefix;
>> +            }else{
>> +                sb.append(config.getHBaseStor
>> ageNameSpace()).append(":");
>> +                sb.append(IRealizationConstan
>> ts.SharedHbaseStorageLocationPrefix);
>> +                tableNamePrefix = sb.toString();
>> +            }
>> +
>>
>>
>>
>>
>>
>> 发件人: ShaoFeng Shi <shaofengshi@apache.org>
>> 答复: user <user@kylin.apache.org>
>> 日期: 2017年10月17日 星期二 下午4:16
>> 至: user <user@kylin.apache.org>
>> 主题: [外部邮件] Re: [外部邮件] 回复:kylin drop cube segment Htable
仍然存在
>>
>> Hi yaowu,
>>
>> The StorageCleanupJob will delete the hbase tables that has no reference;
>> If you found it didn't work, please check the output log of that command.
>> It should say something.
>>
>> 在 2017年10月17日 下午3:39,yuyong.zhai <yuyong.zhai@ele.me>写道:
>>
>>>
>>> sh metastore.sh clean --delete true
>>>
>>> sh kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob
>>> --delete true
>>>  原始邮件
>>> *发件人:* 曾耀武<zeng.yaowu@immomo.com>
>>> *收件人:* user<user@kylin.apache.org>
>>> *发送时间:* 2017年10月17日(周二) 15:23
>>> *主题:* Re: [外部邮件] 回复:kylin drop cube segment Htable 仍然存在
>>>
>>>  这个我看了,执行之后发现只是删除了 hive 中间表  kylin_intermediate_*
任务失败遗留的中间表
>>>  和 hdfs://nameservice1:8020/kylin/kylin-kylin_metadata/  目录下没用的临时目录,
>>> hbase 里的 segment 表仍然存在的
>>>
>>> 发件人: "yuyong.zhai" <yuyong.zhai@ele.me>
>>> 答复: user <user@kylin.apache.org>
>>> 日期: 2017年10月17日 星期二 下午3:04
>>> 至: user <user@kylin.apache.org>
>>> 主题: [外部邮件] 回复:kylin drop cube segment Htable 仍然存在
>>>
>>> cleanup storage
>>>
>>> http://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>>>
>>>  原始邮件
>>> *发件人:* 曾耀武<zeng.yaowu@immomo.com>
>>> *收件人:* user<user@kylin.apache.org>
>>> *发送时间:* 2017年10月17日(周二) 14:02
>>> *主题:* kylin drop cube segment Htable 仍然存在
>>>
>>>
>>>
>>> 嗨,all ,在清理没用的cube (kylin2.1.0) 时,发现cube 被purge和drop
之后,对应的segment  hbase
>>> 表仍然在
>>> base中存在,只是在kylinmeta 表的里的元信息被清除,  这是我的特例还是为了数据安全特意让他存在,
>>> 只能人工删除相关的表呢?
>>>
>>>
>>> Best regards
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

Mime
View raw message