kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "刘成军" <li...@zqykj.com>
Subject 回复:Help for job build: directory item limit exceeded exception
Date Tue, 27 Nov 2018 16:18:28 GMT
    I solve the problem by adding hbase.fs.tmp.dir and hadoop.tmp.dir to hdfs-site.xml, and
copy the hdfs-site.xml to $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/ .
    I need to debug src code to check:
        1) whether the hadoop.tmp.dir affects?
        2) or the hdfs-site.xml file in classes directory affects?
 
 

------------------------------------------------------------------
发件人:刘成军 <liucj@zqykj.com>
发送时间:2018年11月21日(星期三) 21:50
收件人:user <user@kylin.apache.org>
主 题:回复:Help for job build: directory item limit exceeded exception

Hi all,
    I have try the following method:
    1. try to set "dfs.namenode.fs-limits.max-directory-items" in hdfs-site.xml
 I restart the kylin service.(I cannot restart hdfs service because in use).
 Result: failed with the same error:
                The directory item limit of /tmp is exceeded:limit=1028576 items=1028576

    2. try to add property hbase.fs.tmp.dir(=/wy/tmp/hbase) property to hbase-site.xml
        I restart the kylin service.
 Result: failed with the same error:
                The directory item limit of /tmp is exceeded:limit=1028576 items=1028576

 Any one has other suggestions?

     PS:
        Hadoop env:
            CDH 5.13 with kerberos enabled(many nodes)
        kylin env:
            kylin 2.4.0 with hadoop client installed (not managed by CDH management service)
 
------------------------------------------------------------------
发件人:刘成军 <liucj@zqykj.com>
发送时间:2018年11月20日(星期二) 23:25
收件人:user <user@kylin.apache.org>; JiaTao Tao <taojiatao@gmail.com>
主 题:回复:Help for job build: directory item limit exceeded exception

JiaTao:
    Thx for your reply, i wil try it late.

But i have check the source code, the code set with 
    Configuration conf = HBaseConfiguration.create(HadoopUtil.getCurrentConfiguration());
    ...
    if (StringUtils.isBlank(conf.get("hbase.fs.tmp.dir"))) {
            conf.set("hbase.fs.tmp.dir", "/tmp");
    }

    My question is i have set the hbase.fs.tmp.dir property in hbase-site.xml( and restart
kylin), but it still write data to /tmp directory.

    Any one has other suggestion?

------------------------------------------------------------------
发件人:JiaTao Tao <taojiatao@gmail.com>
发送时间:2018年11月20日(星期二) 22:59
收件人:user <user@kylin.apache.org>; 刘成军 <liucj@zqykj.com>
主 题:Re: Help for job build: directory item limit exceeded exception

Hi 

Seems that there are too many files in "/tmp", try to modify the config below in "hdfs-site.xml".

<property>
  <name>dfs.namenode.fs-limits.max-directory-items</name>
  <value>1048576</value>
  <description>Defines the maximum number of items that a directory may
      contain. Cannot set the property to a value less than 1 or more than
      6400000.</description>
</property>

And here's a link for you: https://tw.saowen.com/a/fa6aea71141c6241f496093d9b0feb0c87bf4c30cf40b4ff6fdc065a8228231a.
It is generally recommended that users do not tune these values except in very unusual circumstances.

刘成军 <liucj@zqykj.com> 于2018年11月20日周二 上午11:01写道:
Hi,
    Build cube from my cdh(5.13 cluster, with kerberos enabled),  when the Job comes with
the step(#10): Convert Cuboid Data to HFile,
it comes the followed exception:
 
 I also change the hbase config(hbase.fs.tmp.dir=/usr/tmp/hbase) in my hbase-site.xml, but
it comes the same exception;
 How can i do with it?

PS:
   I did not have the permission to delete the data in /tmp. 
Best Regards
-----------------------------
刘成军(Gavin)
————————————————
手机:13913036255





-- 

Regards!
Aron Tao



Mime
View raw message