kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: Error using GlobalDictionary
Date Mon, 05 Jun 2017 16:28:50 GMT
Hi ShaoFeg,

This error is in the yarn application log.  The work dir is /kylin

Appears to be a bug in 1.6v

the conf should come from HadoopUtil.getCurrentConfiguration();

instead its creating a new hadoop conf object - therefore reverting to
local vs hdfs.

Thoughts?


On Mon, Jun 5, 2017 at 9:26 AM, ShaoFeng Shi <shaofengshi@apache.org> wrote:

> Hi Sonny, I need more info:
> 1) where you see this error trace, in kylin.log or in Mapreduce's log?
> 2) what's the configuration of "kylin.hdfs.working.dir" in
> conf/kylin.properties?
>
> 2017-06-05 23:58 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>
>> Hey thanks.  I actually do see the hadoop conf dir during start up in the
>> hbase classpath:
>>
>> /usr/hdp/2.4.2.0-258/hadoop/conf
>>
>> which has the correct core-site.xml with hdfs prop:
>>
>>     <property>
>>
>>       <name>fs.defaultFS</name>
>>
>>       <value>hdfs://nn_id</value>
>>
>>     </property>
>>
>>
>> Still not clear why its looking at Local.  Does AppendTrieDictionary use
>> the above hbase classpath when it creates a new hadoop configuration object
>> in readFields?... please advise...
>>
>>
>>
>> On Mon, Jun 5, 2017 at 8:29 AM, ShaoFeng Shi <shaofengshi@apache.org>
>> wrote:
>>
>>> They have upgraded to Kylin 2.0 I remember; Kylin uses "hbase" command
>>> to startup, so `hbase classpath` is used when run Kylin. You can check
>>> whether your *-site.xml are in the classpath and in a front position.
>>>
>>> 2017-06-05 22:18 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>
>>>> okay - checking. Is he running on HDP 2.4 and 1.6.0 version of kylin?
>>>> Where does kylin add core-site to CP?  in hdp core-site is located in dir
>>>> /etc/hadoop/conf.
>>>>
>>>> On Mon, Jun 5, 2017 at 7:07 AM, ShaoFeng Shi <shaofengshi@apache.org>
>>>> wrote:
>>>>
>>>>> Yesterday Kaisen (maintainer of GlobalDict) checked this, and he
>>>>> didn't find problem. In Meituan.com, there are many Cubes using the
>>>>> GlobalDictionary to implement the concice distinct count, and runs well.
I
>>>>> still suggest you check the environment configurations.
>>>>>
>>>>> 2017-06-05 21:31 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>
>>>>>> Does KYLIN-2192 fix this? Anyone run into this?
>>>>>>
>>>>>> On Sun, Jun 4, 2017 at 10:33 PM, Sonny Heer <sonnyheer@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Looks like a new hadoop conf is initialized and the hadoop
>>>>>>> FileSystem object is used after that:
>>>>>>>
>>>>>>>  Configuration conf = new Configuration();
>>>>>>>
>>>>>>> (FileSystem.get(filePath.toUri(), conf))
>>>>>>>
>>>>>>>
>>>>>>> Kylin appears to have find scripts for hive and hbase in the
bin
>>>>>>> install dir, but i don't see where it discovers/adds core-site
to the
>>>>>>> classpath.  I'm a little surprised no one has run into this yet...am
i
>>>>>>> missing something?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jun 4, 2017 at 10:28 PM, ShaoFeng Shi <
>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>
>>>>>>>> @kangkaisen, kaisen, any idea about this error?
>>>>>>>>
>>>>>>>> 2017-06-05 13:00 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>>>
>>>>>>>>> where do i verify this?  BTW all the other hive /hbase
commands
>>>>>>>>> seem to work fine up to that point.  Also this only happens
when I
>>>>>>>>> introduce GlobalDictionary.  Thanks for the response.
>>>>>>>>>
>>>>>>>>> On Sun, Jun 4, 2017 at 9:50 PM, ShaoFeng Shi <
>>>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> It looks like seeking the file in local disk, instead
of HDFS.
>>>>>>>>>> You need check whether the proper core-site.xml (with
hdfs as default file
>>>>>>>>>> system) is used by kylin. Or you can upgrade to kylin
2.0 to see whether it
>>>>>>>>>> works.
>>>>>>>>>>
>>>>>>>>>> 2017-06-05 11:25 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Any ideas on this?  Not sure ,but appears the
dictionary is
>>>>>>>>>>> looking on local FS vs HDFS? ...what am i missing
here?
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jun 3, 2017 at 9:05 PM, Sonny Heer <sonnyheer@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Kylin version 1.6.0
>>>>>>>>>>>>
>>>>>>>>>>>> Our data has High cardinality columns that
require count
>>>>>>>>>>>> distinct measures.  Therefore using GlobalDictionary.
 The .index file
>>>>>>>>>>>> exists on HDFS, but kylin errors out with
FileNotFound exception (see
>>>>>>>>>>>> below).  RowKey is set to "dict".  Any ideas
if this is a known issue or
>>>>>>>>>>>> something misconfigured...?
>>>>>>>>>>>>
>>>>>>>>>>>> At stage: "#7 Step Name: Build Base Cuboid
Data"
>>>>>>>>>>>>
>>>>>>>>>>>> Error:
>>>>>>>>>>>>
>>>>>>>>>>>> Caused by: java.io.FileNotFoundException:
File
>>>>>>>>>>>> /kylin/kylin_metadata/resources/GlobalDict/dict/MY_HIVE_DB.MY_HIVE_VIEW/MY_MEASURE/.index
>>>>>>>>>>>> does not exist
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>>>>>> ileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>>>>>> ileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:
>>>>>>>>>>>> 822)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>>>>>> ileSystem.getFileStatus(RawLocalFileSystem.java:599)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.FilterFil
>>>>>>>>>>>> eSystem.getFileStatus(FilterFileSystem.java:421)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>>>>>>>>> ileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.j
>>>>>>>>>>>> ava:140)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>>>>>>>>> ileSystem.open(ChecksumFileSystem.java:341)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.AppendTr
>>>>>>>>>>>> ieDictionary.readFields(AppendTrieDictionary.java:1076)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:74)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:34)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.common.persis
>>>>>>>>>>>> tence.ResourceStore.getResource(ResourceStore.java:146)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryManager.load(DictionaryManager.java:421)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryManager$1.load(DictionaryManager.java:103)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryManager$1.load(DictionaryManager.java:100)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache$LoadingValueReference.loadFuture(LocalCache.java:3568)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache$Segment.loadSync(LocalCache.java:2350)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache$Segment.lockedGetOrLoad(LocalCache.java:2313)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache$Segment.get(LocalCache.java:2228)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache.get(LocalCache.java:3965)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache.getOrLoad(LocalCache.java:3969)
>>>>>>>>>>>>
>>>>>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>>>>>> ache$LocalManualCache.get(LocalCache.java:4829)
>>>>>>>>>>>>
>>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>>> ryManager.getDictionaryInfo(DictionaryManager.java:120)
>>>>>>>>>>>>
>>>>>>>>>>>>         ... 17 more
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Pushpinder S. Heer
>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Pushpinder S. Heer
>>>>>>>>> Senior Software Engineer
>>>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> Pushpinder S. Heer
>>>>>>> Senior Software Engineer
>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>> <(509)%20884-2574>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> Pushpinder S. Heer
>>>>>> Senior Software Engineer
>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Pushpinder S. Heer
>>>> Senior Software Engineer
>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>>
>>
>> Pushpinder S. Heer
>> Senior Software Engineer
>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 


Pushpinder S. Heer
Senior Software Engineer
m: 360-434-4354 h: 509-884-2574

Mime
View raw message