kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: Error using GlobalDictionary
Date Mon, 05 Jun 2017 14:18:57 GMT
okay - checking. Is he running on HDP 2.4 and 1.6.0 version of kylin?
Where does kylin add core-site to CP?  in hdp core-site is located in dir
/etc/hadoop/conf.

On Mon, Jun 5, 2017 at 7:07 AM, ShaoFeng Shi <shaofengshi@apache.org> wrote:

> Yesterday Kaisen (maintainer of GlobalDict) checked this, and he didn't
> find problem. In Meituan.com, there are many Cubes using the
> GlobalDictionary to implement the concice distinct count, and runs well. I
> still suggest you check the environment configurations.
>
> 2017-06-05 21:31 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>
>> Does KYLIN-2192 fix this? Anyone run into this?
>>
>> On Sun, Jun 4, 2017 at 10:33 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>
>>> Looks like a new hadoop conf is initialized and the hadoop FileSystem
>>> object is used after that:
>>>
>>>  Configuration conf = new Configuration();
>>>
>>> (FileSystem.get(filePath.toUri(), conf))
>>>
>>>
>>> Kylin appears to have find scripts for hive and hbase in the bin install
>>> dir, but i don't see where it discovers/adds core-site to the classpath.
>>> I'm a little surprised no one has run into this yet...am i missing
>>> something?
>>>
>>>
>>>
>>> On Sun, Jun 4, 2017 at 10:28 PM, ShaoFeng Shi <shaofengshi@apache.org>
>>> wrote:
>>>
>>>> @kangkaisen, kaisen, any idea about this error?
>>>>
>>>> 2017-06-05 13:00 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>
>>>>> where do i verify this?  BTW all the other hive /hbase commands seem
>>>>> to work fine up to that point.  Also this only happens when I introduce
>>>>> GlobalDictionary.  Thanks for the response.
>>>>>
>>>>> On Sun, Jun 4, 2017 at 9:50 PM, ShaoFeng Shi <shaofengshi@apache.org>
>>>>> wrote:
>>>>>
>>>>>> It looks like seeking the file in local disk, instead of HDFS. You
>>>>>> need check whether the proper core-site.xml (with hdfs as default
file
>>>>>> system) is used by kylin. Or you can upgrade to kylin 2.0 to see
whether it
>>>>>> works.
>>>>>>
>>>>>> 2017-06-05 11:25 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>
>>>>>>> Any ideas on this?  Not sure ,but appears the dictionary is looking
>>>>>>> on local FS vs HDFS? ...what am i missing here?
>>>>>>>
>>>>>>> On Sat, Jun 3, 2017 at 9:05 PM, Sonny Heer <sonnyheer@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Kylin version 1.6.0
>>>>>>>>
>>>>>>>> Our data has High cardinality columns that require count
distinct
>>>>>>>> measures.  Therefore using GlobalDictionary.  The .index
file exists on
>>>>>>>> HDFS, but kylin errors out with FileNotFound exception (see
below).  RowKey
>>>>>>>> is set to "dict".  Any ideas if this is a known issue or
something
>>>>>>>> misconfigured...?
>>>>>>>>
>>>>>>>> At stage: "#7 Step Name: Build Base Cuboid Data"
>>>>>>>>
>>>>>>>> Error:
>>>>>>>>
>>>>>>>> Caused by: java.io.FileNotFoundException: File
>>>>>>>> /kylin/kylin_metadata/resources/GlobalDict/dict/MY_HIVE_DB.MY_HIVE_VIEW/MY_MEASURE/.index
>>>>>>>> does not exist
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>> ileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>> ileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>>>>> ileSystem.getFileStatus(RawLocalFileSystem.java:599)
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.FilterFil
>>>>>>>> eSystem.getFileStatus(FilterFileSystem.java:421)
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>>>>> ileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.j
>>>>>>>> ava:140)
>>>>>>>>
>>>>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>>>>> ileSystem.open(ChecksumFileSystem.java:341)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.AppendTr
>>>>>>>> ieDictionary.readFields(AppendTrieDictionary.java:1076)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:74)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:34)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.common.persis
>>>>>>>> tence.ResourceStore.getResource(ResourceStore.java:146)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryManager.load(DictionaryManager.java:421)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryManager$1.load(DictionaryManager.java:103)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryManager$1.load(DictionaryManager.java:100)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache$LoadingValueReference.loadFuture(LocalCache.java:3568)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache$Segment.loadSync(LocalCache.java:2350)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache$Segment.lockedGetOrLoad(LocalCache.java:2313)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache$Segment.get(LocalCache.java:2228)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache.get(LocalCache.java:3965)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache.getOrLoad(LocalCache.java:3969)
>>>>>>>>
>>>>>>>>         at com.google.common.cache.LocalC
>>>>>>>> ache$LocalManualCache.get(LocalCache.java:4829)
>>>>>>>>
>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>> ryManager.getDictionaryInfo(DictionaryManager.java:120)
>>>>>>>>
>>>>>>>>         ... 17 more
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> Pushpinder S. Heer
>>>>>>> Senior Software Engineer
>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>> <(509)%20884-2574>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi 史少锋
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> Pushpinder S. Heer
>>>>> Senior Software Engineer
>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>> Pushpinder S. Heer
>>> Senior Software Engineer
>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>
>>
>>
>>
>> --
>>
>>
>> Pushpinder S. Heer
>> Senior Software Engineer
>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 


Pushpinder S. Heer
Senior Software Engineer
m: 360-434-4354 h: 509-884-2574

Mime
View raw message