kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: Error using GlobalDictionary
Date Mon, 05 Jun 2017 05:33:48 GMT
Looks like a new hadoop conf is initialized and the hadoop FileSystem
object is used after that:

 Configuration conf = new Configuration();

(FileSystem.get(filePath.toUri(), conf))


Kylin appears to have find scripts for hive and hbase in the bin install
dir, but i don't see where it discovers/adds core-site to the classpath.
I'm a little surprised no one has run into this yet...am i missing
something?



On Sun, Jun 4, 2017 at 10:28 PM, ShaoFeng Shi <shaofengshi@apache.org>
wrote:

> @kangkaisen, kaisen, any idea about this error?
>
> 2017-06-05 13:00 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>
>> where do i verify this?  BTW all the other hive /hbase commands seem to
>> work fine up to that point.  Also this only happens when I introduce
>> GlobalDictionary.  Thanks for the response.
>>
>> On Sun, Jun 4, 2017 at 9:50 PM, ShaoFeng Shi <shaofengshi@apache.org>
>> wrote:
>>
>>> It looks like seeking the file in local disk, instead of HDFS. You need
>>> check whether the proper core-site.xml (with hdfs as default file system)
>>> is used by kylin. Or you can upgrade to kylin 2.0 to see whether it works.
>>>
>>> 2017-06-05 11:25 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>
>>>> Any ideas on this?  Not sure ,but appears the dictionary is looking on
>>>> local FS vs HDFS? ...what am i missing here?
>>>>
>>>> On Sat, Jun 3, 2017 at 9:05 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>>>
>>>>> Kylin version 1.6.0
>>>>>
>>>>> Our data has High cardinality columns that require count distinct
>>>>> measures.  Therefore using GlobalDictionary.  The .index file exists
on
>>>>> HDFS, but kylin errors out with FileNotFound exception (see below). 
RowKey
>>>>> is set to "dict".  Any ideas if this is a known issue or something
>>>>> misconfigured...?
>>>>>
>>>>> At stage: "#7 Step Name: Build Base Cuboid Data"
>>>>>
>>>>> Error:
>>>>>
>>>>> Caused by: java.io.FileNotFoundException: File
>>>>> /kylin/kylin_metadata/resources/GlobalDict/dict/MY_HIVE_DB.MY_HIVE_VIEW/MY_MEASURE/.index
>>>>> does not exist
>>>>>
>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>> ileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
>>>>>
>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>> ileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
>>>>>
>>>>>         at org.apache.hadoop.fs.RawLocalF
>>>>> ileSystem.getFileStatus(RawLocalFileSystem.java:599)
>>>>>
>>>>>         at org.apache.hadoop.fs.FilterFil
>>>>> eSystem.getFileStatus(FilterFileSystem.java:421)
>>>>>
>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>> ileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
>>>>>
>>>>>         at org.apache.hadoop.fs.ChecksumF
>>>>> ileSystem.open(ChecksumFileSystem.java:341)
>>>>>
>>>>>         at org.apache.kylin.dict.AppendTr
>>>>> ieDictionary.readFields(AppendTrieDictionary.java:1076)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:74)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryInfoSerializer.deserialize(DictionaryInfoSerializer.java:34)
>>>>>
>>>>>         at org.apache.kylin.common.persis
>>>>> tence.ResourceStore.getResource(ResourceStore.java:146)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryManager.load(DictionaryManager.java:421)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryManager$1.load(DictionaryManager.java:103)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryManager$1.load(DictionaryManager.java:100)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache$LoadingValueReference.loadFuture(LocalCache.java:3568)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache$Segment.loadSync(LocalCache.java:2350)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache$Segment.lockedGetOrLoad(LocalCache.java:2313)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache$Segment.get(LocalCache.java:2228)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache.get(LocalCache.java:3965)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache.getOrLoad(LocalCache.java:3969)
>>>>>
>>>>>         at com.google.common.cache.LocalC
>>>>> ache$LocalManualCache.get(LocalCache.java:4829)
>>>>>
>>>>>         at org.apache.kylin.dict.Dictiona
>>>>> ryManager.getDictionaryInfo(DictionaryManager.java:120)
>>>>>
>>>>>         ... 17 more
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Pushpinder S. Heer
>>>> Senior Software Engineer
>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>>
>>
>> Pushpinder S. Heer
>> Senior Software Engineer
>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 


Pushpinder S. Heer
Senior Software Engineer
m: 360-434-4354 h: 509-884-2574

Mime
View raw message