kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: AppendTrieDictionary with GlobalDictionary 1.6
Date Mon, 26 Jun 2017 21:06:23 GMT
Yeah that is what we ended up doing.  moved to integer and it doesn't
create a dict then.  Does Kylin have a way to set different m/r properties
per job / step?    the Build base cuboid step requires more memory
(*kylin.job.mr.config.override.mapreduce.map.memory.mb)
 *but convert to HFile requires wider and less mb.

On Sun, Jun 25, 2017 at 6:09 PM, ShaoFeng Shi <shaofengshi@apache.org>
wrote:

> Hi Sonny,
>
> If the column type is not integer, Kylin need build dictionary for it, as
> Bitmap only accept integer as input. In your case, I think it is because
> the column type is bigint, which exceeds the scope of integer, so Kylin
> need a dict; If not specify using GD, it will use default dictionary, which
> causes OOM when the cardinality is high.
>
> If your USER_ID is under integer's scope, you can try to change the data
> type to INT. Hope this clarifies your question.
>
> 2017-06-24 4:18 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>
>>
>> According to this logic it still will build a dictionary (if not GD),
>> since count distinct is bitmap:
>>
>>
>> getAllColumnsHaveDictionary
>>
>>         // dictionaries in dimensions
>>
>>         for (RowKeyColDesc rowKeyColDesc : rowkey.getRowKeyColumns()) {
>>
>>             TblColRef colRef = rowKeyColDesc.getColRef();
>>
>>             if (rowkey.isUseDictionary(colRef)) {
>>
>>                 result.add(colRef);
>>
>>             }
>>
>>         }
>>
>>
>>         // dictionaries in measures
>>
>>         for (MeasureDesc measure : measures) {
>>
>>             MeasureType<?> aggrType = measure.getFunction().getMeasu
>> reType();
>>
>>             result.addAll(aggrType.getColumnsNeedDictionary(measure.
>> getFunction()));
>>
>>         }
>>
>>
>> I don't know enough of about the overall design, but it appears if i have
>> any type set for dimension and it is being used as a count distinct
>> measure.  then it will fall into the second for loop and it is a column
>> that needs a dictionary because its bitmap (getcolumnsNeedDictionary
>> method).
>>
>>
>> It is still running out of java heap.  where is this actually running?
>>  is it on the kylin server?  I see it uses ToolRunner in that job without
>> mapper/reducer class defined.  Thanks.
>>
>>
>>
>>
>>
>> On Fri, Jun 23, 2017 at 12:12 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>
>>> If it is fix length dimension and also a count distinct measure.  the
>>> hive type is bigint.  then should that be building a dictionary or not?
>>>
>>> On Fri, Jun 23, 2017 at 11:08 AM, Sonny Heer <sonnyheer@gmail.com>
>>> wrote:
>>>
>>>> Another question.  Is there any way to set properties per step in cube
>>>> building?
>>>>
>>>> On Fri, Jun 23, 2017 at 6:56 AM, Sonny Heer <sonnyheer@gmail.com>
>>>> wrote:
>>>>
>>>>> Yeah...it is.  making it fix length doesn't require dict.  I thought
>>>>> it was int in hive, but yah its bigint.  It got past that, but is now
stuck
>>>>> in " #16 Step Name: Convert Cuboid Data to HFile"
>>>>>
>>>>> Now the yarn container errors during HFile step:
>>>>>
>>>>> 2017-06-23 10:23:23,249 INFO [AsyncDispatcher event handler]
>>>>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>>>>> Diagnostics report from attempt_1496215660925_10363_m_000007_0:
>>>>> Container killed by the ApplicationMaster.
>>>>>
>>>>> Container killed on request. Exit code is 143
>>>>>
>>>>> Digging into this a bit more today.  Thanks for the support
>>>>>
>>>>> On Fri, Jun 23, 2017 at 4:24 AM, ShaoFeng Shi <shaofengshi@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Is the data type of "USER_ID" is bigInt (long)?
>>>>>> https://github.com/apache/kylin/blob/kylin-1.6.0/core-metada
>>>>>> ta/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasu
>>>>>> reType.java#L159
>>>>>>
>>>>>> Please provide detail metadata for trouble shooting, that is
>>>>>> important for analysis; otherwise we can only guess, but there are
many
>>>>>> possiblilies cause a problem...
>>>>>>
>>>>>> 2017-06-23 14:31 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>
>>>>>>> It's a dimension and count distinct measure.  No GD
>>>>>>>
>>>>>>> On Thu, Jun 22, 2017 at 11:27 PM ShaoFeng Shi <
>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>
>>>>>>>> Does the "USER_ID" column appear in other measures?
>>>>>>>>
>>>>>>>> 2017-06-23 13:57 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>>>
>>>>>>>>> It is set to this:
>>>>>>>>>
>>>>>>>>>       {
>>>>>>>>>
>>>>>>>>>         "column": "USER_ID",
>>>>>>>>>
>>>>>>>>>         "encoding": "integer:4",
>>>>>>>>>
>>>>>>>>>         "isShardBy": true
>>>>>>>>>
>>>>>>>>>       },
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Not sure why its still trying to build the dictionary.
 Keep in mind this column is a measure too (is that why it tries to create dict?).
>>>>>>>>>
>>>>>>>>> This is in logs before the exception:
>>>>>>>>>
>>>>>>>>> dict.DictionaryManager:314 : Building dictionary object
{"uuid":"4fbbfdb1-5ae3-4e0b-ae73-c071745b60d6","last_modified":0,"version":"1.6.0","source_table":"DB.USER","source_column":"USER_ID","source_column_index":0,"data_type":"bigint","input":{"path":"/kylin/kylin_metadata/kylin-30760e39-c2bb-4bc7-8239-e18e98d697e0/My_Cube_Name/fact_distinct_columns/USER_ID","size":177579241,"last_modified_time":1498183660303},"dictionary_class":null,"cardinality":0}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 22, 2017 at 10:47 PM, ShaoFeng Shi <
>>>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Seems Kylin still trying to build dictionary for
the UHC
>>>>>>>>>> dimension. Could you double check the dimension encoding
setting in the
>>>>>>>>>> "Advanced" step?
>>>>>>>>>>
>>>>>>>>>> 2017-06-23 12:54 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Step 4.  we gave 4gb to kylin server.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #4 Step Name: Build Dimension Dictionary
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Caused by: java.lang.OutOfMemoryError: Java heap
space
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at java.util.IdentityHashMap.resi
>>>>>>>>>>> ze(IdentityHashMap.java:471)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at java.util.IdentityHashMap.put(
>>>>>>>>>>> IdentityHashMap.java:440)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.TrieDict
>>>>>>>>>>> ionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:464)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.NumberDi
>>>>>>>>>>> ctionaryBuilder.build(NumberDictionaryBuilder.java:43)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>> ryGenerator$NumberDictBuilder.build(DictionaryGenerator.java
>>>>>>>>>>> :186)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>> ryGenerator.buildDictionary(DictionaryGenerator.java:81)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>> ryGenerator.buildDictionary(DictionaryGenerator.java:73)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.dict.Dictiona
>>>>>>>>>>> ryManager.buildDictionary(DictionaryManager.java:321)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.cube.CubeMana
>>>>>>>>>>> ger.buildDictionary(CubeManager.java:222)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.cube.cli.Dict
>>>>>>>>>>> ionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.jav
>>>>>>>>>>> a:50)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.cube.cli.Dict
>>>>>>>>>>> ionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.jav
>>>>>>>>>>> a:41)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.engine.mr.ste
>>>>>>>>>>> ps.CreateDictionaryJob.run(CreateDictionaryJob.java:54)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.hadoop.util.ToolRun
>>>>>>>>>>> ner.run(ToolRunner.java:76)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.hadoop.util.ToolRun
>>>>>>>>>>> ner.run(ToolRunner.java:90)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.engine.mr.com
>>>>>>>>>>> mon.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         at org.apache.kylin.job.execution
>>>>>>>>>>> .AbstractExecutable.execute(AbstractExecutable.java:113)
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 22, 2017 at 7:59 PM, ShaoFeng Shi
<
>>>>>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In which step it ran out of memory? could
you share the JSON of
>>>>>>>>>>>> the Cube definition? It can be found in the
"JSON(Cube)" tab.
>>>>>>>>>>>>
>>>>>>>>>>>> 2017-06-23 8:48 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> The column has count distinct measure
as well.  so it still
>>>>>>>>>>>>> doesn't need GD?  i tried, but appears
it ran out of memory.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jun 22, 2017 at 5:36 PM, ShaoFeng
Shi <
>>>>>>>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> For integer values, Global Dictionary
is not needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So what you do is just set "integer:4"
as the encoding in the
>>>>>>>>>>>>>> dimension, and leave blank for the
global dictionary.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-06-23 6:30 GMT+08:00 Sonny Heer
<sonnyheer@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks ShaoFeng.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> so to clarify.  for UHC dimension.
 It is integer.  So i can
>>>>>>>>>>>>>>> set encoding to integer and then
also include it in GD for count distinct?
>>>>>>>>>>>>>>>  or leave it out of GD and add
it as integer encoding only?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jun 21, 2017 at 10:55
PM, ShaoFeng Shi <
>>>>>>>>>>>>>>> shaofengshi@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sonny,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see; it is a defect: for
one column Kylin at most use 1
>>>>>>>>>>>>>>>> dictionary, it couldn't differenciate
ordinary dict and Global dict when
>>>>>>>>>>>>>>>> that column is used in both
dimension and measure.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 25million is a Ultra High
Cardinality dimension, it is not
>>>>>>>>>>>>>>>> suitable for dict as the
dict size will beyond Java heap size. In this
>>>>>>>>>>>>>>>> case, please use fixed_length
encoding; If that column is integer or long
>>>>>>>>>>>>>>>> type, you can use "integer"
encoding. In the meanwhile, keep using GD for
>>>>>>>>>>>>>>>> the count distinct measure.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2017-06-22 13:37 GMT+08:00
Sonny Heer <sonnyheer@gmail.com>
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I see what you mean @ShaoFeng
Shi.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I noticed one of the
measures I have defined is also a
>>>>>>>>>>>>>>>>> dimension.  So what can
I do in this case?  it is both needed as a count
>>>>>>>>>>>>>>>>> distinct measure and
dimension.  The typical dictionary gives java heap
>>>>>>>>>>>>>>>>> space error.  its approximately
25m unique keys.  Any ideas on how best
>>>>>>>>>>>>>>>>> kylin can handle this?
 should I remove it as GD and add as dim & fix
>>>>>>>>>>>>>>>>> length?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jun 21, 2017
at 10:33 PM, Sonny Heer <
>>>>>>>>>>>>>>>>> sonnyheer@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> No, not as a dimension.
 Only for Count distinct
>>>>>>>>>>>>>>>>>> measures.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Jun 21, 2017
at 10:25 PM, ShaoFeng Shi <
>>>>>>>>>>>>>>>>>> shaofengshi@apache.org>
wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Sonny, are
you using GlobalDictionary for a
>>>>>>>>>>>>>>>>>>> dimension? If
so, pls change to use ordinary dictionary.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The GlobalDictionary
is a "one-way" dictionary, as it
>>>>>>>>>>>>>>>>>>> can only encode
a String to an integer, it doesn't support decode the
>>>>>>>>>>>>>>>>>>> String from an
integer. The main usage for GlobalDictionary is the precise
>>>>>>>>>>>>>>>>>>> Count Distinct,
as bitmap only accepts integer as input, so Kylin use the
>>>>>>>>>>>>>>>>>>> GD to do the
conversion.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2017-06-22 6:23
GMT+08:00 Sonny Heer <
>>>>>>>>>>>>>>>>>>> sonnyheer@gmail.com>:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After finally
getting the global dictionary to work
>>>>>>>>>>>>>>>>>>>> with building
the cube there are now exceptions during query.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ERROR in
query:
>>>>>>>>>>>>>>>>>>>> "AppendTrieDictionary
can't retrive value from id"
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here is where
it ends up in the code::: ->
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     @Override
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     final
protected T getValueFromIdImpl(int id) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         throw
new UnsupportedOperationException("AppendTrieDictionary
>>>>>>>>>>>>>>>>>>>> can't retrive
value from id");
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     @Override
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     protected
byte[] getValueBytesFromIdImpl(int id) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         throw
new UnsupportedOperationException("AppendTrieDictionary
>>>>>>>>>>>>>>>>>>>> can't retrive
value from id");
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     @Override
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     protected
int getValueBytesFromIdImpl(int id, byte[]
>>>>>>>>>>>>>>>>>>>> returnValue,
int offset) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         throw
new UnsupportedOperationException("AppendTrieDictionary
>>>>>>>>>>>>>>>>>>>> can't retrive
value from id");
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>     }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Shaofeng Shi
史少锋
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sonny S. Heer
>>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354>
h: 509-884-2574
>>>>>>>>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sonny S. Heer
>>>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354>
h: 509-884-2574
>>>>>>>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sonny S. Heer
>>>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354>
h: 509-884-2574
>>>>>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sonny S. Heer
>>>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354>
h: 509-884-2574
>>>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sonny S. Heer
>>>>>>>>>>> Senior Software Engineer
>>>>>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sonny S. Heer
>>>>>>>>> Senior Software Engineer
>>>>>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574
>>>>>>>>> <(509)%20884-2574>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi 史少锋
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> Sonny S. Heer
>>>>> Senior Software Engineer
>>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Sonny S. Heer
>>>> Senior Software Engineer
>>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> Sonny S. Heer
>>> Senior Software Engineer
>>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>>
>>
>>
>>
>> --
>>
>>
>> Sonny S. Heer
>> Senior Software Engineer
>> m: 360-434-4354 <(360)%20434-4354> h: 509-884-2574 <(509)%20884-2574>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 


Sonny S. Heer
Senior Software Engineer
m: 360-434-4354 h: 509-884-2574

Mime
View raw message