hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Yongqiang (JIRA)" <>
Subject [jira] Commented: (HIVE-477) Some optimization thoughts for Hive
Date Wed, 13 May 2009 06:54:45 GMT


He Yongqiang commented on HIVE-477:

One Comment for 1):
Avoiding byte copy when init LazyString seems will not save CPU time. 
In my test, i use two tables of 30 1K columns, and insert one from the other. The table's
size is about 140M.
Two tests, one with byte copy and the other without byte copy, cost the same time.

So it seems java's array copy time can be ignored.

> Some optimization thoughts for Hive
> -----------------------------------
>                 Key: HIVE-477
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: Improvement
>            Reporter: He Yongqiang
> Before we can start working on Hive-461. I am doing some profiling for hive. And here
are some thoughts for improvements:
> minor :
> 1) add a new HiveText to replace Text. It can avoid byte copy when init LazyString. I
have done a draft one, it shows  ~1% performance gains.
> 2) let StructObjectInspector's 
>     {noformat}
>      public List<Object> getStructFieldsDataAsList(Object data);
>     {noformat}
> to be 
>     {noformat}
>      public Object[] getStructFieldsDataAsArray(Object data);
>     {noformat}
> In my profiling test, it shows some performace gains. but in acutal execution it did
not. Anyway, let it return java array will reduce gc's burden of collection ArrayList
> not so minor:
> 3) split FileSinkOperator's Writer into another Thread. Adding a producer-consumer array
as the bridge between the Operators thread and the Writer thread.
> 4) the operator stack is kind of deep. In order to avoid instruction cache misses, and
increase the efficiency data cache, I suggest to let Hive's operator can process an array
of rows instead of processing only one row at a time.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message