hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siying Dong <siyin...@fb.com>
Subject RE: Regarding HIVE-1737
Date Thu, 03 Mar 2011 09:15:28 GMT
Hi Mohit,



Can you be more precise how the fixed and variable row size are evaluated wrongly? I don't
quite understand what you mean. Did I miss any context?



I guess you are running a previous version and try to figure out whether you need to port
this patch? In that case, I think OOM is the worst possible case. We also care about whether
one task uses more resource than it really needs and competes resource with other tasks. I
don't think there can be other impact. If you want to try to repro a OOM, you should produce
a condition that sum of distinct string key size > maximum heap size, and fix size + aggregate
parameter size much smaller than average key size. You can try very long distinct string keys
as input and group by it. My feeling is that it is not such a common case, since we never
hit OOM for this.



For current trunk or version 0.7, now the codes are really not the same as when we did HIVE-1737,
since we've had HIVE-1830 now, which put a memory usage check and force to flush the disk
when memory is more than a threshold, so that even without HIVE-1737, there won't be OOM any
way.



Thanks,



Siying



________________________________
From: Mohit [mohitsikri@huawei.com]
Sent: Tuesday, March 01, 2011 7:08 AM
To: Siying Dong
Cc: Namit Jain; chinnarao@huawei.com; hive-dev@hadoop.apache.org
Subject: FW: Regarding HIVE-1737

Hi Namit/Siying,

Ok, even I agree with your analysis. Both the fixed and variable row size evaluated wrongly
here.

But what I was more interested in how critical is the change; like what if hash aggregation
map is not flushed, even if the number of existing entries overshot the false entries stats
calculated on basis of configured property hive.map.aggr.map.percentmemory (whereas if it
happens faithfully by the code changes you did, it will trigger flush), any issues apart from
out of memory in child JVM or there is more to it, something else bad can happen?

If you can provide me the pointers to re-produce it's side effect, It will be great.

-Mohit

***************************************************************************************
This e-mail and attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient's) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!
________________________________
From: Mohit [mailto:mohitsikri@huawei.com]
Sent: Tuesday, March 01, 2011 12:39 PM
To: 'siying.d@fb.com'
Subject: Regarding HIVE-1737

Hi Siying,

Hope you doing great.
Well, I have one request regarding this defect, I'm not able to understand and hence reproduce
this issue.
May be you can help in that, I need to know what queries you ran.

-Mohit

***************************************************************************************
This e-mail and attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient's) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message