hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <>
Subject Re: HIVE mappers eat a lot of RAM
Date Tue, 20 Mar 2012 11:22:00 GMT
Hi Alex
      In good clusters you have the child task JVM size as 1.5 or  2GB (or at least 1G).
IMHO, 500MB for a task is a pretty normal memory consumption.
Now for 50G of data you are having just 7 mappers, need to increase the number of mappers
for better parallelism.


 From: Alexander Ershov <>
Sent: Tuesday, March 20, 2012 4:13 PM
Subject: HIVE mappers eat a lot of RAM


I'm using HIVE 0.7.1 with
1) moderate 50GB table, let's call it `temp_view`
2) query: select max(length(get_json_object(json, '$.user_id'))) from temp_view. From my point
of view this query is a total joke, nothing serious.

Query runs just fine, everyone's happy.

But I have massive memory consumption at the map phase:  7 active mappers eating 500 Mb of
RAM each.

This is a really bad stuff, it means real mappers on real queries will throw OutOfMemory exception
(they do throw it actually).

Anyone has any ideas of what I'm doing wrong? Cause I have zero.
View raw message