hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Ershov <vohs...@gmail.com>
Subject Re: HIVE mappers eat a lot of RAM
Date Tue, 20 Mar 2012 12:19:43 GMT
I said it wrong: what really bothers me is not 500MB of RAM usage - it's
that mapper starting as 70-200Mb happy chimp becomes 500MB-600MB
bad-smelling gorilla. And that's on a simplest query! As far as I
understand Hive source code UDF length and UDAF max are super careful with
memory allocations. Same with get_json_object. And it's Java, it has modest
gc capabilites.

The question is: Is increasing RAM consumption an unavoidable feature of
Hive? Or I somehow has fouled up Java or Hive configuration? Not-Hive
Hadoop jobs work fine using constant amount of memory.

Thanks for you support.

Actually I have total of about 180 mappers. I meant 7 mappers per node.

2012/3/20 Bejoy Ks <bejoy_ks@yahoo.com>

> Hi Alex
>       In good clusters you have the child task JVM size as 1.5 or  2GB
> (or at least 1G). IMHO, 500MB for a task is a pretty normal
> memory consumption.
> Now for 50G of data you are having just 7 mappers, need to increase the
> number of mappers for better parallelism.
>
> Regards
> Bejoy
>
>   ------------------------------
> *From:* Alexander Ershov <vohsrea@gmail.com>
> *To:* user@hive.apache.org
> *Sent:* Tuesday, March 20, 2012 4:13 PM
> *Subject:* HIVE mappers eat a lot of RAM
>
> Hiya,
>
> I'm using HIVE 0.7.1 with
> 1) moderate 50GB table, let's call it `temp_view`
> 2) query: select max(length(get_json_object(json, '$.user_id'))) from
> temp_view. From my point of view this query is a total joke, nothing
> serious.
>
> Query runs just fine, everyone's happy.
>
> But I have massive memory consumption at the map phase: 7 active mappers
> eating 500 Mb of RAM each.
>
> This is a really bad stuff, it means real mappers on real queries will
> throw OutOfMemory exception (they do throw it actually).
>
> Anyone has any ideas of what I'm doing wrong? Cause I have zero.
>
>
>

Mime
View raw message