Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Message-ID: <13563839.435361282177396791.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 20:23:16 -0400 (EDT)
From: "Thejas M Nair (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Subject: [jira] Updated: (PIG-1466) Improve log messages for memory usage
In-Reply-To: <19322370.65931277498329898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thejas M Nair updated PIG-1466:
-------------------------------

    Attachment: PIG-1466.patch

With the changes in the patch, the logs from SpillableMemoryManager will look like this -

2010-08-18 18:22:44,365 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call- Usage threshold exceeded init = 5439488(5312K) used = 657702912(642288K) committed = 715849728(699072K) max = 715849728(699072K)
2010-08-18 18:22:44,728 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call - Collection threshold exceeded init = 5439488(5312K) used = 658437416(643005K) committed = 715849728(699072K) max = 715849728(699072K)
2010-08-18 18:27:51,174 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 73675890 bytes from 1 objects. init = 5439488(5312K) used = 511215656(499234K) committed = 659881984(644416K) max = 715849728(699072K)
2010-08-18 18:29:16,739 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 418063724 bytes from 1 objects. init = 5439488(5312K) used = 563165736(549966K) committed = 659881984(644416K) max = 715849728(699072K)
2010-08-18 18:29:22,274 INFO org.apache.pig.impl.util.SpillableMemoryManager: Spilled an estimate of 443888012 bytes from 1 objects. init = 5439488(5312K) used = 553174984(540209K) committed = 715849728(699072K) max = 715849728(699072K)


> Improve log messages for memory usage
> -------------------------------------
>
>                 Key: PIG-1466
>                 URL: https://issues.apache.org/jira/browse/PIG-1466
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Thejas M Nair
>            Priority: Minor
>             Fix For: 0.8.0
>
>         Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be cruising along at ease. We should be little more careful what to print in logs. Currently these are printed when a notification is sent by JVM and some other conditions are met which may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness. At the every least, we should lower the log level at which these are printed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.