hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1466) Improve log messages for memory usage
Date Thu, 19 Aug 2010 00:25:24 GMT

     [ https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thejas M Nair updated PIG-1466:
-------------------------------

    Status: Patch Available  (was: Open)

test-patch results. No new test cases are included as this changes only the log messages.

     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified
tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of
release audit warnings.
     [exec]
     [exec]
     [exec]
     [exec]
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================


> Improve log messages for memory usage
> -------------------------------------
>
>                 Key: PIG-1466
>                 URL: https://issues.apache.org/jira/browse/PIG-1466
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Thejas M Nair
>            Priority: Minor
>             Fix For: 0.8.0
>
>         Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory
handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed = 954466304(932096K)
max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory
handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed = 954466304(932096K)
max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users tend to believe
that Pig is having hard time with memory, is spilling to disk etc. but in fact Pig might be
cruising along at ease. We should be little more careful what to print in logs. Currently
these are printed when a notification is sent by JVM and some other conditions are met which
may not necessarily indicate low memory condition. Furthermore, with {{InternalCachedBag}}
embraced everywhere in favor of {{DefaultBag}}, these messages have lost their usefulness.
At the every least, we should lower the log level at which these are printed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message