Return-Path: 
 <pig-dev-return-1710-apmail-incubator-pig-dev-archive=incubator.apache.org@incubator.apache.org>
Delivered-To: apmail-incubator-pig-dev-archive@locus.apache.org
Received: (qmail 23580 invoked from network); 25 Mar 2008 13:41:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 25 Mar 2008 13:41:32 -0000
Received: (qmail 37162 invoked by uid 500); 25 Mar 2008 13:41:31 -0000
Delivered-To: apmail-incubator-pig-dev-archive@incubator.apache.org
Received: (qmail 37082 invoked by uid 500); 25 Mar 2008 13:41:31 -0000
Mailing-List: contact pig-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:pig-dev-help@incubator.apache.org>
List-Unsubscribe: <mailto:pig-dev-unsubscribe@incubator.apache.org>
List-Post: <mailto:pig-dev@incubator.apache.org>
List-Id: <pig-dev.incubator.apache.org>
Reply-To: pig-dev@incubator.apache.org
Delivered-To: mailing list pig-dev@incubator.apache.org
Received: (qmail 37073 invoked by uid 99); 25 Mar 2008 13:41:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2008 06:41:31 -0700
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Mar 2008 13:40:49 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 912C0234C0B2
	for <pig-dev@incubator.apache.org>; Tue, 25 Mar 2008 06:39:24 -0700 (PDT)
Message-ID: <631902902.1206452364593.JavaMail.jira@brutus>
Date: Tue, 25 Mar 2008 06:39:24 -0700 (PDT)
From: "Pi Song (JIRA)" <jira@apache.org>
To: pig-dev@incubator.apache.org
Subject: [jira] Updated: (PIG-167) Experiment : A proper bag memory manager.
In-Reply-To: <1821272792.1206392004345.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/PIG-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated PIG-167:
------------------------

    Attachment:     (was: diagram.gif)

> Experiment : A proper bag memory manager.
> -----------------------------------------
>
>                 Key: PIG-167
>                 URL: https://issues.apache.org/jira/browse/PIG-167
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Pi Song
>         Attachments: MemManager0.patch
>
>
> According to PIG-164, I think we still have room for improvement:-
> 1) Alan said
> {quote}
> "It rests on the assumption that data bags generally live about the same amount of time, thus there won't be a long lived databag at the head of the list blocking the cleaning of many stale references later in the list."
> {quote}
> By looking at a line of code in SpillableMemoryManager
> {noformat}
> Collections.sort(spillables, new Comparator<WeakReference<Spillable>>() {
> {noformat}
> - Alan's assumption might be wrong after the memory manager tries to spill the list.
> - I don't understand why this has to be sorted and start spilling from the smallest bags first. Most file systems are not good at handling small files (specially ext2/ext3).
> 2) We use a linkedlist to maintain WeakReference. Normally a linkedlist consumes double as much memory that an array would consume(for pointers). Should it be better to change LinkedList to Array or ArrayList?
> 3) In SpillableMemoryManager, handleNotification which does a kind of I/O intensive job shares the same lock with registerSpillable. This doesn't seem to be efficient.
> 4) Sometimes I recognized that the bag currently in use got spilled and read back over and over again. Essentially, the memory manager should consider spilling bags currently not in use first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.