asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taewoo Kim (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ASTERIXDB-1556) Hash Table used by External hash group-by doesn't conform to the budget.
Date Tue, 01 Nov 2016 02:44:58 GMT

    [ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624137#comment-15624137
] 

Taewoo Kim commented on ASTERIXDB-1556:
---------------------------------------

One more issue (NO~~~): 

[~buyingyi] told me that there was a sporadic test failure on ExternalHashGroupbyTest. And
I found the causing issue with [~javierjia].

As we know, on the current master, only the data table size (D) is controlled by the budget
(group.memory M). The hash table size (H) is out of control and can increase indefinitely.
To fix this, after a tuple is inserted, we check whether D + H > M, and if that's the case,
we continuosly deallocate the frames in buffer manager (data table) and compact the hash table
to reduce the space occupation until D + H < M. I just found that the buffer manager uses
Deallocatable frame pool class that has frame pools. If the process calls deallocation of
a frame, rather than physically deallocating a frame, this class add the frame into the pool
of frames so that the future allocation call can get a frame in the pool. And this totally
makes sense. But, for our purpose, this is an issue.

That is, H + D can eventually hit the space limit. Spilling a partition of Data table and
garbage collecting Hash Table cannot always reduce the space. That is, even after calling
(spill D + garbage collection on H), the budget is exceeded and we can't shrink it. 

One viable solution is deallocating frames physically when deallocation() is called. But it
might affect the performance. 

> Hash Table used by External hash group-by doesn't conform to the budget.
> ------------------------------------------------------------------------
>
>                 Key: ASTERIXDB-1556
>                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556
>             Project: Apache AsterixDB
>          Issue Type: Bug
>            Reporter: Taewoo Kim
>            Assignee: Taewoo Kim
>            Priority: Critical
>              Labels: soon
>         Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, 3wayjoin.pdf, 3wayjoin.rtf,
3wayjoinplan.rtf
>
>
> When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > 2),
the system generates an out-of-memory exception. 
> Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is translated
into massive number of operators (more than 200 operators in the plan for a 3-way fuzzy join),
it could generate out-of-memory exception.
> /// Update: as the discussion goes, we found that hash table in the external hash group
by doesn't conform to the frame limit. So, an out of memory exception happens during the execution
of an external hash group by operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message