Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
Date: Fri, 24 Feb 2017 05:33:44 +0000 (UTC)
From: "ASF subversion and git services (JIRA)" <jira@apache.org>
To: notifications@asterixdb.incubator.apache.org
Message-ID: <JIRA.13044154.1487375820000.52150.1487914424257@Atlassian.JIRA>
In-Reply-To: <JIRA.13044154.1487375820000@Atlassian.JIRA>
References: <JIRA.13044154.1487375820000@Atlassian.JIRA> <JIRA.13044154.1487375820048@jira-lw-us.apache.org>
Subject: [jira] [Commented] (ASTERIXDB-1796) Job hanging after "no space on
 device"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 24 Feb 2017 05:33:50 -0000


    [ https://issues.apache.org/jira/browse/ASTERIXDB-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882005#comment-15882005 ] 

ASF subversion and git services commented on ASTERIXDB-1796:
------------------------------------------------------------

Commit 34f2384ea66a4b62627cb5a3ff3544b6d4558469 in asterixdb's branch refs/heads/master from [~buyingyi]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=34f2384 ]

ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators.

This change includes the following parts:
- Fix the implementation of fail() and close() in several runtime operators
  to avoid file handle leak and job hang;
- Add an erase method to RunFileWriter which closes files before deleting
  them in order release the holding disk space;
- Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks.
- Fix RunFileReader to not truncate files to be deleted - it is not the root
  cause of un-released disk space - open deleted files are the root cuase;
- Check file handle leaks in LangExecutionUtil.tearDown().

Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>


> Job hanging after "no space on device"
> --------------------------------------
>
>                 Key: ASTERIXDB-1796
>                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1796
>             Project: Apache AsterixDB
>          Issue Type: Bug
>          Components: Hyracks Core
>            Reporter: Yingyi Bu
>            Assignee: Yingyi Bu
>
> Setting:
> 2 NCs
> 1GB RAM per NC
> one partition per NC
> TPC-H data: 2GB
> Query:
> {noformat}
> USE tpch;
> SET `compiler.joinmemory` "8MB"
> SET `compiler.groupmemory` "8MB"
> SET `compiler.sortmemory` "8MB"
> WITH tmp AS
> (
>     SELECT l_orderkey, SUM(l_quantity) t_sum_quantity
>     FROM  LineItem
>     GROUP BY l_orderkey
> )
> SELECT c.c_name, c.c_custkey, o.o_orderkey, o.o_orderdate, o.o_totalprice,
>        SUM(l.l_quantity) sum_quantity
> FROM  LineItem l
> JOIN  tmp t ON t.l_orderkey = l.l_orderkey
> JOIN  Orders o ON o.o_orderkey = t.l_orderkey
> JOIN  Customer c ON c.c_custkey = o.o_custkey
> WHERE t.t_sum_quantity > 30
> GROUP BY c.c_name, c.c_custkey, o.o_orderkey, o.o_orderdate, o.o_totalprice
> ORDER BY o.o_totalprice DESC, o.o_orderdate
> LIMIT 100
> ;
> {noformat}
> {noformat}
> org.apache.hyracks.api.exceptions.HyracksDataException: No space left on device
> at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:376)
> at org.apache.hyracks.control.nc.Task.run(Task.java:308)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: No space left on device
> at org.apache.hyracks.control.nc.io.IOManager.syncWrite(IOManager.java:143)
> at org.apache.hyracks.dataflow.common.io.RunFileWriter.nextFrame(RunFileWriter.java:60)
> at org.apache.hyracks.dataflow.std.buffermanager.VPartitionTupleBufferManager.flushPartition(VPartitionTupleBufferManager.java:285)
> at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.probe(OptimizedHybridHashJoin.java:514)
> at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoinOperatorDescriptor$ProbeAndJoinActivityNode$1.nextFrame(OptimizedHybridHashJoinOperatorDescriptor.java:405)
> at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:360)
> ... 4 more
> Caused by: java.io.IOException: No space left on device
> at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764)
> at org.apache.hyracks.control.nc.io.IOManager.syncWrite(IOManager.java:130)
> ... 9 more
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)