asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Maxon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ASTERIXDB-1146) Cleaning up left-overs once the query is done - External datasets
Date Wed, 21 Oct 2015 23:59:27 GMT

    [ https://issues.apache.org/jira/browse/ASTERIXDB-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968204#comment-14968204
] 

Ian Maxon commented on ASTERIXDB-1146:
--------------------------------------

This feels like a silly question,  but, did you try playing with "storage.buffercache.maxopenfiles"
in the asterix-configuration.xml? We have it set to a really absurdly high value by default,
and the buffer cache is lazy and not eager about closing file handles. If it's not set to
something that will never cause the NC to break whatever the real limit is in ulimit, this
can easily happen. 

> Cleaning up left-overs once the query is done  - External datasets
> ------------------------------------------------------------------
>
>                 Key: ASTERIXDB-1146
>                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1146
>             Project: Apache AsterixDB
>          Issue Type: Bug
>          Components: AsterixDB
>            Reporter: Pouria
>            Assignee: Abdullah Alamoudi
>            Priority: Minor
>
> Running queries which use external datasets for a long time, without restarting the asterixdb
instance in between, causes the number of open files grow, and it can eventually break the
system.
> Using 'lsof' command , it seems that there are left-over 'ESTABLISHED' TCP connections
between NC and Datanode:
> java       7576    pouria  292u     IPv4          393409691       0t0        TCP asterix-10.ics.uci.edu:46965->asterix-10.ics.uci.edu:50010
(CLOSE_WAIT)
> java       7576    pouria  293u     IPv4          393412907       0t0        TCP asterix-10.ics.uci.edu:47005->asterix-10.ics.uci.edu:50010
(ESTABLISHED)
> …
> java      32205    pouria  576u     IPv4          393415126       0t0        TCP asterix-10.ics.uci.edu:50010->asterix-10.ics.uci.edu:47056
(ESTABLISHED)
> java      32205    pouria  586u     IPv4          393414645       0t0        TCP asterix-10.ics.uci.edu:50010->asterix-10.ics.uci.edu:47045
(ESTABLISHED)
> …
> Here is the error upon system breakage from CC logs:
> org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException: java.io.IOException: Too many open
files
>         at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
>         at org.apache.hyracks.control.nc.Task.run(Task.java:312)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException:
java.io.IOException: Too many open files
>         at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:358)
>         at org.apache.hyracks.control.nc.Task.run(Task.java:290)
>         ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.io.IOException:
Too many open files
>         at org.apache.hyracks.control.nc.io.IOManager.createWorkspaceFile(IOManager.java:171)
>         at org.apache.hyracks.control.nc.io.WorkspaceFileFactory.createManagedWorkspaceFile(WorkspaceFileFactory.java:39)
>         at org.apache.hyracks.control.nc.Joblet.createManagedWorkspaceFile(Joblet.java:262)
>         at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.buildWrite(OptimizedHybridHashJoin.java:332)
>         at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.spillPartition(OptimizedHybridHashJoin.java:311)
>         at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.processTuple(OptimizedHybridHashJoin.java:237)
>         at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.build(OptimizedHybridHashJoin.java:215)
>         at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoinOperatorDescriptor$PartitionAndBuildActivityNode$1.nextFrame(OptimizedHybridHashJoinOperatorDescriptor.java:313)
>         at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:342)
>         ... 4 more
> Caused by: java.io.IOException: Too many open files
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.createNewFile(File.java:1006)
>         at java.io.File.createTempFile(File.java:1989)
>         at org.apache.hyracks.control.nc.io.IOManager.createWorkspaceFile(IOManager.java:169)
>         ... 12 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message