tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TEZ-1277) Tez Spill handler should truncate files to reserve space on disk
Date Mon, 06 Oct 2014 19:40:33 GMT

     [ https://issues.apache.org/jira/browse/TEZ-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gopal V updated TEZ-1277:
-------------------------
    Description: 
Occasionally tasks fail due to full disks because the disks had space when the task was allocating
via LocalDirAllocator, but the disk space was actually promised to many tasks instead of just
one.

This race condition shows up when a 1Gb spill can be done in ~10s or so.

There is no way to do this via the hadoop-fs abstraction - but an SSD based spill wastes most
of the IOPS on journal updates about the file length changing.

  was:
Occasionally tasks fail due to full disks because the disks had space when the task was allocating
via LocalDirAllocator, but the disk space was actually promised to many tasks instead of just
one.

This race condition shows up when a 1Gb spill can be done in ~10s or so.


> Tez Spill handler should truncate files to reserve space on disk
> ----------------------------------------------------------------
>
>                 Key: TEZ-1277
>                 URL: https://issues.apache.org/jira/browse/TEZ-1277
>             Project: Apache Tez
>          Issue Type: Improvement
>    Affects Versions: 0.5.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>
> Occasionally tasks fail due to full disks because the disks had space when the task was
allocating via LocalDirAllocator, but the disk space was actually promised to many tasks instead
of just one.
> This race condition shows up when a 1Gb spill can be done in ~10s or so.
> There is no way to do this via the hadoop-fs abstraction - but an SSD based spill wastes
most of the IOPS on journal updates about the file length changing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message