hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sverre Bakke (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7941) hsync() not working
Date Tue, 31 Mar 2015 14:47:52 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sverre Bakke updated HDFS-7941:
-------------------------------
    Description: 
When using SequenceFile.Writer and appending+syncing to file repeatedly, the sync does not
appear to work other than:
- once after writing headers
- when closing.

Imagine the following test case:
http://pastebin.com/Y9xysCRX

This code would append a new record every second and then immediately sync it. One would also
imagine that the file would grow for every append, however, this does not happen.

After watching the behavior I have noticed that it only syncs the headers at the very beginning
(providing a file of 164 bytes) and then never again until its closed. This despite it is
asked to hsync() after every append.

Looking into the debug logs, this also claims the same behavior (executed the provided code
example and grepped for "sync"):

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms

This was the only time the code ran fsync throughout the entire execution.

This has been tested (with similar result) for the following deployments:
- sequencefile with no compression
- sequencefile with record compression
- sequencefile with block compression
- textfile with no compression

  was:
When using SequenceFile.Writer and appending+syncing to file repeatedly, the sync does not
appear to work other than:
- once after writing headers
- when closing.

Imagine the following test case:
http://pastebin.com/Y9xysCRX

This code would append a new record every second and then immediately sync it. One would also
imagine that the file would grow for every append, however, this does not happen.

After watching the behavior I have noticed that it only syncs the headers at the very beginning
(providing a file of 164 bytes) and then never again until its closed. This despite it is
asked to hsync() after every append.

Looking into the debug logs, this also claims the same behavior (executed the provided code
example and grepped for "sync"):

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms

This was the only time the code ran fsync throughout the entire execution.


> hsync() not working
> -------------------
>
>                 Key: HDFS-7941
>                 URL: https://issues.apache.org/jira/browse/HDFS-7941
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.6.0
>         Environment: HDP 2.2 running on Redhat
>            Reporter: Sverre Bakke
>
> When using SequenceFile.Writer and appending+syncing to file repeatedly, the sync does
not appear to work other than:
> - once after writing headers
> - when closing.
> Imagine the following test case:
> http://pastebin.com/Y9xysCRX
> This code would append a new record every second and then immediately sync it. One would
also imagine that the file would grow for every append, however, this does not happen.
> After watching the behavior I have noticed that it only syncs the headers at the very
beginning (providing a file of 164 bytes) and then never again until its closed. This despite
it is asked to hsync() after every append.
> Looking into the debug logs, this also claims the same behavior (executed the provided
code example and grepped for "sync"):
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> 2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms
> This was the only time the code ran fsync throughout the entire execution.
> This has been tested (with similar result) for the following deployments:
> - sequencefile with no compression
> - sequencefile with record compression
> - sequencefile with block compression
> - textfile with no compression



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message