hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14596) AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
Date Wed, 28 Jun 2017 13:21:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066474#comment-16066474
] 

Steve Loughran commented on HADOOP-14596:
-----------------------------------------

really hard to see from the docs of what's going on with the skip. Think maybe it should be
pulled out into something self contained

Anyway, it needs to be skip(remaining + 1) to make the message go away

with {{skip(remaining)}}

{code}
2017-06-28 12:53:00,290 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - ReadBytesReturned
Read in blocks and assess their size and duration.
 This is to identify buffering quirks. 
-------------------------------------------
2017-06-28 12:53:00,996 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3a.S3AFileSystem (S3AFileSystem.java:initialize(276))
- Magic committer is enabled
2017-06-28 12:53:02,463 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:04,043 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:05,631 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:07,203 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:08,718 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:10,286 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:11,863 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:13,386 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:14,937 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:16,489 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
...etc

 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite (Logging.scala:logInfo(54))
- 5566 blocks of size 8192;
 total #of read operations 10760;
 total read time=266,711,826,528 nS;
 5849 ns/byte

{code}

With {{skip(remaining+1)}}

{code}
2017-06-28 13:06:13,751 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - ReadBytesReturned
Read in blocks and assess their size and duration.
 This is to identify buffering quirks. 
-------------------------------------------
2017-06-28 13:06:14,451 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3a.S3AFileSystem (S3AFileSystem.java:initialize(276))
- Magic committer is enabled
2017-06-28 13:07:17,899 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - 5566 blocks of size 8192;
 total #of read operations 10810;
 total read time=63,028,617,716 nS;
 1382 ns/byte
2017-06-28 13:07:17,899 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - Read sizes
2017-06-28 13:07:17,902 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - [7] count = 8 average duration = 573 nS nS/byte = 81
2017-06-28 13:07:17,903 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - [10] count = 4 average duration = 1,959,862 nS nS/byte = 195986
2017-06-28 13:07:17,903 [ScalaTest-main-running-S3ACSVReadSuite] INFO  s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - [12] count = 2 average duration = 1,410,564 nS nS/byte = 117547
{code}

That's an interesting fact. Maybe our skip logic is itself off by one.

> AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
> --------------------------------------------------------------------
>
>                 Key: HADOOP-14596
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14596
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-14596-001.patch, testlog.txt
>
>
> The latest SDK now tells us off when we do a seek() by aborting the TCP stream
> {code}
> - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This
is likely an error and may result in sub-optimal behavior. Request only the bytes you need
via a ranged GET or drain the input stream after use.
> 2017-06-27 15:47:35,789 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
> 2017-06-27 15:47:37,409 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
> 2017-06-27 15:47:39,003 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
> 2017-06-27 15:47:40,627 [ScalaTest-main-running-S3ACSVReadSuite] WARN  internal.S3AbortableInputStream
(S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream,
aborting HTTP connection. This is likely an error and may result in sub-optimal behavior.
Request only the bytes you need via a ranged GET or drain the input stream after use.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message