hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
Date Thu, 23 Aug 2018 01:37:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589556#comment-16589556

Sean Mackrory commented on HADOOP-15688:

Thanks, Thomas. +1 on the patch. I didn't think to look in places other than the obvious open()
path. I haven't been tracking specific instances of timeout tests, as there have been quite
a lot of them when it does happen. I'm also suspicious that my ISP might be partly to blame
as I've been having some performance problems not just with Azure networks.

> ABFS: InputStream wrapped in FSDataInputStream twice
> ----------------------------------------------------
>                 Key: HADOOP-15688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15688
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sean Mackrory
>            Assignee: Sean Mackrory
>            Priority: Major
>         Attachments: HADOOP-15688-HADOOP-15407-002.patch, HADOOP-15688.001.patch
> I can't read Parquet files from ABFS. It has 2 different implementations to read seekable
streams, and it'll use the one that uses ByteBuffer reads if it can. It currently decides
to use the ByteBuffer read implementation because the FSDataInputStream it gets back wraps
another FSDataInputStream, which implements ByteBufferReadable.
> That's not the most robust way to check that ByteBufferReads are supported by the ultimately
underlying InputStream, but it's unnecessary and probably a mistake to double-wrap the InputStream,
so let's not.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message