hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
Date Mon, 24 Nov 2014 09:57:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222836#comment-14222836

Steve Loughran commented on HDFS-6803:

sorry, been at apachecon and neglecting that jira-list.

+1 from me


bq.  Would it be better marking those fs that do NOT support ConcurrentPositionalReads? (Probably
not – too much work).

I think ultimately we may want to make the contract XML files describing filesystems more
visible, so you can see whether an FS is case sensitive, raises EOF on out of bound seeks,
etc. But having all that information would only complicate code, wouldn't it? you don't want
to start looking for the specific features of the FS you are working with. Simpler to say
"assume it is HDFS or something which acts like it" and be done. 

I am planning to add a {{Blobstore}} tag to object stores though, to highlight they may be
way off what HDFS offers, especially in a consistency model (which would then be listed per-instance
to let callers understand what's going to be unreliable)

> Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent
> --------------------------------------------------------------------------------------------
>                 Key: HDFS-6803
>                 URL: https://issues.apache.org/jira/browse/HDFS-6803
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: 2.4.1
>            Reporter: stack
>         Attachments: 9117.md.txt, DocumentingDFSClientDFSInputStream (1).pdf, DocumentingDFSClientDFSInputStream.v2.pdf,
HDFS-6803v2.txt, HDFS-6803v3.txt, fsdatainputstream.md.v3.html
> Reviews of the patch posted the parent task suggest that we be more explicit about how
DFSIS is expected to behave when being read by contending threads. It is also suggested that
presumptions made internally be made explicit documenting expectations.
> Before we put up a patch we've made a document of assertions we'd like to make into tenets
of DFSInputSteam.  If agreement, we'll attach to this issue a patch that weaves the assumptions
into DFSIS as javadoc and class comments. 

This message was sent by Atlassian JIRA

View raw message