hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
Date Fri, 15 Jan 2016 18:15:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102230#comment-15102230

Chris Nauroth commented on HADOOP-12635:

It turns out I misinterpreted something in the design document.  There is a statement about
supporting "a single writer".  I interpreted this to mean that the goal of the feature was
to match HDFS single-writer enforcement semantics exactly.  Instead, the goal is to support
append for applications that can guarantee a single writer themselves.  WASB will not provide
this enforcement internally.  It becomes a responsibility of the application either to ensure
single-threaded handling for a particular file path, or rely on some external locking mechanism
of its own.

Exactly matching HDFS single-writer semantics might be a nice future goal, but this would
be a huge undertaking.  It would nearly be a full rewrite with reconsideration of locking
strategies inside the entire API footprint.  It also will be challenging to achieve this without
harming current performance.

Based on this, I think we can proceed with a patch similar to what has been provided already,
pending resolution of my code review feedback.  Usage of the API will carry some risk, so
I propose that we guard it behind a configuration flag, which would reject append calls with
an error unless explicitly enabled.  Let's also call out the risks in documentation and make
it clear to end users that this differs from HDFS in that single-writer enforcement is an
application responsibility with WASB.

> Adding Append API support for WASB
> ----------------------------------
>                 Key: HADOOP-12635
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12635
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: azure
>    Affects Versions: 2.7.1
>            Reporter: Dushyanth
>            Assignee: Dushyanth
>         Attachments: Append API.docx, HADOOP-12635-004.patch, HADOOP-12635.001.patch,
HADOOP-12635.002.patch, HADOOP-12635.003.patch
> Currently the WASB implementation of the HDFS interface does not support Append API.
This JIRA is added to design and implement the Append API support to WASB. The intended support
for Append would only support a single writer.  

This message was sent by Atlassian JIRA

View raw message