Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BC836200CEF for ; Mon, 4 Sep 2017 17:37:46 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BB107163A00; Mon, 4 Sep 2017 15:37:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0F0591639FE for ; Mon, 4 Sep 2017 17:37:45 +0200 (CEST) Received: (qmail 98028 invoked by uid 500); 4 Sep 2017 15:37:44 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 98017 invoked by uid 99); 4 Sep 2017 15:37:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Sep 2017 15:37:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 1D1ACC07D5 for ; Mon, 4 Sep 2017 15:37:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 6PoCDnBADcU6 for ; Mon, 4 Sep 2017 15:37:39 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id AB63C62F30 for ; Mon, 4 Sep 2017 15:19:10 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 466CDE0EDB for ; Mon, 4 Sep 2017 15:19:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id AC41824157 for ; Mon, 4 Sep 2017 15:19:02 +0000 (UTC) Date: Mon, 4 Sep 2017 15:19:02 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 04 Sep 2017 15:37:46 -0000 [ https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13282: ------------------------------------ Target Version/s: 3.1.0 > S3 blob etags to be made visible in status/getFileChecksum() calls > ------------------------------------------------------------------ > > Key: HADOOP-13282 > URL: https://issues.apache.org/jira/browse/HADOOP-13282 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.9.0 > Reporter: Steve Loughran > Priority: Minor > > If the etags of blobs were exported via {{getFileChecksum()}}, it'd be possible to probe for a blob being in sync with a local file. Distcp could use this to decide whether to skip a file or not. > Now, there's a problem there: distcp needs source and dest filesystems to implement the same algorithm. It'd only work out the box if you were copying between S3 instances. There are also quirks with encryption and multipart: [s3 docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html]. At the very least, it's something which could be used when indexing the FS, to check for changes later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org