Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 422C217C8F for ; Thu, 29 Jan 2015 19:30:35 +0000 (UTC) Received: (qmail 60076 invoked by uid 500); 29 Jan 2015 19:30:35 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 59951 invoked by uid 500); 29 Jan 2015 19:30:35 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 59709 invoked by uid 99); 29 Jan 2015 19:30:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jan 2015 19:30:34 +0000 Date: Thu, 29 Jan 2015 19:30:34 +0000 (UTC) From: "Duo Xu (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HADOOP-11523) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Xu updated HADOOP-11523: ---------------------------- Description: In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds, WASB will update the parent folder's "last modified time" property. By default we do not acquire lease on this folder when updating its property and simply pass "null" to it. In HBase scenario, when doing distributed log splitting, there might be a case that multiple processes from different region servers will access the same folder, and randomly we will see this exception in regionserver's log, which makes log splitting fail. So we should acquire the lease when updating the folder property rather than pass "null" to it. {code} ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015 org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558) at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569) at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016) at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} was: In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds, WASB will update the parent folder's "last modified time" property. By default we do not acquire lease on this folder when updating its property and simply pass "null" to it. In HBase scenario, when doing distributed log splitting, there might be a case that multiple processes from different region servers will access the same folder, and randomly we will see this exception in regionserver's log, which makes log splitting fail. {code} ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015 org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558) at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569) at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016) at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} > StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB > ---------------------------------------------------------------------------------------- > > Key: HADOOP-11523 > URL: https://issues.apache.org/jira/browse/HADOOP-11523 > Project: Hadoop Common > Issue Type: Bug > Components: tools > Affects Versions: 2.2.0, 2.4.0, 2.6.0 > Reporter: Duo Xu > Assignee: Duo Xu > Attachments: HADOOP-11523.1.patch > > > In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds, WASB will update the parent folder's "last modified time" property. By default we do not acquire lease on this folder when updating its property and simply pass "null" to it. > In HBase scenario, when doing distributed log splitting, there might be a case that multiple processes from different region servers will access the same folder, and randomly we will see this exception in regionserver's log, which makes log splitting fail. > So we should acquire the lease when updating the folder property rather than pass "null" to it. > {code} > ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015 > org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException: There is currently a lease on the blob and no lease ID was specified in the request. > at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558) > at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569) > at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016) > at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983) > at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161) > at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)