hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11523) StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
Date Thu, 29 Jan 2015 19:30:34 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Duo Xu updated HADOOP-11523:
----------------------------
    Description: 
In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds,
WASB will update the parent folder's "last modified time" property. By default we do not acquire
lease on this folder when updating its property and simply pass "null" to it.

In HBase scenario, when doing distributed log splitting, there might be a case that multiple
processes from different region servers will access the same folder, and randomly we will
see this exception in regionserver's log, which makes log splitting fail.

So we should acquire the lease when updating the folder property rather than pass "null" to
it.

{code}
ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp
to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015
org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException:
There is currently a lease on the blob and no lease ID was specified in the request.
	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558)
	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569)
	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016)
	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983)
	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161)
	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
{code}

  was:
In current WASB (Windows Azure Storage - Blob) implementation, when rename operation succeeds,
WASB will update the parent folder's "last modified time" property. By default we do not acquire
lease on this folder when updating its property and simply pass "null" to it.
In HBase scenario, when doing distributed log splitting, there might be a case that multiple
processes from different region servers will access the same folder, and randomly we will
see this exception in regionserver's log, which makes log splitting fail.

{code}
ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp
to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015
org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException:
There is currently a lease on the blob and no lease ID was specified in the request.
	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558)
	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569)
	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016)
	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983)
	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161)
	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
{code}


> StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11523
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 2.2.0, 2.4.0, 2.6.0
>            Reporter: Duo Xu
>            Assignee: Duo Xu
>         Attachments: HADOOP-11523.1.patch
>
>
> In current WASB (Windows Azure Storage - Blob) implementation, when rename operation
succeeds, WASB will update the parent folder's "last modified time" property. By default we
do not acquire lease on this folder when updating its property and simply pass "null" to it.
> In HBase scenario, when doing distributed log splitting, there might be a case that multiple
processes from different region servers will access the same folder, and randomly we will
see this exception in regionserver's log, which makes log splitting fail.
> So we should acquire the lease when updating the folder property rather than pass "null"
to it.
> {code}
> ERROR org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't rename wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015.temp
to wasb://asvhbase22-2015-01-2808-29-06@hwxasvtesting.blob.core.windows.net/hbase/data/default/tdelrowtbl/3c842e8823c192d1028dc72ac3f22886/recovered.edits/0000000000000000015
> org.apache.hadoop.fs.azure.AzureException: com.microsoft.windowsazure.storage.StorageException:
There is currently a lease on the blob and no lease ID was specified in the request.
> 	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2558)
> 	at org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2569)
> 	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2016)
> 	at org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1983)
> 	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1161)
> 	at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink$2.call(HLogSplitter.java:1121)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message