hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14461) Azure: handle failure gracefully in case of missing account access key
Date Thu, 01 Jun 2017 21:15:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033716#comment-16033716
] 

Mingliang Liu commented on HADOOP-14461:
----------------------------------------

[~snayak], thanks for the suggestion. I want to make it clear, what else can we do here to
handle failure in {{AzureNativeFileSystemStore::getAccountKeyFromConfiguration()}}. Its current
usage is like: if the account access key is configured, we will pick it up and connect; otherwise
we use anonymous credentials, and will complain "container not found" in {{connectUsingAnonymousCredentials()}},
exception messages like:
{code}
org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container
test in account liuml07.blob.core.windows.net not found, and we can't create it using anonymous
credentials, and no credentials found for them in the configuration.
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1027)
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:487)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1267)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3258)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3307)
{code}
Actually here the container does exist, and the problem is that the account access key was
not provided. The exception message was not clear about this. From the source code:
{code:title=connectUsingAnonymousCredentials}
    // Check for container existence, and our ability to access it.
    try {
      if (!container.exists(getInstrumentedContext())) {
        throw new AzureException("Container " + containerName + " in account "
            + accountName + " not found, and we can't create"
            + " it using anoynomous credentials, and no credentials found for them"
            + " in the configuration.");
      }
    } catch (StorageException ex) {
      throw new AzureException("Unable to access container " + containerName
          + " in account " + accountName
          + " using anonymous credentials, and no credentials found for them "
          + " in the configuration.", ex);
    }
{code}
It's not triggering the {{catch (StorageException ex)}} clause. Maybe we should change the
code here, or is there any other mechanism to probe the access ability of a counter@account?

The v1 patch added a logging message, but that's not enough at all.

> Azure: handle failure gracefully in case of missing account access key
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-14461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-14461.000.patch
>
>
> Currently if the {{fs.azure.account.key.youraccount}} is missing, we will get error stack
like this:
> {code}
> java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string.
> 	at com.microsoft.azure.storage.core.Base64.decode(Base64.java:63)
> 	at com.microsoft.azure.storage.StorageCredentialsAccountAndKey.<init>(StorageCredentialsAccountAndKey.java:81)
> 	at org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createStorageAccount(AzureBlobStorageTestAccount.java:464)
> 	at org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createTestAccount(AzureBlobStorageTestAccount.java:501)
> 	at org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:522)
> 	at org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:451)
> 	at org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization.createTestAccount(TestNativeAzureFileSystemAuthorization.java:50)
> 	at org.apache.hadoop.fs.azure.AbstractWasbTestBase.setUp(AbstractWasbTestBase.java:47)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:497)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> 	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> 	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> 	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> 	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> 	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
> Actually, this error message is not very helpful. I'd admire you if you can immediately
find the root cause of this failure.
> Currently if the test *account* is missing, we simply skip the test with meaningful error
message. Here if the *account key* is missing, we should do the same thing: skip the test,
and tell the user why we skip it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message