accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (ACCUMULO-3215) Import tries to use default DFS directory instead of configured
Date Thu, 09 Oct 2014 19:15:34 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165577#comment-14165577
] 

Josh Elser edited comment on ACCUMULO-3215 at 10/9/14 7:15 PM:
---------------------------------------------------------------

I can reproduce this trivially in an IT and in the shell. On a standalone instance, I configured
accumulo-site with

{noformat}
  <property>
    <name>instance.volumes</name>
    <value>hdfs://localhost:8020/accumulo1.6.2</value>
  </property>
{noformat}

But in core-site.xml

{noformat}
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://127.0.0.1:8020</value>
    </property>
{noformat}

Ultimately, in the fate op for import table, we tried to find the matching filesystem for
127.0.0.1:8020 out of choices: localhost:8020. This then threw an NPE because there are no
matching options. This is actually a really nasty issue...

We could try to make things a little better by converting any URIs with IP addrs as the hostname
component to resolved the resolved hostname via DNS, but we should already know that relying
on rDNS to be properly configured is a shot in the dark at best.

Edit: this is actually an entirely different issue. opening a separate issue for it.


was (Author: elserj):
I can reproduce this trivially in an IT and in the shell. On a standalone instance, I configured
accumulo-site with

{noformat}
  <property>
    <name>instance.volumes</name>
    <value>hdfs://localhost:8020/accumulo1.6.2</value>
  </property>
{noformat}

But in core-site.xml

{noformat}
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://127.0.0.1:8020</value>
    </property>
{noformat}

Ultimately, in the fate op for import table, we tried to find the matching filesystem for
127.0.0.1:8020 out of choices: localhost:8020. This then threw an NPE because there are no
matching options. This is actually a really nasty issue...

We could try to make things a little better by converting any URIs with IP addrs as the hostname
component to resolved the resolved hostname via DNS, but we should already know that relying
on rDNS to be properly configured is a shot in the dark at best.

> Import tries to use default DFS directory instead of configured
> ---------------------------------------------------------------
>
>                 Key: ACCUMULO-3215
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3215
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.6.1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Critical
>             Fix For: 1.6.2, 1.7.0
>
>
> Ran Shard randomwalk and saw a failure about a bulk imported file. Oddly, the path that
it referenced was for the default dfs directory /accumulo instead of the configured volume
I set /apps/accumulo.
> Additionally, bulk import is placing relative paths into metadata and not absolute paths,
which also seems rather strange.
> {noformat}
> 2014-10-08 12:13:56,230 [tserver.Tablet] DEBUG: Looking at metadata {2s< file:/b-00004ee/I00004ef.rf
[] 33312 false=269198,66282, 2s< file:/b-00004ee/I00004eg.rf [] 33312 false=251133,75656,
..., 2s< file:/b-00004ee/I00004f0.rf [] 33312 false=281377,54556, ..., 2s< future:148f0574f320007
[] 33313 false=node3.example.com:9997, 2s< srv:dir [] 33312 false=/c-00000000, 2s< srv:time
[] 33312 false=M1412795627612, 2s< ~tab:~pr [] 33312 false=^@}
> {noformat}
> {noformat}
> 2014-10-08 12:13:56,658 [fs.VolumeManagerImpl] DEBUG: Found candidate Volumes for Path
but none of the Paths are valid on the candidates: hdfs://node1.example.com:8020/accumulo/tables/2s/b-00004ee/I00004f0.rf
> 2014-10-08 12:13:56,658 [fs.VolumeManagerImpl] DEBUG: Could not determine volume for
Path: hdfs://node1.example.com:8020/accumulo/tables/2s/b-00004ee/I00004f0.rf
> 2014-10-08 12:13:56,673 [tserver.TabletServer] DEBUG: MultiScanSess 240.0.0.10:36725
22 entries in 0.05 secs (lookup_time:0.05 secs tablets:1 ranges:1)
> 2014-10-08 12:13:56,678 [tserver.FileManager] ERROR: Failed to open file hdfs://node1.example.com:8020/accumulo/tables/2s/b-00004ee/I00004f0.rf
File does not exist: /accumulo/tables/2s/b-00004ee/I00004f0.rf
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1876)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1818)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1798)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1770)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> 2014-10-08 12:13:56,679 [tserver.TabletServer] WARN : exception while scanning tablet
2s<<
> java.io.IOException: Failed to open hdfs://node1.example.com:8020/accumulo/tables/2s/b-00004ee/I00004f0.rf
>         at org.apache.accumulo.tserver.FileManager.reserveReaders(FileManager.java:334)
>         at org.apache.accumulo.tserver.FileManager.access$500(FileManager.java:59)
>         at org.apache.accumulo.tserver.FileManager$ScanFileManager.openFiles(FileManager.java:491)
>         at org.apache.accumulo.tserver.FileManager$ScanFileManager.openFileRefs(FileManager.java:479)
>         at org.apache.accumulo.tserver.FileManager$ScanFileManager.openFiles(FileManager.java:499)
>         at org.apache.accumulo.tserver.Tablet$ScanDataSource.createIterator(Tablet.java:2032)
>         at org.apache.accumulo.tserver.Tablet$ScanDataSource.iterator(Tablet.java:1994)
>         at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:165)
>         at org.apache.accumulo.tserver.Tablet.nextBatch(Tablet.java:1709)
>         at org.apache.accumulo.tserver.Tablet.access$3200(Tablet.java:177)
>         at org.apache.accumulo.tserver.Tablet$Scanner.read(Tablet.java:1851)
>         at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler$NextBatchTask.run(TabletServer.java:1049)
>         at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>         at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist: /accumulo/tables/2s/b-00004ee/I00004f0.rf
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1876)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1818)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1798)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1770)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
>         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1192)
>         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1180)
>         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1170)
>         at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
>         at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
>         at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:230)
>         at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1430)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:261)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.access$100(CachableBlockFile.java:144)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$MetaBlockLoader.get(CachableBlockFile.java:216)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBlock(CachableBlockFile.java:318)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:372)
>         at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:144)
>         at org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:825)
>         at org.apache.accumulo.core.file.rfile.RFileOperations.openReader(RFileOperations.java:79)
>         at org.apache.accumulo.core.file.DispatchingFileFactory.openReader(FileOperations.java:119)
>         at org.apache.accumulo.tserver.FileManager.reserveReaders(FileManager.java:315)
>         ... 17 more
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
does not exist: /accumulo/tables/2s/b-00004ee/I00004f0.rf
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1876)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1818)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1798)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1770)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:362)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>         at com.sun.proxy.$Proxy19.getBlockLocations(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:253)
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1190)
>         ... 38 more
> 2014-10-08 12:13:56,684 [problems.ProblemReports] DEBUG: Filing problem report 2s FILE_READ
hdfs://node1.example.com:8020/accumulo/tables/2s/b-00004ee/I00004f0.rf
> {noformat}
> {{/accumulo/tables/2s/b-00004ee/I00004f0.rf}} is an invalid path on my system given the
configuration
> {noformat}
>   <property>
>     <name>instance.volumes</name>
>     <value>hdfs://node1.example.com:8020/apps/accumulo</value>
>   </property>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message