accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Created] (ACCUMULO-3647) Improve bulk-import sanity check with HDFS permissions enabled
Date Thu, 05 Mar 2015 18:50:39 GMT
Josh Elser created ACCUMULO-3647:

             Summary: Improve bulk-import sanity check with HDFS permissions enabled
                 Key: ACCUMULO-3647
             Project: Accumulo
          Issue Type: Improvement
          Components: tserver
            Reporter: Josh Elser
             Fix For: 1.7.0

If HDFS permissions are enabled, the ownership of the directories used to bulk import files
into Accumulo are important. For example:

  public Repo<Master> call(final long tid, final Master master) throws Exception {
    ExecutorService executor = getThreadPool(master);
    final AccumuloConfiguration conf = master.getConfiguration();
    VolumeManager fs = master.getFileSystem();
    List<FileStatus> files = new ArrayList<FileStatus>();
    for (FileStatus entry : fs.listStatus(new Path(bulk))) {
    log.debug("tid " + tid + " importing " + files.size() + " files");

    Path writable = new Path(this.errorDir, ".iswritable");
    if (!fs.createNewFile(writable)) {
      // Maybe this is a re-try... clear the flag and try again
      if (!fs.createNewFile(writable))
        throw new ThriftTableOperationException(tableId, null, TableOperation.BULK_IMPORT,
            "Unable to write to " + this.errorDir);

{{fs.createNewFile(writable)}} will fail if the Accumulo user cannot write to that directory

{noformat} Permission denied: user=accumulo, access=WRITE,

We should also be checking the permissions of the directories so that a good error message
can be returned instead of what a user would presently see:

org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
	at org.apache.thrift.TServiceClient.receiveBase(
	at org.apache.accumulo.core.master.thrift.FateService$Client.recv_waitForFateOperation(
	at org.apache.accumulo.core.master.thrift.FateService$Client.waitForFateOperation(
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.waitForFateOperation(
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.doTableFateOperation(
	at org.apache.accumulo.core.client.impl.TableOperationsImpl.importDirectory(
	at org.apache.accumulo.test.BulkImportVolumeIT.testBulkImportFailure(

This message was sent by Atlassian JIRA

View raw message