hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
Date Tue, 10 Dec 2013 05:16:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843956#comment-13843956
] 

Ted Yu commented on HBASE-10000:
--------------------------------

I tried to add the following test to TestFSHDFSUtils by creating a subclass of FileSystem
which doesn't have isFileClosed:
{code}
  /**
   * Test that absence of isFileClosed doesn't make lease recovery slower.
   * @throws IOException
   */
  @Test (timeout = 30000)
  public void testIsFileClosedAbsent() throws IOException {
    // Make this time long so it is plain we broke out because of the isFileClosed invocation.
    HTU.getConfiguration().setInt("hbase.lease.recovery.dfs.timeout", 100000);
    CancelableProgressable reporter = Mockito.mock(CancelableProgressable.class);
    Mockito.when(reporter.progress()).thenReturn(true);
    FileSystemWithoutIsFileClosed dfs = Mockito.mock(FileSystemWithoutIsFileClosed.class);
    // Now make it so we fail the first two times -- the two fast invocations, then we fall
into
    // the long loop during which we will call isFileClosed.... the next invocation should
    // therefore return true if we are to break the loop.
    Mockito.when(dfs.recoverLease(FILE)).
      thenReturn(false).thenReturn(false).thenReturn(true);
    assertTrue(this.fsHDFSUtils.recoverDFSFileLease(dfs, FILE, HTU.getConfiguration(), reporter,
      HConstants.LEASE_RECOVERY_UNREQUESTED));
  }
  
  class FileSystemWithoutIsFileClosed extends FileSystem {
    public boolean recoverLease(final Path f) throws IOException {
      return true;
    }
    
    @Override
    public FSDataOutputStream create(Path f,
        FsPermission permission,
        boolean overwrite,
        int bufferSize,
        short replication,
        long blockSize,
        Progressable progress) throws IOException {
      return null;
    }

    @Override
    public FSDataOutputStream append(Path f, int bufferSize,
        Progressable progress) throws IOException {
      return null;
    }
    @Override
    public FileStatus getFileStatus(Path f) throws IOException {
      return null;
    }
    @Override
    public FSDataInputStream open(Path f, int bufferSize)
        throws IOException {
      return null;
    }
    @Override
    public boolean rename(Path src, Path dst) throws IOException {
      return true;
    }

    @Override
    public boolean delete(Path f) throws IOException {
      return true;
    }
    @Override
    public boolean delete(Path f, boolean recursive) throws IOException {
      return true;
    }
    
    @Override
    public FileStatus[] listStatus(Path f) throws FileNotFoundException, IOException {
      return null;
    }
    @Override
    public void setWorkingDirectory(Path new_dir) { }
    
    @Override
    public Path getWorkingDirectory() {
      return null;
    }
    @Override
    public boolean mkdirs(Path f, FsPermission permission) throws IOException {
      return true;
    }
    @Override
    public URI getUri() {
      return null;
    }
  }
{code}
However, it won't compile - fsHDFSUtils.recoverDFSFileLease() expects a DistributedFileSystem
parameter.
I then changed DistributedFileSystem parameter to FileSystem type - FSHDFSUtils won't compile
because we call recoverLease() directly and there is no class in between FileSystem and DistributedFileSystem
which has recoverLease() method:
{code}
      recovered = dfs.recoverLease(p);
{code}
Looks like using reflection on dfs to discover recoverLease() would be an option but I don't
know if it is acceptable.

> Initiate lease recovery for outstanding WAL files at the very beginning of recovery
> -----------------------------------------------------------------------------------
>
>                 Key: HBASE-10000
>                 URL: https://issues.apache.org/jira/browse/HBASE-10000
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>             Fix For: 0.98.1
>
>         Attachments: 10000-0.96-v5.txt, 10000-0.96-v6.txt, 10000-recover-ts-with-pb-2.txt,
10000-recover-ts-with-pb-3.txt, 10000-recover-ts-with-pb-4.txt, 10000-recover-ts-with-pb-5.txt,
10000-recover-ts-with-pb-6.txt, 10000-v4.txt, 10000-v5.txt, 10000-v6.txt
>
>
> At the beginning of recovery, master can send lease recovery requests concurrently for
outstanding WAL files using a thread pool.
> Each split worker would first check whether the WAL file it processes is closed.
> Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this idea. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message