hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chia-Ping Tsai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
Date Wed, 28 Mar 2018 11:58:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417234#comment-16417234

Chia-Ping Tsai commented on HBASE-15291:

{quote}So ideally the instantiation of srcFs will not fail and if there is any runtime exception
then anyways we will not call failedBulkload also.
Why instantiation of srcFs won't fail?

If the instantiation of srcFs fails (throw IOE), the srcFs is null. 
try {
  String finalPath = path;
  if (bulkLoadListener != null) {
    finalPath = bulkLoadListener.prepareBulkLoad(familyName, path, copyFile);
  Pair<Path, Path> pair = store.preBulkLoadHFile(finalPath, seqId);
} catch (IOException ioe) {
  // A failure here can cause an atomicity violation that we currently
  // cannot recover from since it is likely a failed HDFS operation.

  LOG.error("There was a partial failure due to IO when attempting to" +
      " load " + Bytes.toString(p.getFirst()) + " : " + p.getSecond(), ioe);
  if (bulkLoadListener != null) {
    try {
      bulkLoadListener.failedBulkLoad(familyName, path);{code}
HRegion catch the IOE, and then call failedBulkLoad. 
public void failedBulkLoad(final byte[] family, final String srcPath) throws IOException {
  try {
    if (!FSHDFSUtils.isSameHdfs(conf, srcFs, fs)) {  --> NPE?
      // files are copied so no need to move them back
FSHDFSUtils.isSameHdfs(conf, srcFs, fs) may throw NPE.

> FileSystem not closed in secure bulkLoad
> ----------------------------------------
>                 Key: HBASE-15291
>                 URL: https://issues.apache.org/jira/browse/HBASE-15291
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.0.2,
>            Reporter: Yong Zhang
>            Assignee: Ashish Singhi
>            Priority: Major
>         Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, HBASE-15291.002.patch,
HBASE-15291.003.patch, HBASE-15291.004.patch, HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch,
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will cause memory
used more and more if too many bulkLoad .

This message was sent by Atlassian JIRA

View raw message