hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chia-Ping Tsai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
Date Wed, 28 Mar 2018 11:58:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417234#comment-16417234
] 

Chia-Ping Tsai commented on HBASE-15291:
----------------------------------------

{quote}So ideally the instantiation of srcFs will not fail and if there is any runtime exception
then anyways we will not call failedBulkload also.
{quote}
Why instantiation of srcFs won't fail?

If the instantiation of srcFs fails (throw IOE), the srcFs is null. 
{code:java}
try {
  String finalPath = path;
  if (bulkLoadListener != null) {
    finalPath = bulkLoadListener.prepareBulkLoad(familyName, path, copyFile);
  }
  Pair<Path, Path> pair = store.preBulkLoadHFile(finalPath, seqId);
  lst.add(pair);
} catch (IOException ioe) {
  // A failure here can cause an atomicity violation that we currently
  // cannot recover from since it is likely a failed HDFS operation.

  LOG.error("There was a partial failure due to IO when attempting to" +
      " load " + Bytes.toString(p.getFirst()) + " : " + p.getSecond(), ioe);
  if (bulkLoadListener != null) {
    try {
      bulkLoadListener.failedBulkLoad(familyName, path);{code}
HRegion catch the IOE, and then call failedBulkLoad. 
{code:java}
@Override
public void failedBulkLoad(final byte[] family, final String srcPath) throws IOException {
  try {
    if (!FSHDFSUtils.isSameHdfs(conf, srcFs, fs)) {  --> NPE?
      // files are copied so no need to move them back
      return;
    }{code}
FSHDFSUtils.isSameHdfs(conf, srcFs, fs) may throw NPE.

> FileSystem not closed in secure bulkLoad
> ----------------------------------------
>
>                 Key: HBASE-15291
>                 URL: https://issues.apache.org/jira/browse/HBASE-15291
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.0.2, 0.98.16.1
>            Reporter: Yong Zhang
>            Assignee: Ashish Singhi
>            Priority: Major
>         Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, HBASE-15291.002.patch,
HBASE-15291.003.patch, HBASE-15291.004.patch, HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch,
patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will cause memory
used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message