hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Klimontovich (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-2280) HFileOutputFormat writes output to "unsafe" directory
Date Wed, 03 Mar 2010 11:06:27 GMT
HFileOutputFormat writes output to "unsafe" directory

                 Key: HBASE-2280
                 URL: https://issues.apache.org/jira/browse/HBASE-2280
             Project: Hadoop HBase
          Issue Type: Bug
    Affects Versions: 0.20.3
            Reporter: Vladimir Klimontovich
            Priority: Critical
             Fix For: 0.20.5, 0.21.0, 0.22.0

HFileOutputFormat writes data direct to output folder. It's incorrect as failed (or killed,
or interrupted) reducers leaves inconsistent files in output folder.

The convinient way to ouput data from OutputFormat is to use "working directory". The content
of this directory is being moved to output directory at the end of reducer process if only
reducer succeeded (this process is called "output commit" or "atomic commit").

If means that instead of

 final Path outputdir = FileOutputFormat.getOutputPath(context);

hbase should use

 final Path outputdir = FileOutputFormat.getWorkOutputPath(context);

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message