incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gagan Deep Juneja (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BLUR-107) Create checkOutputSpecs check in the BlurOutputFormat
Date Sat, 25 May 2013 04:13:20 GMT

    [ https://issues.apache.org/jira/browse/BLUR-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666958#comment-13666958
] 

Gagan Deep Juneja commented on BLUR-107:
----------------------------------------

I tried below implementation for validation 

public void checkOutputSpecs(JobContext context) throws IOException, InterruptedException
{
    Configuration config = context.getConfiguration();
    TableDescriptor tableDescriptor = getTableDescriptor(config);
    if (tableDescriptor == null) {
      throw new IOException("setTableDescriptor needs to be called first.");
    }
    int shardCountByTableDescriptor = tableDescriptor.getShardCount();
    Path outputPath = getOutputPath(config);
    FileSystem fileSystem = outputPath.getFileSystem(config);
    int shardCountOnFileSystem = 0;
    for (FileStatus fileStatus : fileSystem.listStatus(outputPath)) {
      if (fileStatus.getPath().getName().startsWith(BlurConstants.SHARD_PREFIX)) {
        shardCountOnFileSystem++;
      }
    }
    if( shardCountOnFileSystem != shardCountByTableDescriptor){
      throw new IOException("Actual shards on file system " + "[ " + shardCountOnFileSystem
+ " ]" + " differs from the value set in Table Descriptor " + "[ " +shardCountByTableDescriptor
+" ]");
    }
    int reducers = context.getNumReduceTasks();
    if(reducers < 0 ){
      
    }
  }

What is the definition of valid integer in case of number of reducers? Is there any formula
for that?

Regards,
Gagan
                
> Create checkOutputSpecs check in the BlurOutputFormat
> -----------------------------------------------------
>
>                 Key: BLUR-107
>                 URL: https://issues.apache.org/jira/browse/BLUR-107
>             Project: Apache Blur
>          Issue Type: Bug
>    Affects Versions: 0.1.5
>            Reporter: Aaron McCurry
>             Fix For: 0.1.5
>
>
> Using the checkOutputSpecs method on the BlurOutputFormat validate that the number of
reducers is valid number.  Also check that the Blur table exists and that the number of shards
in the FS match the table descriptor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message