Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
Date: Fri, 30 Sep 2016 12:27:20 +0000 (UTC)
From: "Ivan Veselovsky (JIRA)" <jira@apache.org>
To: issues@ignite.apache.org
Message-ID: <JIRA.13003880.1473433067000.707340.1475238440527@Atlassian.JIRA>
In-Reply-To: <JIRA.13003880.1473433067000@Atlassian.JIRA>
References: <JIRA.13003880.1473433067000@Atlassian.JIRA> <JIRA.13003880.1473433067108@arcas>
Subject: [jira] [Comment Edited] (IGNITE-3877) Clarify if IgfsFile ->
 FileStatus conversion should treat groupBlockSize as blockSize
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 30 Sep 2016 12:27:23 -0000


    [ https://issues.apache.org/jira/browse/IGNITE-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485008#comment-15485008 ] 

Ivan Veselovsky edited comment on IGNITE-3877 at 9/30/16 12:27 PM:
-------------------------------------------------------------------

It appears that block size in IgfsFile and in IgfsFileInfo are different for the same file (code sample below).

This happens because of  org.apache.ignite.internal.processors.igfs.IgfsImpl#create0 , where block size is simply taken from cfg.getBlockSize(), but all the other considerations are ignored. It looks like this is okay to have block size different for primary and underlying file systems (due to the group size feature), but from primary Fs viewpoint the block size should be consistent.  So, I guess, org.apache.ignite.internal.processors.igfs.IgfsImpl#create0() should be fixed: block size there should be obtained in the same way as in org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem#convert(org.apache.ignite.igfs.IgfsFile) , that is , via  groupBlockSize().

{code}
        igfs.create(file1, 256, true, null, 1, 256, null).close();
        IgfsFile f1 = igfs.info(file1);

        int blockSize = f1.blockSize();

        IgfsEntryInfo info = ((IgfsImpl)igfs).meta.infoForPath(file1);
        final int blockSize2 = info.blockSize();
        
        assert blockSize == blockSize2; // **** this fails
{code}


was (Author: iveselovskiy):
It appears that block size in IgfsFile and in IgfsFileInfo are different for the same file (code sample below).

This happens because of  org.apache.ignite.internal.processors.igfs.IgfsImpl#create0 , where block siae is simply taken from cfg.getBlockSize()  , but all the other considerations are ignored. It looks like this is okay to have block size different for primary and underlying file systems (due to the group size feature), but from promary Fs viewpoint the block size should be consistent.  So, I guess, org.apache.ignite.internal.processors.igfs.IgfsImpl#create0() should be fixed: block size there should be taken from in the same way as in org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem#convert(org.apache.ignite.igfs.IgfsFile) , that is , via  groupBlockSize().

{code}
        igfs.create(file1, 256, true, null, 1, 256, null).close();
        IgfsFile f1 = igfs.info(file1);

        int blockSize = f1.blockSize();

        IgfsEntryInfo info = ((IgfsImpl)igfs).meta.infoForPath(file1);
        final int blockSize2 = info.blockSize();
        
        assert blockSize == blockSize2; // **** this fails
{code}

> Clarify if IgfsFile -> FileStatus conversion should treat groupBlockSize as blockSize
> -------------------------------------------------------------------------------------
>
>                 Key: IGNITE-3877
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3877
>             Project: Ignite
>          Issue Type: Bug
>          Components: IGFS
>    Affects Versions: 1.6
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
>             Fix For: 1.8
>
>
> During Metrics tests repairing test org.apache.ignite.igfs.Hadoop1DualAbstractTest#testMetricsBlock revealed the following problem:
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem#convert(org.apache.ignite.igfs.IgfsFile) method treats groupBlockSize as blockSize for Hadoop FileStatus. groupBlockSize can be several times larger than blockSize, so blockSize in status gets different to that in original IgfsFile .
> changing file.groupBlockSize() to file.blockSize()  fixes problem in metrics tests, but creates problems in Hadoop tests that are bound to splits calculation, since split calculation related to blockSizes.
> Ned to 
> 1) clarify if the treatment of groupBlcokSize was intentional.
> 2) fix either metrics tests or Hadoop tests. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)