phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3228) Index table are configured with a custom/smaller MAX_FILESIZE
Date Tue, 30 Aug 2016 23:37:20 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450492#comment-15450492
] 

James Taylor commented on PHOENIX-3228:
---------------------------------------

There's also this code in MetaDataClient that sets the MAX_FILESIZE for an index, based on
the ratio of the index row size versus data row size:
{code}
            } else if (tableType == PTableType.INDEX && indexId == null) {
                if (tableProps.get(HTableDescriptor.MAX_FILESIZE) == null) {
                    int nIndexRowKeyColumns = isPK ? 1 : pkColumnsNames.size();
                    int nIndexKeyValueColumns = columns.size() - nIndexRowKeyColumns;
                    int nBaseRowKeyColumns = parent.getPKColumns().size() - (parent.getBucketNum()
== null ? 0 : 1);
                    int nBaseKeyValueColumns = parent.getColumns().size() - parent.getPKColumns().size();
                    /*
                     * Approximate ratio between index table size and data table size:
                     * More or less equal to the ratio between the number of key value columns
in each. We add one to
                     * the key value column count to take into account our empty key value.
We add 1/4 for any key
                     * value data table column that was moved into the index table row key.
                     */
                    double ratio = (1+nIndexKeyValueColumns + (nIndexRowKeyColumns - nBaseRowKeyColumns)/4d)/(1+nBaseKeyValueColumns);
                    HTableDescriptor descriptor = connection.getQueryServices().getTableDescriptor(parent.getPhysicalName().getBytes());
                    if (descriptor != null) { // Is null for connectionless
                        long maxFileSize = descriptor.getMaxFileSize();
                        if (maxFileSize == -1) { // If unset, use default
                            maxFileSize = HConstants.DEFAULT_MAX_FILE_SIZE;
                        }
                        tableProps.put(HTableDescriptor.MAX_FILESIZE, (long)(maxFileSize *
ratio));
                    }
                }
{code}



> Index table are configured with a custom/smaller MAX_FILESIZE
> -------------------------------------------------------------
>
>                 Key: PHOENIX-3228
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3228
>             Project: Phoenix
>          Issue Type: Sub-task
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>             Fix For: 4.9.0, 4.8.1
>
>         Attachments: 3228-remove.txt, 3228.txt
>
>
> I do not think we should do this. The default of 10G is chosen to keep HBase happy. For
smaller tables or initially until the index gets large it might lead to more index regions
and hence be able to utilize more region server, but generally, this is not the right thing
to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message