hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl Steinbach (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1364) Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
Date Tue, 22 Jun 2010 22:29:52 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881416#action_12881416
] 

Carl Steinbach commented on HIVE-1364:
--------------------------------------

bq. Also why do we make everything 4000 bytes - I presume things like ftype will never hit
that limit.

Currently the ORM is the de facto enforcement mechanism for string length limitations. I think
this is a bad approach since 1) users can work around it by manually altering the underlying
tables, and 2) the limits are stated in terms of bytes so the actual length restriction in
terms of number of characters will depend on the character set of the underlying DB. In light
of this I bumped every size limit to 4000 bytes, and also because I did not want to try to
predict which property length limit someone would next bump into. I'm willing to revert these
limits to their original values. Are there any properties besides ftype which you want me
to revert? Should I revert everything except SERDEPROPERTIES.PARAM_VALUE?

bq. Also changes to upgrade SQL should also be a part of the patch, no? Where are the scripts
for the view change located?

I'll update the patch with the necessary scripts. Should these go in bin/ or somewhere under
metastore/ ?

@John: Yes, I think this falls under the responsibility of the release manager. I will take
care of it.




I think the current approach of using the ORM as the de facto enforcement mechanism for checking


> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
>                 Key: HIVE-1364
>                 URL: https://issues.apache.org/jira/browse/HIVE-1364
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.5.0
>            Reporter: Carl Steinbach
>            Assignee: Carl Steinbach
>             Fix For: 0.6.0
>
>         Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the
length to 
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE)
to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE
when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message