hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-134) job conf being passed to serde (initialize)
Date Mon, 08 Dec 2008 22:37:44 GMT

    [ https://issues.apache.org/jira/browse/HIVE-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654615#action_12654615

Joydeep Sen Sarma commented on HIVE-134:

of course it makes it harder. it's one more interface that you are forced to implement for
full functionality.

the other problem you are not considering is if the SerDe invokes some other class that requires
a Conf to bootstrap (see below for why). Now the SerDe author has to know what options they
might be interested in and what options whatever code they are invoking might be interested
in. This is a non-starter.

here's some background: almost all hadoop classes implement Configurable - which means they
get a Configuration object passed in at initialization time. If someone asked me what made
it possible to implement Hive on Hadoop easily - this would be one of the single biggest reasons.
Entire Hive execution system is bootstrapped by configuring all manners of parameters inside
Hadoop's Conf with minimal overhead from our side.

also - the goal you have is nice (create a registry of options to control serde behavior -
possibly with metadata about semantics/values). so yes - we can have additional interface.
But we should not FORCE people to provide this metadata and implement this interface for correct
functioning (and take away the flexibility to have parameters that are not declared via the
new interface). does that make sense?

> job conf being passed to serde (initialize)
> -------------------------------------------
>                 Key: HIVE-134
>                 URL: https://issues.apache.org/jira/browse/HIVE-134
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Namit Jain
> Currently, job conf is being passed to serde - serde should not need job conf. Anyway,
properties is being passed around which can contain any name/value pairs.
> Once jira-127 is committed, this should be done in a follow-up

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message