hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4986) Problem in creating configration in org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils
Date Fri, 09 Aug 2013 02:59:48 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734353#comment-13734353
] 

Thejas M Nair commented on HIVE-4986:
-------------------------------------

[~subrotosanyal] This does look like a bug. Is setting core-site.xml/hdfs-site.xml in path
a suitable work around for this issue ?

                
> Problem in creating configration in org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-4986
>                 URL: https://issues.apache.org/jira/browse/HIVE-4986
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Subroto Sanyal
>
> While using AvroSerDe but, I am facing the problem:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS: hdfs://xxx:8020/abc/bbc/test.avsc, expected:
file:///
> 	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
> 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromHDFS(AvroSerdeUtils.java:105)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:70)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
>         at test.subroto.HiveUtil.getDeserializer(HiveUtil.java:102)
> {noformat}
> While debugging the problem I saw a piece of code in:
> {code:title=org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.java|borderStyle=solid}
> public static Schema More ...determineSchemaOrThrowException(Properties properties)
>            throws IOException, AvroSerdeException {
>      String schemaString = properties.getProperty(SCHEMA_LITERAL);
>      if(schemaString != null && !schemaString.equals(SCHEMA_NONE))
>        return Schema.parse(schemaString);
>  
>      // Try pulling directly from URL
>      schemaString = properties.getProperty(SCHEMA_URL);
>      if(schemaString == null || schemaString.equals(SCHEMA_NONE))
>        throw new AvroSerdeException(EXCEPTION_MESSAGE);
>  
>      try {
>        if(schemaString.toLowerCase().startsWith("hdfs://"))
>          return getSchemaFromHDFS(schemaString, new Configuration());
>      } catch(IOException ioe) {
>        throw new AvroSerdeException("Unable to read schema from HDFS: " + schemaString,
ioe);
>      }
>  
>      return Schema.parse(new URL(schemaString).openStream());
>    }
> {code} 
> Here the configuration passed from the Deserilaizer is completely ignored and a new Configuration
object is created. 
> In my opinion the AvroSerDe should honor the Configuration being passed to Deserializer
interface ; rather than creating a new one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message