avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacopo Moronato (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-1682) Support for bigdecimal type
Date Fri, 12 Jun 2015 09:54:02 GMT

     [ https://issues.apache.org/jira/browse/AVRO-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jacopo Moronato updated AVRO-1682:
----------------------------------
    Description: 
I'm using a pig script to:
1) load a semicolon-separate file
2) convert some chararray fields to bigdecimal
3) store into avro file.

Point 3 fails raising this exception:

 {{
  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
  at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
  at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
  at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
  at org.apache.pig.PigServer.compilePp(PigServer.java:1392)
  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1317)
  at org.apache.pig.PigServer.execute(PigServer.java:1309)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
  at org.apache.pig.Main.run(Main.java:478)
  at org.apache.pig.PigRunner.run(PigRunner.java:49)
  at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
  at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
  at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: java.io.IOException: Don't know how to encode type bigdecimal in schema
  
  at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceFieldSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:511)
  at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:336)
  at org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:486)
  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
  ... 48 more
}}
 



  was:
I'm using a pig script to:
1) load a semicolon-separate file
2) convert some chararray fields to bigdecimal
3) store into avro file.

Point 3 fails raising this exception:

  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
  at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
  at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
  at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
  at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
  at org.apache.pig.PigServer.compilePp(PigServer.java:1392)
  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1317)
  at org.apache.pig.PigServer.execute(PigServer.java:1309)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
  at org.apache.pig.Main.run(Main.java:478)
  at org.apache.pig.PigRunner.run(PigRunner.java:49)
  at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
  at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
  at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: java.io.IOException: Don't know how to encode type bigdecimal in schema
  
  at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceFieldSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:511)
  at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:336)
  at org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:486)
  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
  ... 48 more
 




> Support for bigdecimal type
> ---------------------------
>
>                 Key: AVRO-1682
>                 URL: https://issues.apache.org/jira/browse/AVRO-1682
>             Project: Avro
>          Issue Type: Wish
>    Affects Versions: 1.7.6
>         Environment: Apache Pig version 0.12.0-cdh5.4.2
>            Reporter: Jacopo Moronato
>            Priority: Minor
>
> I'm using a pig script to:
> 1) load a semicolon-separate file
> 2) convert some chararray fields to bigdecimal
> 3) store into avro file.
> Point 3 fails raising this exception:
>  {{
>   at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
>   at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
>   at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
>   at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
>   at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
>   at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:311)
>   at org.apache.pig.PigServer.compilePp(PigServer.java:1392)
>   at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1317)
>   at org.apache.pig.PigServer.execute(PigServer.java:1309)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
>   at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
>   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
>   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:478)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>   Caused by: java.io.IOException: Don't know how to encode type bigdecimal in schema
>   
>   at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceFieldSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:511)
>   at org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:336)
>   at org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:486)
>   at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
>   ... 48 more
> }}
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message