spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <>
Subject Re: Is there a way to write spark RDD to Avro files
Date Wed, 30 Jul 2014 15:38:05 GMT
Have you checked out SchemaRDD?
There should be an examp[le of writing to Parquet files there.
BTW, FYI I was discussing this with the SparlSQL developers last week and
possibly using Apache Gora [0] for achieving this.

On Wed, Jul 30, 2014 at 5:14 AM, Fengyun RAO <> wrote:

> We used mapreduce for ETL and storing results in Avro files, which are
> loaded to hive/impala for query.
> Now we are trying to migrate to spark, but didn't find a way to write
> resulting RDD to Avro files.
> I wonder if there is a way to make it, or if not, why spark doesn't
> support Avro as well as mapreduce? Are there any plans?
> Or what's the recommended way to output spark results with schema? I don't
> think plain text is a good choice.


View raw message