avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: is this an appropirate Avro use case?
Date Mon, 09 May 2016 17:21:45 GMT
you can use avro to handle columns being added without warning. you can
also use avro to handle column renames, etc.

you cannot use avro to ensure the data comes in the format you expect (the
negative numbers issue). you will have to parse these variations before
converting to avro.

On Mon, May 9, 2016 at 12:14 PM, Bob Wakefield <adaryl.wakefield@hotmail.com
> wrote:

> I was watching a video presentation by Jay Kreps where we was talking
> about some data challenges he was dealing with that he solved with Avro.
> The thing is, he glosses over the details.
> I am in a situation where I am dealing with having to ingest CSVs. The
> files are picked up by SSIS and imported into a data warehouse. My problem
> is the files are created by  a system that apparently isn’t stable. The
> developers of the system like to add columns without warning. What is
> particularly annoying is they can’t seem to decide how to represent
> negative numbers. Sometimes they have a negative sign which is fine.
> Sometimes numbers come in accounting notation with parentheses to denote
> negative numbers. That is not fine as SQL Server doesn’t understand that as
> a negative.
> Can I somehow use Avro to ENSURE that the file from the third party system
> comes as something expected?
> Adaryl "Bob" Wakefield, MBA
> Principal
> Mass Street Analytics, LLC
> 913.938.6685
> www.massstreet.net
> www.linkedin.com/in/bobwakefieldmba
> Twitter: @BobLovesData

View raw message