avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bob Wakefield" <adaryl.wakefi...@hotmail.com>
Subject is this an appropirate Avro use case?
Date Mon, 09 May 2016 16:14:37 GMT
I was watching a video presentation by Jay Kreps where we was talking about some data challenges
he was dealing with that he solved with Avro. The thing is, he glosses over the details.

I am in a situation where I am dealing with having to ingest CSVs. The files are picked up
by SSIS and imported into a data warehouse. My problem is the files are created by  a system
that apparently isn’t stable. The developers of the system like to add columns without warning.
What is particularly annoying is they can’t seem to decide how to represent negative numbers.
Sometimes they have a negative sign which is fine. Sometimes numbers come in accounting notation
with parentheses to denote negative numbers. That is not fine as SQL Server doesn’t understand
that as a negative.

Can I somehow use Avro to ENSURE that the file from the third party system comes as something
expected?


Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.massstreet.net
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
Mime
View raw message