drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefán Baxter <ste...@activitystream.com>
Subject Re: Continued Avro Frustration
Date Fri, 01 Apr 2016 18:33:19 GMT
Anybody wishing to verify this is welcome to search this email archives for
my email address/name and find the countless of help request regarding
contribution.

The Lucene plugin is an example where I tried to contribute but
pointer/documentation requests were unanswered and pull requests were
ignored.

The same is tru for the pull request I made to improve schema changes
errors to make them partially useful.

*Only a project leader that makes sure new contributes can really
contribute can play the "ungrateful you, this is a OS project" and you,
Jacques, have not demonstrated that to any real extent.*

Look forward to discuss this in the upcoming hang-out.

- Stefán

On Fri, Apr 1, 2016 at 6:27 PM, Stefán Baxter <stefan@activitystream.com>
wrote:

> Jacques, You are way out of line here!
>
> Let me remind you that I have volunteered resources, mine and other, on
> some of these and asked for pointers several times, to NO AVAIL!
>
> It's not as if the Drill project is easy to wrap ones head around or test
> (this latter has improved a bit).
>
> I have also contribute some UDF and made available so you don't get to
> play this card.
>
> We have also invested hundreds of hours trying to use the Avro plugin,
> hours that I wish we had to do something more productive.
>
>
> - Stefán
>
>
>
> On Fri, Apr 1, 2016 at 6:10 PM, Jacques Nadeau <jacques@dremio.com> wrote:
>
>> Stefan,
>>
>> It makes sense to me to mark the Avro plugin experimental. Clearly, there
>> are bugs. I also want to note your requirements and expectations haven't
>> always been in alignment with what the Avro plugin developers
>> built/envisioned (especially around schemas). As part of trying to address
>> these gaps, I'd like to ask again for you to provide actual data and tests
>> cases so we make sure that the Avro plugin includes those as future test
>> cases. (This is absolutely the best way to ensure that the project
>> continues to work for your use case.)
>>
>> The bigger issue I see here is that you expect the community to spend time
>> doing what you want. You have already received a lot of that via free
>> support and numerous bug fixes by myself, Jason and others. You need to
>> remember: this community is run by a bunch of volunteers. Everybody here
>> has a day job. A lot of time I spend in the community is at the cost of my
>> personal life. For others, it is the same.
>>
>> This is a good place to ask for help but you should never demand it. If
>> you
>> want paid support, I know Ted offered this from MapR and I'm sure if you
>> went that route, your issues would get addressed very quickly. If you
>> don't
>> want to go that route, then I suggest that you help by creating more
>> example data and test cases and focusing on what are the most important
>> issues that you need to solve. From there, you can continue to expect that
>> people will help you--as they can. There are no guarantees in open source.
>> Everything comes through the kindness and shared goals of those in the
>> community.
>>
>> thanks,
>> Jacques
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Fri, Apr 1, 2016 at 5:43 AM, Stefán Baxter <stefan@activitystream.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Is it at all possible that we are the only company trying to use Avro
>> with
>> > Drill to some serious extent?
>> >
>> > We continue to coma across all sorts of embarrassing shortcomings like
>> the
>> > one we are dealing with now where a schema change exception is thrown
>> even
>> > when working with a single Avro file (that has the same schema).
>> >
>> > Can a non project member call for a discussion on this topic and the
>> level
>> > of support that is offered for Avro in Drill?
>> >
>> > My discussion topics would be:
>> >
>> >    - Strange schema validation that ... :
>> >    ... currently fails on single file
>> >    ... prevents dirX variables to work
>> >    ... would require Drill to scan all Avro files to establish schema
>> (even
>> >    when pruning would be used)
>> >    ... would ALWAY fail for old queries if the an old Avro file,
>> containing
>> >    the original fields, was removed and could not be scanned
>> >    ... does not rhyme with the "eliminate ETL" and "Evolving Schema"
>> goals
>> >    of Drill
>> >
>> >    - Simple union types do not work to declare nullable fields
>> >
>> >    - Drill can not read Parquet that is created by parquet-mr-avro
>> >
>> >    - What is the intention for Avro in Drill
>> >    - Should we select to use some other format to buffer/badge data
>> before
>> >    creating a Parquet file for it?
>> >
>> >    - The culture here regarding talking about boring/hard topics like
>> this
>> >    - Where serious complaints/issues are met with silence
>> >    - I know full well that my frustration shines through here and that
>> it
>> >    not helping but this Drill+Avro mess is really getting too much for
>> us
>> > to
>> >    handle
>> >
>> > Look forward do discuss this here or during the next hangout.
>> >
>> > Regards,
>> >  -Stefán (or ... mr. old & frustrated)
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message