drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefán Baxter <ste...@activitystream.com>
Subject Re: Continued Avro Frustration
Date Sat, 02 Apr 2016 06:11:32 GMT
Julian, you can give this speech to someone that has not been trying to
contribute.

Contributions come in more forms than fixing bugs or building features,
especially when dealing with complex project like Drill.

My contribution have been:
 - UDF samples (MIT licenesed and available to all)
 - Lucene plugin (which now seems to be abandoned)
 - Testing (and quite a few submitted tickets)
 - Minor improvement for Drill (Ignored pull request)

Now on towards more constructive things.




On Fri, Apr 1, 2016 at 11:16 PM, Julian Hyde <jhyde@apache.org> wrote:

> Stefan,
>
> I wanted to chime in. I don’t think Jacques was out of line.
>
> I understand your frustration, but the project does not owe you anything.
> The only surefire way to get a feature into the project is to contribute it
> yourself AND to work through the process of getting it accepted.
>
> If you contribute a feature I know the committers will make their best
> effort to accept it into the project, but they have other commitments on
> their time.
>
> Also, please remember that Drill is an Apache project. It is not MapR, nor
> any other company, even though some of the contributors on Drill may be
> employees of MapR. If you buy support from MapR that is a contract between
> you and MapR, not with the project. (To be clear, MapR has never made such
> claims, in my experience, and is always respectful of its proper
> relationship with Apache projects.)
>
> Julian
>
>
> > On Apr 1, 2016, at 11:43 AM, Stefán Baxter <stefan@activitystream.com>
> wrote:
> >
> > To give credit where it is due.
> >
> > MapR seems to be willing to provide us with paid support even though we
> are
> > not using their platform and we welcome that but we are still waiting
> for a
> > followup email from the people Ted sent an introduction to.
> >
> > If that is a misunderstanding on my part, and they are waiting for me,
> then
> > I will correct that ASAP.
> >
> > - Stefán
> >
> > On Fri, Apr 1, 2016 at 6:10 PM, Jacques Nadeau <jacques@dremio.com>
> wrote:
> >
> >> Stefan,
> >>
> >> It makes sense to me to mark the Avro plugin experimental. Clearly,
> there
> >> are bugs. I also want to note your requirements and expectations haven't
> >> always been in alignment with what the Avro plugin developers
> >> built/envisioned (especially around schemas). As part of trying to
> address
> >> these gaps, I'd like to ask again for you to provide actual data and
> tests
> >> cases so we make sure that the Avro plugin includes those as future test
> >> cases. (This is absolutely the best way to ensure that the project
> >> continues to work for your use case.)
> >>
> >> The bigger issue I see here is that you expect the community to spend
> time
> >> doing what you want. You have already received a lot of that via free
> >> support and numerous bug fixes by myself, Jason and others. You need to
> >> remember: this community is run by a bunch of volunteers. Everybody here
> >> has a day job. A lot of time I spend in the community is at the cost of
> my
> >> personal life. For others, it is the same.
> >>
> >> This is a good place to ask for help but you should never demand it. If
> you
> >> want paid support, I know Ted offered this from MapR and I'm sure if you
> >> went that route, your issues would get addressed very quickly. If you
> don't
> >> want to go that route, then I suggest that you help by creating more
> >> example data and test cases and focusing on what are the most important
> >> issues that you need to solve. From there, you can continue to expect
> that
> >> people will help you--as they can. There are no guarantees in open
> source.
> >> Everything comes through the kindness and shared goals of those in the
> >> community.
> >>
> >> thanks,
> >> Jacques
> >>
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Fri, Apr 1, 2016 at 5:43 AM, Stefán Baxter <
> stefan@activitystream.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Is it at all possible that we are the only company trying to use Avro
> >> with
> >>> Drill to some serious extent?
> >>>
> >>> We continue to coma across all sorts of embarrassing shortcomings like
> >> the
> >>> one we are dealing with now where a schema change exception is thrown
> >> even
> >>> when working with a single Avro file (that has the same schema).
> >>>
> >>> Can a non project member call for a discussion on this topic and the
> >> level
> >>> of support that is offered for Avro in Drill?
> >>>
> >>> My discussion topics would be:
> >>>
> >>>   - Strange schema validation that ... :
> >>>   ... currently fails on single file
> >>>   ... prevents dirX variables to work
> >>>   ... would require Drill to scan all Avro files to establish schema
> >> (even
> >>>   when pruning would be used)
> >>>   ... would ALWAY fail for old queries if the an old Avro file,
> >> containing
> >>>   the original fields, was removed and could not be scanned
> >>>   ... does not rhyme with the "eliminate ETL" and "Evolving Schema"
> >> goals
> >>>   of Drill
> >>>
> >>>   - Simple union types do not work to declare nullable fields
> >>>
> >>>   - Drill can not read Parquet that is created by parquet-mr-avro
> >>>
> >>>   - What is the intention for Avro in Drill
> >>>   - Should we select to use some other format to buffer/badge data
> >> before
> >>>   creating a Parquet file for it?
> >>>
> >>>   - The culture here regarding talking about boring/hard topics like
> >> this
> >>>   - Where serious complaints/issues are met with silence
> >>>   - I know full well that my frustration shines through here and that
> it
> >>>   not helping but this Drill+Avro mess is really getting too much for
> us
> >>> to
> >>>   handle
> >>>
> >>> Look forward do discuss this here or during the next hangout.
> >>>
> >>> Regards,
> >>> -Stefán (or ... mr. old & frustrated)
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message