beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobiasz Kędzierski <tobiasz.kedzier...@polidea.com>
Subject Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Date Mon, 01 Jun 2020 15:26:54 GMT
Hi everyone,

Austin Benett I would like cooperate with you with introducing GitHub actions for python wheels.

Since last week I am trying to figure out how to do this and I was able to build wheels for
Linux and MacOS with help of https://github.com/joerick/cibuildwheel/ <https://github.com/joerick/cibuildwheel/>.

I’ve created draft PR to apache/beam: https://github.com/apache/beam/pull/11877 <https://github.com/apache/beam/pull/11877>
and you can check GitHub actions logs on fork PR here:
https://github.com/TobKed/beam/pull/3 <https://github.com/TobKed/beam/pull/3>

I am struggling with building wheels for Windows due to the error:
> Cannot open include file: 'unistd.h': No such file or directory

caused by file sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
Adding this file to be excluded by cythonize method in setup.py allowed build to be completed
however I don’t think it is a solution.
Mentioned error on GitHub actions: https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977
<https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977> and
my local PR: https://github.com/TobKed/beam/pull/2 <https://github.com/TobKed/beam/pull/2>

What do you think about it?

Tobiasz



On 2020/05/30 20:53:51, Austin Bennett <w...@gmail.com> wrote: 
> There seems to be support here for this idea.  After digging through> 
> things, I *think* I understand the moving pieces and can address (reason I> 
> was digging through the code, including beam-wheels repo [1]).  Though, not> 
> 100% that I have not overlooked bits.> 
> 
> Would be happy to pickup> 
> https://issues.apache.org/jira/browse/BEAM-9388 (imagine> 
> would get to it over the course of a month or few -- so also wouldn't want> 
> to take it on if others would be looking to do it sooner) -- seems> 
> worthwhile to reduce manual steps for the release process, as well as the> 
> additional implications (daily builds) if can easily manage.  I have some> 
> experience with GitHub Actions, but not with our (BEAM) deployment/release> 
> process, nor much with building wheels.  I assume there are sufficient docs> 
> available to go through for the mechanics of the latter.  If picking up> 
> this issue, I suspect may need to bother some of those that have done> 
> recent releases -- that would best occur on the jira ticket, once at that> 
> point?> 
> 
> @Ismail <ie...@gmail.com>-- given your experience with GitHub actions,> 
> any reason to think GitHub Actions not appropriate/ready for this specific> 
> task?> 
> 
> @Ahmet Altay <al...@google.com> happy to understand the extent of what you>

> had in mind, maybe the extensions are not as important to plan out, as> 
> they're straightforwardly bolted on (ex: daily builds).  More tactically> 
> would be valuable to ensure I understand what all needs to occur.  Any> 
> other source of info to consume other than> 
> https://github.com/apache/beam-wheels and> 
> https://beam.apache.org/contribute/release-guide/.> 
> 
> Also, open to the thought that this might be taking on too much, without> 
> more experience with the release process and such.  Do advise...?> 
> 
> 
> [1]> 
> https://lists.apache.org/thread.html/r93884eb080297647207f7d2b8a393e224029fc2c3509017886e84051%40%3Cdev.beam.apache.org%3E>

> 
> 
> 
> ---------- Forwarded message ---------> 
> From: Ismaël Mejía <ie...@gmail.com>> 
> Date: Wed, Feb 26, 2020 at 12:09 PM> 
> Subject: Re: [DISCUSSION] Use github actions for python wheels ?> 
> To: dev <de...@beam.apache.org>> 
> 
> 
> +1 I have been migrating multiple projects into github actions recently and> 
> even if for some tasks it is not as mature and polished as travis it has> 
> proven to be way more reliable.> 
> 
> 
> On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay <al...@google.com> wrote:> 
> 
> > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore> 
> > this. To be explicit and not to do cookie licking, I would not be able to> 
> > work on this at the moment. If anyone is interested please take it.> 
> > Otherwise I will try to come back and explore this when I can.> 
> >> 
> > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw <ro...@google.com>> 
> > wrote:> 
> >> 
> >> I'd be in favor of this, assuming it actually simplifies things.> 
> >> 
> >> 
> > This is also my concern. I do think that it will simplify things, but I am> 
> > not certain as I am not very familiar with the github actions.> 
> >> 
> >> 
> >> (Note> 
> >> that the wheels are for several variants of linux, presumably we could> 
> >> do cross-compiles. Also, manylinux is a "minimal" linux specifically> 
> >> built as to produce shared object libraries compatible with a wide> 
> >> variety of distributions--we can't just assume that a shared object> 
> >> library built on one modern linux will just work on another. (But> 
> >> maybe it's sufficient to do this within a docker environment?)> 
> >>> 
> >> 
> > There will be no change in this area. Both in Both Travis and github> 
> > actions offer a comparable set of options.> 
> >> 
> >> 
> >>> 
> >> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles <ke...@apache.org> wrote:>

> >> >> 
> >> > +1 to exploring this.> 
> >> >> 
> >> > On builds@apache.org there is lots of discussion and general approval>

> >> for trying it. It is enabled and used by some projects. Calcite uses it to>

> >> build their website, for example.> 
> >>> 
> >> 
> > Great.> 
> >> 
> >> 
> >> >> 
> >> > Kenn> 
> >> >> 
> >> >> 
> >> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay <al...@google.com> wrote:>

> >> >>> 
> >> >> Hi all,> 
> >> >>> 
> >> >> I recently had a chance to look at the documentation for github>

> >> actions. I think we could use github actions instead of travis to for> 
> >> building python wheels during releases. This will have the following> 
> >> advantages:> 
> >> >>> 
> >> >> - We will eliminate one repo. (If you don't know, we have> 
> >> https://github.com/apache/beam-wheels for the sole purpose of building> 
> >> wheels file.)> 
> >> >> - Workflow will be stored in the same repo. This will prevent bit rot>

> >> that is only discovered at release times. (happened a few times, although>

> >> usually easy to fix.)> 
> >> >> - github actions supports ubuntu, mac, windows environments. We could>

> >> try to build wheels for windows as well. (Travis also supports the same>

> >> environments but we only use linux and mac environments. Maybe there are>

> >> other blockers for building wheels for Windows.)> 
> >> >> - We could do more, like daily python builds.> 
> >> >>> 
> >> >> Downsides would be:> 
> >> >> - I do not know if github actions will require some special set of>

> >> permissions that require an approval from infra.> 
> >> >> - Travis works fine most of the time. This might be unnecessary work.>

> >> >>> 
> >> >> What do you think? Is this feasible, would this be useful?> 
> >> >>> 
> >> >> Ahmet> 
> >>> 
> >> 
> 
Mime
View raw message