cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: [DISCUSS] - Releases, Project Management & Funding Thereof
Date Mon, 03 Jul 2017 18:34:34 GMT
I have added Syed to this.  He has done some initial review to get a port
to KVM working, but I am not sure how far he got yet.

*Will Stevens*
CTO

<https://goo.gl/NYZ8KK>

On Mon, Jul 3, 2017 at 2:26 PM, Paul Angus <paul.angus@shapeblue.com> wrote:

> I will take an action to look at trillian on KVM.  There's nothing
> explicit or implicit in trillian itself that it requires vmware as long as
> we can trunk the guest VLANs and virtualise the hypervisors.
>
> ________________________________
> From: Will Stevens <wstevens@cloudops.com>
> Sent: 3 Jul 2017 2:06 am
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] - Releases, Project Management & Funding Thereof
>
> Sorry, I have been keeping up with these threads while on vacation at a
> campsite.  :)  Finally back to a computer.
>
> Yes, ideally we would have more people actually committing code and
> validating the PRs are ready for merge.  Right now, we have VERY limited CI
> setups, so we are bottlenecked on the ability to actually test the PRs in a
> timely fashion.  This leads to PRs sitting for a week at a time in some
> 'phase' of the process.  This means that the developers get pushed off to
> other items to pay the bills and it then causes a lag everywhere.  Because
> of this, the RM has basically had to fill the role of making sure
> everything is moving and understanding and unblocking the different PRs as
> they move through the review/test/commit phases.
>
> Yes, we need more reviewers.  That is very true.
>
> Also true, is that we need more CI environments.  I would love to see
> Trillian be able to be run on KVM on a developers laptop to at least test
> the core components.  We could then start to standardize the dev/test cycle
> for developers so we can start focusing on a 'minimum support feature
> set'.  We could also hopefully leverage the developers setups to run CI
> overnight if they choose to participate (or at least their own PRs).  If we
> could standardize well enough to push the workload to the edge, I think we
> would end up with more active rigs in the game.
>
> I personally feel that if we can put the CI on rails and standardize our
> dev environment, a lot of our 'we need an full time RM' problems go away.
> However, I do think we will need a full time RM for at least a year to be
> able to shepherd that into existence though.
>
> *Will Stevens*
> CTO
>
> <https://goo.gl/NYZ8KK>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> On Sun, Jul 2, 2017 at 8:06 PM, Syed Ahmed <sahmed@cloudops.com> wrote:
>
> > Agree with Ron about this being a role of the commiter but in what I have
> > seen, it is mostly the RM who has to run around and ask for updates/make
> > sure fixes are done.
> >
> > Part of the problem also is that there is a lack of reviewers. We've had
> > some issues recently [1] which were code which was committed was not
> > properly reviewed and later lead to problems. Having a RM and some core
> > reviewers is essential to maintain quality and sanity of the release.
> >
> > I also agree with testing on a known setup with known parameters. The
> > community can pool resources for hardware and Trillian can be used as the
> > CI framework. There was supposedly hadware donated by Citrix to the ASF.
> > Anyone know what happened to that?
> >
> > [1] https://github.com/apache/cloudstack/pull/1834
> >
> >
> > On Sun, Jul 2, 2017 at 9:55 AM, Ron Wheeler <rwheeler@artifact-software.
> > com>
> > wrote:
> >
> > > I think you are describing the roles of all of the committers
> > >
> > > Is it normal at Apache for the RM to be doing all of this stuff?
> > >
> > > I would expect that the RM has a QC role in these activities but others
> > > are doing the work.
> > >
> > > Ron
> > >
> > >
> > > On 01/07/2017 7:18 PM, Will Stevens wrote:
> > >
> > >> Oh, and if a system VM is touched, then you have to add in a new
> system
> > VM
> > >> build and install into the CI setup prior to testing...
> > >>
> > >> On Jul 1, 2017 6:41 PM, wrote:
> > >>
> > >> Which is part of the reason the RM job is hard and time consuming.
> > >> - checking the PRs have the appropriate tests.
> > >> - updating the CI to include the new tests.
> > >> - run and report CI for the PR (with very limited CI infra community
> > >> wide).
> > >> - chase PR authors to get their PRs to a point where you are happy
> they
> > >> are
> > >> not breaking master
> > >> - rinse repeat for 200+ PRs...
> > >>
> > >> On Jul 1, 2017 6:34 PM, "Will Stevens" <williamstevens@gmail.com>
> > wrote:
> > >>
> > >> Yes, we can totally reject PRs until we are happy with the associated
> > >> tests.
> > >>
> > >> On Jul 1, 2017 5:48 PM, "Alex Hitchins" <alex@alexhitchins.com>
> wrote:
> > >>
> > >> Out of interest, are there any guidelines/rules in place to reject
> PR's
> > >>> without unit tests?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Alexander Hitchins
> > >>> ------------------------
> > >>> E: alex@alexhitchins.com
> > >>> W: alexhitchins.com
> > >>> M: 07788 423 969
> > >>> T: 01892 523 587
> > >>>
> > >>> -----Original Message-----
> > >>> From: Paul Angus [mailto:paul.angus@shapeblue.com]
> > >>> Sent: 30 June 2017 21:58
> > >>> To: dev@cloudstack.apache.org
> > >>> Subject: RE: [DISCUSS] - Releases, Project Management & Funding
> Thereof
> > >>>
> > >>> Taken from a talk on CloudStack testing [1]...
> > >>>
> > >>> There are Many, many, MANY permutations of a CloudStack deployment….
> > >>> • Basic / Advanced
> > >>> • Local / shared / mixed storage
> > >>> • More than 8 common hypervisor types/versions • 4 or 5 Management
> > server
> > >>> OS possibilities • That’s 144 combinations only looking the basics.
> > >>>
> > >>> [1] https://www.slideshare.net/ShapeBlue/cloudstack-eu-user-grou
> > >>> p-trillian?qid=74ff2be0-664c-4bca-a3dc-f30d880ca088&v=&b=&
> > from_search=1
> > >>>
> > >>> Trillian [2], can create any of those, and multiple at the same time,
> > but
> > >>> the amount of hardware required to do that means that we have to pick
> > our
> > >>> battles. Running the blueorangutan bot command '@blueorangutan
> matrix'
> > >>> in a
> > >>> PR will run the smoke test suite against a PR using 3 environments,
> one
> > >>> each of KVM, XenServer and vSphere and takes around 8 hours.
> > >>>
> > >>> But that is only looking for major regressions.  A full component
> test
> > >>> run
> > >>> takes around 5 days to run and is riddled with bugs in the tests.
> > >>>
> > >>> Ultimately these are still of limited scope, few people are as
> diligent
> > >>> as
> > >>> say Mike T in creating practical marvin tests for their code /
> > features.
> > >>>
> > >>> [2] https://github.com/shapeblue/Trillian
> > >>>
> > >>> Therefore we need hardware to run tests on, but more importantly we
> > need
> > >>> the tests to exist and work in the first place.  Then we can really
> do
> > >>> something.
> > >>>
> > >>>
> > >>>
> > >>> paul.angus@shapeblue.com
> > >>> www.shapeblue.com<http://www.shapeblue.com>
> > >>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Alex Hitchins [mailto:alex@alexhitchins.com]
> > >>> Sent: 30 June 2017 21:34
> > >>> To: dev@cloudstack.apache.org; dev@cloudstack.apache.org
> > >>> Subject: RE: [DISCUSS] - Releases, Project Management & Funding
> Thereof
> > >>>
> > >>> Consultation with users is something that should definite be done.
> > Canvas
> > >>> as many as possible.
> > >>>
> > >>> I agree that most people will be running test environments before
> full
> > >>> rollout of any technology, I guess see it a little from a CTO eyes -
> > why
> > >>> shortlist a technology that doesn't even endorse its own releases?
> > >>>
> > >>> Hopefully we will get some more replies to this thread from other
> > >>> CloudStack enthusiasts to help shape this conversation.
> > >>>
> > >>> I'm setting up a new development environment now to get my hands
> mildly
> > >>> soiled. Going the Windows route again. Fancy a challenge for the
> > weekend.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Alexander Hitchins
> > >>> ------------------------
> > >>> E: alex@alexhitchins.com
> > >>> W: alexhitchins.com
> > >>> M: 07788 423 969
> > >>> T: 01892 523 587
> > >>>
> > >>> -----Original Message-----
> > >>> From: Ron Wheeler [mailto:rwheeler@artifact-software.com]
> > >>> Sent: 30 June 2017 21:08
> > >>> To: dev@cloudstack.apache.org
> > >>> Subject: Re: [DISCUSS] - Releases, Project Management & Funding
> Thereof
> > >>>
> > >>>
> > >>> On 30/06/2017 3:28 PM, Alex Hitchins wrote:
> > >>>
> > >>>> We can't validate all scenarios no, hence suggesting several common
> > >>>>
> > >>> setups as a reasonable baseline. I like the idea of users posting
> their
> > >>> hardware and versions as a community endeavour.
> > >>>
> > >>>> I strongly feel there should be an established, physical setup that
> > the
> > >>>>
> > >>> release team have access to in order to validate a release.
> > >>>
> > >>> This is perhaps something that should be requested from the user
> > >>> community.
> > >>> I would expect that anyone running Cloudstack in production on a
> major
> > >>> site would have a test setup and might be very happy to have the dev
> > team
> > >>> test on their setup.
> > >>> This would save them a lot of resources at the expense of a bit of
> time
> > >>> on
> > >>> their test environment.
> > >>>
> > >>> If this was some random cat meme generator on GitHub, I'd accept the
> > >>>>
> > >>> risk in running an untested version. For something I'd be running my
> > >>> business on however I'd expect there being some weight behind the
> > >>> release.
> > >>>
> > >>>> Perhaps I have unrealistic expectations!
> > >>>>
> > >>> Not at all.
> > >>> Your expectations might be the key to making a pitch to the user
> > >>> community
> > >>> for some help from people and organizations that are not interested
> in
> > >>> writing code but have a major interest in testing.
> > >>> In addition to access to test equipment, this might actually get new
> > >>> people on the team with the right skills required to extend the test
> > >>> scripts and test procedure documentation.
> > >>>
> > >>> Does anyone have a list of the configuration specifications that are
> > >>> required to test a new release?
> > >>>
> > >>> Would it help to approach major users of Cloudstack with a direct
> > request
> > >>> for use of their test equipment and QA staff in return for early
> access
> > >>> to
> > >>> new releases and testing on their hardware?
> > >>>
> > >>> Ron
> > >>>
> > >>> Alexander Hitchins
> > >>>> ------------------------
> > >>>> E: alex@alexhitchins.com
> > >>>> W: alexhitchins.com
> > >>>> M: 07788 423 969
> > >>>> T: 01892 523 587
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Ron Wheeler [mailto:rwheeler@artifact-software.com]
> > >>>> Sent: 30 June 2017 20:13
> > >>>> To: dev@cloudstack.apache.org
> > >>>> Subject: Re: [DISCUSS] - Releases, Project Management & Funding
> > >>>> Thereof
> > >>>>
> > >>>> On 30/06/2017 2:19 PM, Alex Hitchins wrote:
> > >>>>
> > >>>>> Releasing against a defined reference rig would be a very good
> idea,
> > >>>>>
> > >>>> especially if we could replicate several.
> > >>>
> > >>>> It concerns me slightly that we are building a platform we want to
> > >>>>>
> > >>>> promote people to deploy in enterprise environments with the caveat
> > >>> 'run at
> > >>> your own risk'.
> > >>>
> > >>>> There is no choice as near as I can tell.
> > >>>> It seems that there are too many combinations of hardware, network
> > >>>>
> > >>> configurations and OSs to guarantee that a release will work on all
> of
> > >>> them
> > >>> and still get a release delivered.
> > >>>
> > >>>> As Will pointed out, the Release Team does not have access to every
> > >>>>
> > >>> combination where previous releases are in production use, to test
> the
> > >>> new
> > >>> release candidate.
> > >>>
> > >>>> Currently it may be  not very explicit about what are the fully
> tested
> > >>>>
> > >>> configurations and from what Will said, I gather that there is no
> > policy
> > >>> saying what the minimum test set is to declare a release ready to go.
> > >>>
> > >>>> There is no reason preventing a release being tested after release
> by
> > an
> > >>>>
> > >>> end-user or a developer and adding to the release documentation
> > something
> > >>> to the effect that "Users have reported that this release has been
> put
> > >>> into
> > >>> production on XYZ configuration with no modifications."
> > >>>
> > >>>> This at least gets the release out the door for the 95% of the users
> > >>>>
> > >>> that do not have an XYZ rather than waiting for someone with an XYZ
> to
> > >>> find
> > >>> time to test it.
> > >>>
> > >>>> It may also encourage companies using or selling XYZs to put up some
> > >>>>
> > >>> resources (hardware and people) dedicated to testing so that they get
> > >>> into
> > >>> the initial release.
> > >>>
> > >>>> Ron
> > >>>>
> > >>>> We need to up our game.
> > >>>>>
> > >>>>> 'We' he says, after two years MIA!
> > >>>>>
> > >>>>> On 30 Jun 2017, at 18:41, Ron Wheeler <rwheeler@artifact-software.
> > com>
> > >>>>>>
> > >>>>> wrote:
> > >>>
> > >>>> How much time is there between a feature freeze and the RC being
> cut.?
> > >>>>>> Do people know far enough in advance that their feature is in or
> out
> > >>>>>>
> > >>>>> and if in must be ready to go to a RC release by such and such a
> > date?
> > >>>
> > >>>> Is the use case testing well defined - hardware, configurations,
> etc.
> > >>>>>> Can you put out a release that says: "This release has been tested
> > on
> > >>>>>>
> > >>>>> these configurations (A, B ,C) but the following configurations/use
> > >>> cases
> > >>> are not yet fully tested and other configuration may be used at your
> > own
> > >>> risk after your own internal tests have been run successfully."
> > >>>
> > >>>> Is there any concept that "Cloudstack is verified to run on the
> > >>>>>>
> > >>>>> following configurations and should also run on these
> configurations
> > >>> but
> > >>> has not been tested fully. It may run on these configurations but is
> > not
> > >>> tested during the release cycle."
> > >>>
> > >>>> Ron
> > >>>>>>
> > >>>>>> On 30/06/2017 1:14 PM, Will Stevens wrote:
> > >>>>>>> Have not looked at Release Tsar, but worth checking out.
> > >>>>>>>
> > >>>>>>> In general, the biggest problem we have with releasing on a
> > >>>>>>> schedule is the lack of a CI setup which covers the entire
> > >>>>>>> software. Or at least a 'supported' set of features. This means
> > >>>>>>> that the release is always bound to a bunch of volunteers getting
> > >>>>>>> around to testing their use case. Solidfire and Nuage are pretty
> > good
> > >>>>>>>
> > >>>>>> about getting some CI run on some pieces.
> > >>>
> > >>>> Trillian is great for covering a portion of the tests, but it
> > >>>>>>> currently does not cover the whole software use case. We also
> need
> > >>>>>>> more trillian deployments in the wild to support the CI
> initiative.
> > >>>>>>>
> > >>>>>>> We do need to be stricter about nothing going in after an RC is
> cut
> > >>>>>>> but blockers. The limited CI coverage and the dependence on a few
> > >>>>>>> people for testing exasperates this problem.
> > >>>>>>>
> > >>>>>>> So there is multiple layers to this. I think someone dedicates to
> > >>>>>>> the RM role would help this a lot because they would have a
> single
> > >>>>>>> community focus mandate, so it is in their best interest to
> > >>>>>>> implement a flow which does not inhibit their ability to deliver
> on
> > >>>>>>>
> > >>>>>> their mandate.
> > >>>
> > >>>> On Jun 30, 2017 12:53 PM, "Ron Wheeler"
> > >>>>>>> <rwheeler@artifact-software.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Perhaps a Release Tsar would be a better solution.
> > >>>>>>>> The RM needs to have absolute control over what is in or out.
> > >>>>>>>> Reasonable discussion allowed and then a decision once the RM
> > >>>>>>>> feels that the case has been fully explored and that a positive
> > vote
> > >>>>>>>>
> > >>>>>>> is expected.
> > >>>
> > >>>> The importance on meeting deadlines needs to have a higher
> > >>>>>>>> priority. If a feature/fix can not meet the quality/testing
> > >>>>>>>> threshold on time then it gets dropped from the RC and scheduled
> > for
> > >>>>>>>>
> > >>>>>>> the next release.
> > >>>
> > >>>> A few cycles of a bit of ruthlessness should get everyone`s
> > >>>>>>>> intention and shorten the release cycle.
> > >>>>>>>>
> > >>>>>>>> Meeting release schedules would also reduce the pain of a
> feature
> > >>>>>>>> being deferred.
> > >>>>>>>> According to the schedule proposed last
> > >>>>>>>> year,(https://cwiki.apache.org
> > >>>>>>>> /confluence/display/CLOUDSTACK/%5BPROPOSAL%5D+2016-2017+
> > >>>>>>>> Release+Cycle+and+Calendar)
> > >>>>>>>>     Cloudstack 4.9.10 (LTS) , 5.04.0 (LTS) as well as 5.1.3.0
> > >>>>>>>> (maintenance)
> > >>>>>>>> 5.2.1.0 (Maintenance) were released June 2017.
> > >>>>>>>>
> > >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > Release+Pro
> > >>>>>>>> c edure seems to be pretty reasonable. The RM probably needs to
> > >>>>>>>> moderate the vote and explain what -1 votes mean to product
> > >>>>>>>> credibility if they delay the release. Negative votes because
> > >>>>>>>> someone`s new feature did not make it should be ignored.
> > >>>>>>>>
> > >>>>>>>> Ron
> > >>>>>>>>
> > >>>>>>>> On 30/06/2017 12:09 PM, Paul Angus wrote:
> > >>>>>>>>>
> > >>>>>>>>> We could probably split this topic down also....
> > >>>>>>>>>
> > >>>>>>>>> I think I may have mentioned previously ?? my view on how we
> have
> > >>>>>>>>> somewhat shot ourselves in the foot with the release process
> this
> > >>>>>>>>> time around.  I think that for the most part, people have been
> > >>>>>>>>> well intentioned, and have been trying to 'make this release as
> > >>>>>>>>> good as possible' which is counter-productive, as it's been
> > >>>>>>>>>
> > >>>>>>>> introducing new blockers.
> > >>>
> > >>>> I'm not sure we have a problem in our 'loosely-agreed' process,
> > >>>>>>>>> it's just that repeatedly people have ignored it.
> > >>>>>>>>>
> > >>>>>>>>> WRT a full-time release manager, I suspect that they would find
> > >>>>>>>>> that "you can lead a horse to water, but you can't make it
> > drink".
> > >>>>>>>>> They would not be able to compel anyone to 'hurry up and fix
> that
> > >>>>>>>>> bug you created', although I guess maybe they could pull a
> > feature
> > >>>>>>>>>
> > >>>>>>>> if the author(s) didn't sort it out.
> > >>>
> > >>>> Because ultimately a release manager, paid or otherwise should
> > >>>>>>>>> only be doing what the 'community' decides the release
> manager's
> > >>>>>>>>> role is.  So we need to be clear about how we want releases to
> > >>>>>>>>> work before worrying about who manages that.
> > >>>>>>>>>
> > >>>>>>>>> Kind regards,
> > >>>>>>>>>
> > >>>>>>>>> Paul Angus
> > >>>>>>>>>
> > >>>>>>>>> paul.angus@shapeblue.com
> > >>>>>>>>> www.shapeblue.com<http://www.shapeblue.com>
> > >>>>>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: Alex Hitchins [mailto:alex@alexhitchins.com]
> > >>>>>>>>> Sent: 30 June 2017 15:05
> > >>>>>>>>> To: dev@cloudstack.apache.org
> > >>>>>>>>> Subject: RE: [DISCUSS] - Releases, Project Management & Funding
> > >>>>>>>>> Thereof
> > >>>>>>>>>
> > >>>>>>>>> I am in complete agreement with you. Also on your other reply
> > >>>>>>>>> regards to a FT release manager.
> > >>>>>>>>>
> > >>>>>>>>> If 'we' don't go down this line, more and more people will
> follow
> > >>>>>>>>> the Cosmic/Schuberg Philis path or even use Cosmic instead.
> > >>>>>>>>>
> > >>>>>>>>> I'm encouraged by your response. Sounds like a few others hold
> > >>>>>>>>> the same concerns.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Alexander Hitchins
> > >>>>>>>>> ------------------------
> > >>>>>>>>> E: alex@alexhitchins.com
> > >>>>>>>>> W: alexhitchins.com
> > >>>>>>>>> M: 07788 423 969
> > >>>>>>>>> T: 01892 523 587
> > >>>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: Will Stevens [mailto:williamstevens@gmail.com]
> > >>>>>>>>> Sent: 30 June 2017 14:54
> > >>>>>>>>> To: dev@cloudstack.apache.org
> > >>>>>>>>> Subject: RE: [DISCUSS] - Releases, Project Management & Funding
> > >>>>>>>>> Thereof
> > >>>>>>>>>
> > >>>>>>>>> Yes, Schuberg Philis, a very active community member forked
> > >>>>>>>>> Cosmic off of CloudStack and has been developing their fork for
> > >>>>>>>>>
> > >>>>>>>> their needs.
> > >>>
> > >>>> I do think we need to have a more consistent front on this
> > >>>>>>>>> matter. I think it would make a big difference on the quality,
> > >>>>>>>>> release cadence and perception of the project.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Jun 30, 2017 9:48 AM, "Alex Hitchins" <
> alex@alexhitchins.com>
> > >>>>>>>>>
> > >>>>>>>> wrote:
> > >>>
> > >>>> Thanks Will,
> > >>>>>>>>>
> > >>>>>>>>> I understand it's something that comes with a big bag of
> > >>>>>>>>> troublesome worries.
> > >>>>>>>>>
> > >>>>>>>>> If this topic comes up again in any discussions, I'd be
> > >>>>>>>>> interested to hear their thoughts on what I see as the
> > >>>>>>>>> alternative; without a dedicated RM/PM/Captain, people will
> fork
> > >>>>>>>>> off CS so they can achieve the same thing, and CS ultimately
> > >>>>>>>>> looses out long term. I can't remember the name of the fork,
> but
> > >>>>>>>>> I think I'm right that a previous large CS contributor/user
> > >>>>>>>>> forked off as they wanted greater management in the areas we
> are
> > >>>>>>>>>
> > >>>>>>>> discussing here.
> > >>>
> > >>>>
> > >>>>>>>>> Alexander Hitchins
> > >>>>>>>>> ------------------------
> > >>>>>>>>> E: alex@alexhitchins.com
> > >>>>>>>>> W: alexhitchins.com
> > >>>>>>>>> M: 07788 423 969
> > >>>>>>>>> T: 01892 523 587
> > >>>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: Will Stevens [mailto:williamstevens@gmail.com]
> > >>>>>>>>> Sent: 30 June 2017 14:31
> > >>>>>>>>> To: dev@cloudstack.apache.org
> > >>>>>>>>> Subject: Re: [DISCUSS] - Releases, Project Management & Funding
> > >>>>>>>>> Thereof
> > >>>>>>>>>
> > >>>>>>>>> Apache has been historically against the idea of a cloudstack
> > >>>>>>>>> foundation and there is a bit of a pandoras box there which we
> > >>>>>>>>> will want to be careful about opening.
> > >>>>>>>>>
> > >>>>>>>>> Apache added direct contribution, but it was unusable for us
> > >>>>>>>>> historically because it required a minimum contribution of 50k,
> > >>>>>>>>> which none of us can afford. However, there have been some
> > >>>>>>>>> changes to the board recently which are in our favour if we
> want
> > to
> > >>>>>>>>>
> > >>>>>>>> put pressure to lower that to say 5-10k.
> > >>>
> > >>>> Even if we do solve for smaller direct contributions, we will
> > >>>>>>>>> have to jump through hoops to be able to use those funds for a
> > >>>>>>>>> dedicated release manager. I do think this is a possibility if
> we
> > >>>>>>>>> manage our needs and communications very well. I had some
> > >>>>>>>>> preliminary discussions with some apache foundation folks to
> > >>>>>>>>> express these specific concerns. I played off the fact that i
> > >>>>>>>>> know they dont want to entertain a cloudstack foundation and
> > >>>>>>>>> tried to see if i could get them to move on the direct
> > >>>>>>>>> contribution mechanism to make it usable for us, specifically
> > >>>>>>>>> with the goal of hiring a full time release manager. I
> definitely
> > >>>>>>>>> had their ear and they acknowledged the problems we are facing
> > >>>>>>>>> (and currently discussing).  They expressed concerns about
> being
> > >>>>>>>>> able to hire someone with the direct contributions, but
> > >>>>>>>>> brainstormed a bit to potentially hire an agency who actually
> > does
> > >>>>>>>>>
> > >>>>>>>> the hire and they pay the persons salary through the agency with
> > >>> the direct
> > >>> contribution funds.
> > >>>
> > >>>> All to say, there are potential options here, but there be
> > >>>>>>>>> dragons, so we have to handle this topic with care.
> > >>>>>>>>>
> > >>>>>>>>> On Jun 30, 2017 9:12 AM, "Ron Wheeler"
> > >>>>>>>>> <rwheeler@artifact-software.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> https://www.apache.org/foundation/contributing.html says:
> > >>>>>>>>>
> > >>>>>>>>>> "If you have a specific target or project that you wish to
> > >>>>>>>>>> directly support, pleasecontact us
> > >>>>>>>>>> <https://www.apache.org/founda
> > >>>>>>>>>> tion/contributing.html#Fundraising>and we will do our best to
> > >>>>>>>>>>
> > >>>>>>>>> satisfy your wishes."
> > >>>
> > >>>> 1) Is Apache willing to allow projects to set up their own
> > >>>>>>>>>> foundations? I doubt but someone would need to check this out.
> > >>>>>>>>>> Does the PMC have the project charter or the agreement that
> was
> > >>>>>>>>>> signed when Cloudstack moved.
> > >>>>>>>>>>
> > >>>>>>>>>> 2) Has anyone tried to contact Apache about directing support
> to
> > >>>>>>>>>> Cloudstack.
> > >>>>>>>>>>
> > >>>>>>>>>> I am not convinced that lack of paid staff is the issue.
> > >>>>>>>>>> This discussion reminded me of this.
> > >>>>>>>>>> Q: How many psychiatrists does it take to change a lightbulb ?
> > >>>>>>>>>> A: Only one, but the lightbulb must want to change
> > >>>>>>>>>>
> > >>>>>>>>>> http://www.lightbulbjokes.com/directory/p.html
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Ron
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 30/06/2017 6:48 AM, Alex Hitchins wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> As per Giles's comment to the previous thread, I thought I
> would
> > >>>>>>>>>>
> > >>>>>>>>>>> start a discussion on the subject to canvas peoples thoughts,
> > >>>>>>>>>>> opinions
> > >>>>>>>>>>>
> > >>>>>>>>>>> and fears.
> > >>>>>>>>>> My question for discussion, is there is any mileage in someone
> > >>>>>>>>>>
> > >>>>>>>>>>> creating a "CloudStack Foundation" as a non-profit entity,
> > >>>>>>>>>>> funded largely by key CloudStack players with the sole
> function
> > >>>>>>>>>>> of employing dedicated resource (part or full time) to handle
> > >>>>>>>>>>> all releases and other essential 'back office' functions. The
> > >>>>>>>>>>> idea being it's in everyone's interest to chip in a little
> each
> > >>>>>>>>>>> to fund core project and
> > >>>>>>>>>>>
> > >>>>>>>>>>> release management.
> > >>>>>>>>>> The idea might be utterly irrelevant, pointless and/or
> straight
> > up
> > >>>>>>>>>>
> > >>>>>>>>> daft.
> > >>>
> > >>>> I urge you all to let me know.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Something for you all to think over this weekend.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Alexander Hitchins
> > >>>>>>>>>>> ------------------------
> > >>>>>>>>>>> E: alex@alexhitchins.com
> > >>>>>>>>>>> W: alexhitchins.com
> > >>>>>>>>>>> M: 07788 423 969
> > >>>>>>>>>>> T: 01892 523 587
> > >>>>>>>>>>>
> > >>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>> From: Giles Sirett [mailto:giles.sirett@shapeblue.com]
> > >>>>>>>>>>> Sent: 30 June 2017 09:51
> > >>>>>>>>>>> To: dev@cloudstack.apache.org
> > >>>>>>>>>>> Subject: RE: JIRA - PLEASE READ
> > >>>>>>>>>>>
> > >>>>>>>>>>> All
> > >>>>>>>>>>> This thread seems to have turned into 2 quite different
> > >>>>>>>>>>>
> > >>>>>>>>>> discussions:
> > >>>
> > >>>> 1. The use (or not) of Jira - which was the original discussion
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. Ways/means of encouraging (and paying for more structured
> > >>>>>>>>>>> contributors)
> > >>>>>>>>>>>
> > >>>>>>>>>>> I know that it could be argued that these are related. Could
> I
> > >>>>>>>>>>> suggest opening up a thread on "release and project
> management
> > >>>>>>>>>>> and funding it"  and keeping this thread to the original
> > >>>>>>>>>>> discussion
> > >>>>>>>>>>>
> > >>>>>>>>>>> (I will weigh in on both of these at some stage)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Kind regards
> > >>>>>>>>>>> Giles
> > >>>>>>>>>>>
> > >>>>>>>>>>> giles.sirett@shapeblue.com
> > >>>>>>>>>>> www.shapeblue.com<http://www.shapeblue.com>
> > >>>>>>>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>> From: Alex Hitchins [mailto:alex@alexhitchins.com]
> > >>>>>>>>>>> Sent: 29 June 2017 18:49
> > >>>>>>>>>>> To: dev@cloudstack.apache.org
> > >>>>>>>>>>> Subject: Re: JIRA - PLEASE READ
> > >>>>>>>>>>>
> > >>>>>>>>>>> If it isn't being treated as a product it will be very
> > >>>>>>>>>>> impossible to market it as enterprise ready.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I know we all know this.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Similar sized projects under the Apache banner must have the
> > >>>>>>>>>>> same issue, what is the best way to gather experience of
> these
> > >>>>>>>>>>>
> > >>>>>>>>>> projects?
> > >>>
> > >>>> See how they handle these growing pains.
> > >>>>>>>>>>>
> > >>>>>>>>>>> A cloudstack foundation entity funded by companies earning
> from
> > >>>>>>>>>>> cloudstack seems a good way forward.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Another tuppence, this is getting expensive.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 29 Jun 2017, at 18:18, Ron Wheeler
> > >>>>>>>>>>> <rwheeler@artifact-software.com>
> > >>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I understand that it is a volunteer organization.
> > >>>>>>>>>>>> I do not know how many (if any) of the committers and PMC
> > >>>>>>>>>>>> members are funded by their organizations (allowed or
> ordered
> > >>>>>>>>>>>> to work on Cloudstack during company time) which is often
> the
> > >>>>>>>>>>>> way that Apache projects get staffed.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Clearly it is hard to tell someone who is being funded by a
> > >>>>>>>>>>>> company to fix a problem or who is working on their own
> time,
> > >>>>>>>>>>>> to do or not do something.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On the other hand, the PMC has to  build a community culture
> > >>>>>>>>>>>> that is good for the project.
> > >>>>>>>>>>>> That means describing a vision, planning and enforcing a
> > >>>>>>>>>>>> roadmap, and maintaining a focused project "marketing"
> effort.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> There is a lot of extremely talented individuals working on
> > >>>>>>>>>>>> Cloudstack and it appears to have a very strong and valuable
> > >>>>>>>>>>>>
> > >>>>>>>>>>> code-base.
> > >>>
> > >>>> To me the key question is about the PMC and the core committers'
> > >>>>>>>>>>>> ability to make Cloudstack a "product" that can compete for
> > >>>>>>>>>>>> market share and acceptance.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Is Cloudstack at a point in its development where it should
> be
> > >>>>>>>>>>>> treated like a product?
> > >>>>>>>>>>>> - sufficient functionality to compete
> > >>>>>>>>>>>> - sufficient user base to be a competitor in the market
> > >>>>>>>>>>>> - production reliability and stability
> > >>>>>>>>>>>> - business model for supporting companies to justify their
> > >>>>>>>>>>>> continued support
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This may not require more effort but requires different
> > >>>>>>>>>>>> policies and different activities.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> There has to be someone or a PMC  that can say "No".
> > >>>>>>>>>>>> - This change can not be included in this release because it
> > >>>>>>>>>>>> will delay the release.
> > >>>>>>>>>>>> - This change adds an unacceptable level of complexity
> > >>>>>>>>>>>> - This bug fix will have to wait for the next release
> because
> > >>>>>>>>>>>> it is too late to test it and fix the docs.
> > >>>>>>>>>>>> - This fix breaks the docs
> > >>>>>>>>>>>> - The release can not be made until this doc is updated.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Does the core group want to make it a competitive product or
> > >>>>>>>>>>>> is it sufficient for the interested players to continue in
> its
> > >>>>>>>>>>>>
> > >>>>>>>>>>> current form?
> > >>>
> > >>>> Ron
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 29/06/2017 9:42 AM, Will Stevens wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I personally don't know how Jira solves any of this, but
> > >>>>>>>>>>>>> assuming it does, fine...
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The bigger problem which you have raised is that CloudStack
> > >>>>>>>>>>>>> has zero funding. So we can't hire a project manager, or a
> > >>>>>>>>>>>>> release manager or someone whose job it is to maintain
> > >>>>>>>>>>>>> documentation. I have been trying to find a way to, at the
> > >>>>>>>>>>>>> very least, fund a full time release manager who can focus
> > >>>>>>>>>>>>> 100% on the project. As the release manager for 4.9, I know
> > >>>>>>>>>>>>> it is a full time job. I did my best, but it is a ton of
> work
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> and is hard to stay on top of.
> > >>>
> > >>>> Everyone contributing to CloudStack is donating their time.
> > >>>>>>>>>>>>> They can't make a living off supporting ACS, so every one
> is
> > >>>>>>>>>>>>> doing their best with the little time they can take away
> from
> > >>>>>>>>>>>>> their day job or their family life.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes, having clear guidelines and sticking to them helps,
> but
> > >>>>>>>>>>>>> without a solid CI infrastructure backing the project and
> > >>>>>>>>>>>>> improved testing and automation, we will always struggles
> > >>>>>>>>>>>>> with release schedules and such.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I have been involved in this project long enough to know
> that
> > >>>>>>>>>>>>> all the problems you point out exist, but they are also not
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> easily solved.
> > >>>
> > >>>> Obviously we have to work with the initiatives we have and
> > >>>>>>>>>>>>> take small steps towards improvement, but we also have to
> be
> > >>>>>>>>>>>>> realistic with our expectations because we are counting on
> > >>>>>>>>>>>>> people's generosity to move them forward.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Simplifying moving parts and streamlining the process will
> > >>>>>>>>>>>>> lead to more contribution because there is less barriers to
> > >>>>>>>>>>>>> entry. This one reason why I struggle to see the value in
> > Jira
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> as it is used today.
> > >>>
> > >>>> I personally don't understand what value it is giving us that
> > >>>>>>>>>>>>> the github PRs and Issues don't solve.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I will remain open minded and will follow along with what
> > >>>>>>>>>>>>> people think is best, but I think it is worth understanding
> > >>>>>>>>>>>>> what we are trying to solve for and simplify our approach
> in
> > >>>>>>>>>>>>> solving it so we can get better systems in place.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Jun 29, 2017 9:17 AM, "Ron Wheeler"
> > >>>>>>>>>>>>> <rwheeler@artifact-software.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> As a real outsider, IMHO Paul is right.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> At times it seems that Cloudstack is a coding hobby rather
> > >>>>>>>>>>>>>> than a project or a production quality product.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Who decides what goes into a release? How does this affect
> > >>>>>>>>>>>>>> the release schedule?
> > >>>>>>>>>>>>>> Who is responsible for meeting the "published" roadmap (of
> > >>>>>>>>>>>>>> which there seem to be many) of releases?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> How is a system admin that is not part of the project
> > >>>>>>>>>>>>>> supposed to plan for upgrade windows?
> > >>>>>>>>>>>>>> How does one know when a feature, bug fix or release will
> be
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> available?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> How does the PMC  manage function creep  in a release,
> > maintain
> > >>>>>>>>>>
> > >>>>>>>>>>> quality and consistency, reject changes that hurt the
> > >>>>>>>>>>>>>> overall vision or add too much complexity?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> No one seems to care about documentation but if someone
> did,
> > >>>>>>>>>>>>>> how would they stop undocumented features or features that
> > >>>>>>>>>>>>>> contradict the documentation from being incorporated?
> > >>>>>>>>>>>>>> Who makes sure that the documentation is correct at the
> time
> > >>>>>>>>>>>>>> of the release?
> > >>>>>>>>>>>>>> Release notes are not much help for someone doing a new
> > >>>>>>>>>>>>>> install or evaluating Cloudstack.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Without a JIRA entry, how does an end-user who encounters
> a
> > >>>>>>>>>>>>>> problem know that it has been fixed already in the next
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> release?
> > >>>
> > >>>> Without a JIRA entry, how does the community comment on a
> > >>>>>>>>>>>>>> proposed change before it gets coded?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If changes are going to be accepted without a JIRA, is
> there
> > >>>>>>>>>>>>>> a definition of a minor fix that does not require a JIRA?
> > >>>>>>>>>>>>>> - does not change functionality?
> > >>>>>>>>>>>>>> - only affects an "edge case" or cleans up an exception
> that
> > >>>>>>>>>>>>>> is not properly handled?
> > >>>>>>>>>>>>>> - only improves code readability or future extensibility?
> > >>>>>>>>>>>>>> - does not affect documentation?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Apache projects that are popular and enjoy wide support do
> > >>>>>>>>>>>>>> have strong management.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> There are other examples where great Apache software is
> > >>>>>>>>>>>>>> failing to get recognized because the PMC is not paying
> > >>>>>>>>>>>>>> attention to the product management side of things.
> > >>>>>>>>>>>>>> I use Apache Jackrabbit which is a quality product with a
> > >>>>>>>>>>>>>> strong technical team supporting it.
> > >>>>>>>>>>>>>> It has very little following because the documentation and
> > >>>>>>>>>>>>>> marketing collateral is very poor.
> > >>>>>>>>>>>>>> It gets by because the audience for it is largely software
> > >>>>>>>>>>>>>> developers who can read code and can test features to work
> > >>>>>>>>>>>>>> out the functionality.
> > >>>>>>>>>>>>>> It would get a lot more attention if they paid attention
> to
> > >>>>>>>>>>>>>> the product management side of the project.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cloudstack needs to avoid this situation and unfortunately
> > >>>>>>>>>>>>>> this takes effort and some discipline.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ron
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 29/06/2017 8:03 AM, Will Stevens wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Why are we still using jira instead of the PRs for that
> > >>>>>>>>>>>>>>> communication? Can we not use issues in github now
> instead
> > >>>>>>>>>>>>>>> of jira if someone needs to open an issue but does not
> yet
> > >>>>>>>>>>>>>>> have code to contribute. If not, jira could still be used
> > for
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> that.
> > >>>
> > >>>> I think duplicating data between jira and the PR is kind of
> > >>>>>>>>>>>>>>> pointless. I feel like the github PRs and the cide going
> in
> > >>>>>>>>>>>>>>> should be the source of truth, not a random third party
> > tool.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> For the 4.9 release notes, i built a tool to generate the
> > >>>>>>>>>>>>>>> release notes from the PRs merged in that release. I
> think
> > >>>>>>>>>>>>>>> that is easier and more accurate than depending on jira
> > >>>>>>>>>>>>>>> since it does not track the actual code tree.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thats my 0.02$.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Jun 29, 2017 5:25 AM, "Paul Angus"
> > >>>>>>>>>>>>>>> <paul.angus@shapeblue.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Such a view of CloudStack is what holds CloudStack back.
> > >>>>>>>>>>>>>>> It stops users/operators from having any chance of
> > >>>>>>>>>>>>>>> understanding what CloudStack does and how it does it.
> > >>>>>>>>>>>>>>> Code for code's sake is no use to anyone.
> > >>>>>>>>>>>>>>> Jira is about communication between developers and to
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> everyone else.
> > >>>
> > >>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Paul Angus
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> paul.angus@shapeblue.com
> > >>>>>>>>>>>>>>> www.shapeblue.com<http://www.shapeblue.com>
> > >>>>>>>>>>>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >>>>>>>>>>>>>>> @shapeblue
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> > >>>>>>>>>>>>>>> Sent: 29 June 2017 10:14
> > >>>>>>>>>>>>>>> To: dev <dev@cloudstack.apache.org>
> > >>>>>>>>>>>>>>> Subject: Re: JIRA - PLEASE READ
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 11:06 AM, Paul Angus
> > >>>>>>>>>>>>>>> <paul.angus@shapeblue.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> + Release notes will be impossible to create without a
> > >>>>>>>>>>>>>>> + proper Jira
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> history.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> And no one will know what has gone into CloudStack.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> No they are not mr Grumpy. they should be base on the
> code
> > >>>>>>>>>>>>>>>> anyway,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> hence on git, not jira. I do not appose to the use of
> Jira
> > >>>>>>>>>>>>>>> but it is not required for good coding practices and as
> we
> > >>>>>>>>>>>>>>> are not and will not function as a corporation, jira is
> an
> > >>>>>>>>>>>>>>> extra for those that grave for it. not a requirement.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Daan
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message