From dev-return-4798-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Mon Apr 2 18:15:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5065E180627 for ; Mon, 2 Apr 2018 18:15:00 +0200 (CEST) Received: (qmail 18872 invoked by uid 500); 2 Apr 2018 16:14:54 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 18856 invoked by uid 99); 2 Apr 2018 16:14:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2018 16:14:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 264FC1A0405 for ; Mon, 2 Apr 2018 16:14:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.88 X-Spam-Level: *** X-Spam-Status: No, score=3.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=quantopian.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 4JUM4msmw3D4 for ; Mon, 2 Apr 2018 16:14:49 +0000 (UTC) Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DA5F85F477 for ; Mon, 2 Apr 2018 16:14:48 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id 188-v6so13285783oih.8 for ; Mon, 02 Apr 2018 09:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quantopian.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=YJDm33HbV7TFvm9QpMcAuZhPo/HJ4/VtvpinUzIhtN0=; b=w3jSwFTE+bU+Qf28eMTYNiVDyPJLo9WB/mgBoK2a8nN/d7AiafZ7ljQjNyjnFW4gpq xtxX50qprfpyYcZokfMikomNGYdMnvq1CkSOYkE/ap5nFkO04qltabwPhlJWPru+S9mg cpYYibpKuhK92p4iVt6FMzljVglCO/KDHNGvzKe1a+BLqSxx38Hfg30/uHy0gSz790gq sZ2RemLK5pA7GLdyA1lY10y4oA9tYHpof42wfTiZWITaNJEXp4kFVWVGA5DUDx0fctTT R7WxP9aTx28LEnGAYBobjr/dNK4HRevdlMNQmzYqsJOjbIMMpNacvbXhvU5L1oErAOLx DTeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=YJDm33HbV7TFvm9QpMcAuZhPo/HJ4/VtvpinUzIhtN0=; b=Mzw/hw9EztQQ9An8CJDOMkkuhPGX1J5AoQjhtlj2tAtKqO/Yw6qdfxTjomxa9CT/4z mbVch29O7DVRBXFO2fT2ex+yajwCpE15jvN0QxQZQtSE4CgfPYMMxVZHs/0xGZxQGehM kujdUOEO7UwHAdQWFOs8O3xg8LGFBd0L/qz1+a64IpxGfs2z0RaaDdaM0no5o6XmsLfW xN7W5trOlDVJLRl4+KgMus/gsTb31I5Ojz9ZvD1FQU5sOTv7zBIpEk8gDiDWqkOAfaU8 LwIcslRX6YKXs8OkxoXVHooa6hcsar+ZAS1c4CM7Pubr2xaFsIeYZ1Z4/ADCeNZuDL6m fb0g== X-Gm-Message-State: AElRT7GpzMhAkCRMueSzvKcwhHB2n/1UwPDYZKC1dfQS+jN6qxyccPLb tgc127BG+wOJro4MjTee9D6tuabUiVeCVNVCh13DbGOVCz4+QXZ2nE2bR2hLfDQA/qoNz10vaH+ Mwv3Spnfthcd6R1Cv+1XnOubRRPszU1Vj0cn0ZYsyMV+lEY97GnDomg+Xn/srLMj0gkzuRSCagB /I/Hh/WnX4h6F8O2aD7MUlaFwhiRNwQrmpBcSZ6s143KIA5W28904= X-Google-Smtp-Source: AIpwx48WEgXOjJ+zvfokxJSMD5awd6VlCpsnj1paNJ4NT4lTqBr8oXFS2lnXORffVjrpsR2sf9RtPKeCJgqtpgXi/8E= X-Received: by 2002:aca:cd8e:: with SMTP id d136-v6mr5662851oig.253.1522685687845; Mon, 02 Apr 2018 09:14:47 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:400b:0:0:0:0:0 with HTTP; Mon, 2 Apr 2018 09:14:27 -0700 (PDT) In-Reply-To: <87CFCF4F-24CB-49A1-B1A5-5440715662EB@heisenbergwoodworking.com> References: <87CFCF4F-24CB-49A1-B1A5-5440715662EB@heisenbergwoodworking.com> From: James Meickle Date: Mon, 2 Apr 2018 12:14:27 -0400 Message-ID: Subject: Re: RBAC Update To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000876e160568dfe354" --000000000000876e160568dfe354 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable To my mind, I would expect the MVP of per-DAG RBAC to include three settings: viewing DAG state, executing or modifying DAGs, and viewing tasks within the DAG (logs/code/details). For instance we would love to expose a view of the production dataload state to our engineers, without exposing production logs or allowing them to run/modify anything. Creating these mappings would be somewhat painful to manage, but that seems fine for an MVP. I would expect some future version to add grouping/tagging to DAGs and to users, to say e.g. "users in group X can see DAGs in group Y and modify DAGs in group Z." On Fri, Mar 30, 2018 at 1:26 PM, Brian Greene < brian@heisenbergwoodworking.com> wrote: > I=E2=80=99d think we=E2=80=99d have privilege =E2=80=98can_view=E2=80=99 = etc, and then a join table (priv) > <-> (dagid) <-> (user/group). Then it=E2=80=99s a simple query to get th= e perms > for a given dag (as you list In option 2 below). > > It also makes a =E2=80=9Csecure by default=E2=80=9D easy - a lack of entr= ies in that table > for a dag can mean only =E2=80=9Cadmin=E2=80=9D access or some such. > > Then any dag can have any combo of permissions for any combo of users. > Adding the groups option raises complexity around nesting, so maybe skip = it > for r1? > > My $.02 > > Brian > > Sent from a device with less than stellar autocorrect > > > On Mar 29, 2018, at 10:27 AM, Maxime Beauchemin < > maximebeauchemin@gmail.com> wrote: > > > > Hijacking the thread further here, any thoughts on how to breakdown per > DAG > > access? > > > > Tao & I are talking about introducing per-DAG permissions and one big > > question is whether we'll need to support different operation-types at = a > > per-DAG level, which changes the way we need to model the perms. > > > > First [simpler] option is to introduce one perm per DAG. If you have > access > > to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogeno= us > > rights on the DAGs you have access to. > > > > Second option is to have a breakdown per DAG. Meaning for each DAG we > > create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So > one > > user could have modify on some DAGs, view on others, and other DAGs wou= ld > > be invisible. This could be broken down further ({dag_id}_can_clear, ..= .) > > but it gets hard to manage. > > > > Thoughts? > > > > Max > > > >> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng wrote= : > >> > >> Great work Joy. This is awesome! I am interested in helping out the pe= r > dag > >> level access. Just created a ticket to check(AIRFLOW-2267). Let me > know if > >> you have any suggestions. I will share my proposal once I am ready. > >> > >>> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao wrote: > >>> > >>> Hey guys! > >>> > >>> The RBAC UI > has > >>> been merged to master. I'm looking forward to early adopters' feedbac= k > >> and > >>> bug reports. I also hope to have more folks helping out with the RBAC > UI, > >>> especially with introducing DAG-Level access control, which is a > feature > >>> that a lot of people have been asking. If you are interested in helpi= ng > >> out > >>> with this effort, let's talk more! > >>> > >>> This commit will be in the 1.10.0 release, and we are going to mainta= in > >>> both UIs simultaneously for a short period of time. Once RBAC UI is > >> stable > >>> and battle-tested, we will deprecate the old UI and eventually remove > it > >>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to > prevent > >>> two UIs from forking into separate paths, as that would become very > >>> difficult to maintain. > >>> > >>> Going forward while both UIs are up, if you are making a change to an= y > >>> files in airflow/www/ (old UI), where applicable, please also make th= e > >>> change to the airflow/www_rbac/ (new UI). If you rather not make > changes > >> in > >>> both UIs, it is recommended that you only make the changes to the RBA= C > >> UI, > >>> since that is the one we are maintaining in the long term. > >>> > >>> I'm excited that the RBAC UI will be able to bring additional securit= y > to > >>> Airflow, and with FAB framework in place we can look into leveraging = it > >> for > >>> a unified set of APIs used by both UI and CLI. > >>> > >>> Joy > >>> > >>> > >>> > >>>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao wrote: > >>>> > >>>> Hi folks, > >>>> > >>>> I have a PR > >> out > >>>> for the new UI. I've included instructions on how to test it out in > the > >>> PR > >>>> description. Looking forward to your feedbacks. > >>>> > >>>> Cheers, > >>>> Joy > >>>> > >>>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao wrote: > >>>>> > >>>>> Thanks for the background info. Would be really awesome for you to > >> have > >>>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB > >>>>> dependency pointing to my fork for the mean time. > >>>>> > >>>>> For folks who are interested in RBAC, I will be giving a talk/demo = at > >>> the Airflow > >>>>> Meet-Up > >>>>> >>> Meetup/events/244525050/> > >>>>> next Monday. Happy to chat afterwards about it as well :) > >>>>> > >>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin < > >>>>> maximebeauchemin@gmail.com> wrote: > >>>>> > >>>>>> A bit of related history here: > >>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399 > >>>>>> > >>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin < > >>>>>> maximebeauchemin@gmail.com> wrote: > >>>>>> > >>>>>>> Given I have merge rights on FAB I could probably do another roun= d > >> of > >>>>>>> review and get your PRs through. I would really like to get the > >> main > >>>>>>> maintainer's input on things that touch the core (composite-key > >>>>>> support) as > >>>>>>> he might have concerns/intuitions that we can't know about. > >>>>>>> > >>>>>>> I do not have Pypi access though so I cannot push new releases > >> out. I > >>>>>>> could ask for that. > >>>>>>> > >>>>>>> I've threatened to fork the project before, that's always an > >> option. > >>>>>> I've > >>>>>>> noticed his involvement is sporadic and comes in bursts. > >>>>>>> > >>>>>>> In the meantime, you can have the dependency in Airflow Webserver > >>>>>> pointing > >>>>>>> straight to your fork. > >>>>>>> > >>>>>>> Max > >>>>>>> > >>>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao wrote: > >>>>>>>> > >>>>>>>> I just created a new webserver instance if you haven't gotten a > >>>>>> chance to > >>>>>>>> fiddle around with the new web UI and the RBAC configurations > >>> (thanks > >>>>>>>> Maxime for getting started with this earlier!): > >>>>>>>> > >>>>>>>> http://104.209.38.171:8080/ > >>>>>>>> > >>>>>>>> Admin Account > >>>>>>>> username: admin > >>>>>>>> password: admin > >>>>>>>> > >>>>>>>> Read-Only Account > >>>>>>>> username: viewer > >>>>>>>> password: password > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao wrote= : > >>>>>>>>> > >>>>>>>>> Hi folks, > >>>>>>>>> > >>>>>>>>> Thanks for all the feedback regarding to the new Airflow > >> Webserver > >>>>>> UI > >>>>>>>>> ! I've been > >> actively > >>>>>>>>> addressing all the bugs that were raised on Github. So I want t= o > >>>>>> take > >>>>>>>> this > >>>>>>>>> opportunity to discuss two issues coming up: > >>>>>>>>> > >>>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continu= e > >>> to > >>>>>> stay > >>>>>>>>> unaddressed, RBAC is blocked from making further progress. If > >> this > >>>>>>>> continue > >>>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not > >>>>>>>> idealistic. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> - PR/631 >>> lask-AppBuilder/pull/631> > >>>>>>>> Binary > >>>>>>>>> column support (merged, unreleased) > >>>>>>>>> > >>>>>>>>> - PR/639 >>> lask-AppBuilder/pull/639> > >>>>>>>> Composite > >>>>>>>>> primary key support (unmerged) > >>>>>>>>> - PR/655 >>> lask-AppBuilder/pull/655> > >>>>>>>> Form > >>>>>>>>> prefill support (unmerged) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> The second issue is an open question about the next step of > >>> Airflow > >>>>>>>>> Webserver itself. Here are the 3 potential directions we could > >>>>>> take, and > >>>>>>>>> I've added my thought on each. > >>>>>>>>> > >>>>>>>>> 1. Permanently keep Airflow Webserver as a separated package > >> from > >>>>>>>> Airflow, > >>>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow > >>>>>>>> development > >>>>>>>>> on both UIs. > >>>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow= , > >>>>>> most > >>>>>>>>> contributors would prefer to maintain the official version that > >> is > >>>>>>>>> installed out-of-the-box. **Having a second UI outside of > >> Airflow > >>>>>> will > >>>>>>>>> make maintaining it very difficult, leading to an eventual deat= h > >>> of > >>>>>> the > >>>>>>>> new > >>>>>>>>> UI :(* > >>>>>>>>> > >>>>>>>>> 2. Permanently keep Airflow Webserver as a separated package > >> from > >>>>>>>> Airflow, > >>>>>>>>> but freeze all development on `www` and direct all future UI > >>>>>>>> development > >>>>>>>>> to Airflow Webserver, eventually removing `www` completely when > >>>>>> Airflow > >>>>>>>>> Webserver is stable. > >>>>>>>>> *I'm not a fan of this either. First of all, the views and > >> models > >>>>>> are > >>>>>>>>> tightly coupled in both old and new UI; until we have a > >>> full-fledged > >>>>>>>> REST > >>>>>>>>> API to build the UI (and cli) on top of it, separating them to = a > >>>>>>>> separate > >>>>>>>>> package now will potentially cause dependency issues and add > >>>>>>>> complication > >>>>>>>>> to our release cycle. **Secondly, **majority of Airflow users > >> run > >>>>>>>> Airflow > >>>>>>>>> with the UI; it's one of Airflow's best features. Separating UI > >>> out > >>>>>> of > >>>>>>>>> Airflow core will complicate setup and configuration, while > >> making > >>>>>>>> Airflow > >>>>>>>>> core less complete.* > >>>>>>>>> > >>>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze > >> all > >>>>>>>>> development on `www`, eventually removing `www` completely when > >>>>>> `www2` > >>>>>>>> is > >>>>>>>>> stable. > >>>>>>>>> *This makes the most sense to me. Airflow Webserver is develope= d > >>>>>> with > >>>>>>>> the > >>>>>>>>> goal of feature parity to the current UI, plus additional RBAC > >>>>>>>> capability, > >>>>>>>>> in hope to replace the old UI completely. Yes, this means there > >>>>>> will be > >>>>>>>> a > >>>>>>>>> short period of having to maintain two UIs, but once we freeze > >>>>>>>> development > >>>>>>>>> on www, it shouldn't be a concern for long.* > >>>>>>>>> > >>>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about > >>>>>> bringing > >>>>>>>>> RBAC to airflow and I hope it's something others will find > >> useful > >>> as > >>>>>>>> well! > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Joy > >>>>>>>>> > >>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao > >> wrote: > >>>>>>>>> > >>>>>>>>>> Thank you everyone for the active feedback so far, and thanks > >> for > >>>>>>>> setting > >>>>>>>>>> up the demo Maxime! > >>>>>>>>>> > >>>>>>>>>> Going to work on pruning through the issues in the upcoming > >> days. > >>>>>>>>>> > >>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message > >> so I > >>>>>> can > >>>>>>>>>> look into it? Otherwise I'll wait until it's down again =3DP > >>>>>>>>>> > >>>>>>>>>> Cheers, > >>>>>>>>>> > >>>>>>>>>> Joy > >>>>>>>>>> > >>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin < > >>>>>>>>>> maximebeauchemin@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> I just restarted it, not sure how long it will take to get in > >> a > >>>>>> bad > >>>>>>>> state > >>>>>>>>>>> again... > >>>>>>>>>>> > >>>>>>>>>>> Max > >>>>>>>>>>> > >>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko > >>>>>>>> >>>>>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Good morning, > >>>>>>>>>>>> > >>>>>>>>>>>> The demo provided by Max is down, it throws a > >>>>>> SQLAlchemyexception > >>>>>>>> :'( > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers, Fokko > >>>>>>>>>>>> > >>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini < > >>>>>> criccomini@apache.org>: > >>>>>>>>>>>> > >>>>>>>>>>>>> @bolke, open issues on the Github repo, please. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin < > >>>>>>>> bdbruin@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Chris, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Do you want us to report bugs somewhere (I have > >>> encountered > >>>>>> a > >>>>>>>>>>> few)? Or > >>>>>>>>>>>>>> just generic user experiences posted here? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers > >>>>>>>>>>>>>> Bolke > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini < > >>>>>>>> criccomini@apache.org > >>>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I know the weekend is coming up, and for those of us > >> in > >>>>>> the > >>>>>>>> US, > >>>>>>>>>>> next > >>>>>>>>>>>>> week > >>>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get > >> some > >>>>>>>> feedback > >>>>>>>>>>> from > >>>>>>>>>>>>>>> everyone on this. The goal would ideally to be to > >>>>>> converge on > >>>>>>>>>>> this > >>>>>>>>>>>> and > >>>>>>>>>>>>>>> eventually replace the existing Airflow UI with this > >>> one. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao < > >>> joyg@wepay.com> > >>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi guys. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin > >> to > >>>>>>>>>>>>> Flask-AppBuilder > >>>>>>>>>>>>>>>> for RBAC > >>>>>>>>>>>>>>>> >> confluence/display/AIRFLOW/ > >>>>>>>>>>>>>> Airflow+RBAC+proposal > >>>>>>>>>>>>>>>>> , > >>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl > >>>>>> ow-webserver. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> It's still a work-in-progress, but most features you > >>> see > >>>>>> in > >>>>>>>> the > >>>>>>>>>>>>>> webserver > >>>>>>>>>>>>>>>> UI today is available there. For those who are > >>>>>> interested in > >>>>>>>>>>> RBAC, > >>>>>>>>>>>> I'd > >>>>>>>>>>>>>> love > >>>>>>>>>>>>>>>> to get some early feedback in terms of the following: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions) > >>>>>>>>>>>>>>>> - Setup issues > >>>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e. > >> LDAP, > >>>>>> AD, > >>>>>>>>>>> OAuth, > >>>>>>>>>>>>> etc.) > >>>>>>>>>>>>>>>> - Any other thoughts/concerns > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks a lot! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>> Joy > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > --000000000000876e160568dfe354--