incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Xu <Xuefei...@citrix.com>
Subject RE: [REVIEW/PROPOSAL/DISCUSS] API refactoring story: goal and notes
Date Thu, 20 Dec 2012 06:52:54 GMT
Joining table still has cost. Each list API might become costly if the setup is big.
Why not add foreign UUID in reference table?
Do you see any issue with it?

Anthony

> -----Original Message-----
> From: Min Chen [mailto:min.chen@citrix.com]
> Sent: Wednesday, December 19, 2012 10:47 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [REVIEW/PROPOSAL/DISCUSS] API refactoring story: goal and
> notes
> 
> In that case, the extra DB query to fetch UUID is not avoidable. As I
> mentioned earlier, for some costly list APIs, we will create DB view to
> join these tables so that we can fetch those UUIDs in sort of batch
> mode
> while searching for objects to alleviate this effect.
> 
> Thanks
> -min
> 
> 
> 
> On 12/19/12 10:41 PM, "Anthony Xu" <Xuefei.Xu@citrix.com> wrote:
> 
> >If foreign key object is not needed, only foreign UUID is needed,
> >does that mean extra DB view query is added for each listed item?
> >Will this impact costly list APIs?
> >
> >
> >Thanks,
> >Anthony
> >
> >> -----Original Message-----
> >> From: Min Chen [mailto:min.chen@citrix.com]
> >> Sent: Wednesday, December 19, 2012 10:30 PM
> >> To: cloudstack-dev@incubator.apache.org
> >> Subject: Re: [REVIEW/PROPOSAL/DISCUSS] API refactoring story: goal
> and
> >> notes
> >>
> >> We will replace all IdentityProxy fields from all response classes
> with
> >> simple String to represent UUID directly. If you look through API
> code,
> >> most times when we generate response from queried resources, we
> already
> >> query foreign key object once to fill in those related fields, like
> >> accountName, domainName, etc. By replacing IdentityProxy fields, we
> can
> >> directly save those already queried uuid in response instead of
> >> currently
> >> querying them again during response serialization phase. For those
> more
> >> costly list APIs, for example, listVms, we will create DB views to
> join
> >> those uuids in the view to get them through one db view query.
> >>
> >> Thanks
> >> -min
> >>
> >> On 12/19/12 10:20 PM, "Anthony Xu" <Xuefei.Xu@citrix.com> wrote:
> >>
> >> >Thanks for explanation,
> >> >
> >> >How does CS handle ID->UUID translation in response object?
> >> >
> >> >Anthony
> >> >
> >> >> -----Original Message-----
> >> >> From: Min Chen [mailto:min.chen@citrix.com]
> >> >> Sent: Wednesday, December 19, 2012 10:13 PM
> >> >> To: cloudstack-dev@incubator.apache.org
> >> >> Subject: Re: [REVIEW/PROPOSAL/DISCUSS] API refactoring story:
> goal
> >> and
> >> >> notes
> >> >>
> >> >> Here Fang is talking about UUID translation to DB_ID in
> constructing
> >> >> our
> >> >> Command class (ApiDispatcher.setupParameters), not in generating
> >> >> response
> >> >> object, so it will not impact list api performance.
> >> >>
> >> >> Thanks
> >> >> -min
> >> >>
> >> >> On 12/19/12 10:07 PM, "Anthony Xu" <Xuefei.Xu@citrix.com> wrote:
> >> >>
> >> >> >>2.	The API layer provides a translation from the UUID to the
> >> >> internal
> >> >> >>DB_ID to the DB entity, but this translation is done internally.
> >> >> Outside
> >> >> >>users will never see DB_ID. Before the response is sending
back
> by
> >> CS,
> >> >> >>API layer replaces the internal DB_ID with the UUID.
> >> >> >
> >> >> >I thought ID->UUID translation in API layer is a workaround,
> since
> >> it
> >> >> >might cause list API performance issue.
> >> >> >Any plan to fix it completely in backend?
> >> >> >
> >> >> >
> >> >> >Anthony
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: Fang Wang [mailto:fang.wang@citrix.com]
> >> >> >> Sent: Wednesday, December 19, 2012 10:02 PM
> >> >> >> To: cloudstack-dev@incubator.apache.org
> >> >> >> Subject: RE: [REVIEW/PROPOSAL/DISCUSS] API refactoring story:
> >> goal
> >> >> and
> >> >> >> notes
> >> >> >>
> >> >> >> Prachi did an excellent FS document for this, and I added
the
> >> high
> >> >> >> level goals section
> >> >> >> To her FS link at wiki:
> >> >> >>
> >> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+API+r
> >> >> >> efactoring
> >> >> >>
> >> >> >> Lots of implementation details are covered in the FS. We'll
> >> update
> >> >> it
> >> >> >> along the way.
> >> >> >> Thanks,
> >> >> >> -Fang
> >> >> >>
> >> >> >> -----Original Message-----
> >> >> >> From: Fang Wang [mailto:fang.wang@citrix.com]
> >> >> >> Sent: Wednesday, December 19, 2012 9:46 PM
> >> >> >> To: cloudstack-dev@incubator.apache.org
> >> >> >> Subject: [REVIEW/PROPOSAL] API refactoring story: goal and
> notes
> >> >> >>
> >> >> >> Hi all,
> >> >> >> After answering questions about why/what we are doing to API
> >> >> >> refactoring, I'll add this to the FS document.
> >> >> >> Probably lots of people are not clear what is our motivation
> and
> >> >> what
> >> >> >> we want to achieve here.
> >> >> >> (Will add this to the wiki page. )
> >> >> >>
> >> >> >> The goal of the API refactoring
> >> >> >> To make the system more modular and dynamic, API refactoring
> is
> >> part
> >> >> of
> >> >> >> the 4.1 new architecture efforts.
> >> >> >> Another goal is to allow developers and partners to quickly
> add
> >> >> plugins
> >> >> >> and new service modules and access control, making cloudstack
> >> more
> >> >> >> adaptive.
> >> >> >> I am new on this as well In the beginning, I do not know too
> much
> >> >> about
> >> >> >> it. Now after working on API refactoring, I got a better
> >> >> understanding
> >> >> >> of the existing problems and our solutions. The solution is
> not
> >> the
> >> >> >> final perfect solution, for sure, feedback and suggestion
is
> >> always
> >> >> >> welcome to make it a much better and adaptive product.
> >> >> >>
> >> >> >> Current problem:
> >> >> >> 1.	Security and access check is lying around in different
> >> layers.
> >> >> >> For example, in apiSevlet we check the web access
> >> username/password
> >> >> >> credentials, apiServer checks the command existence, and
> several
> >> >> other
> >> >> >> checks, then the DB access check mainly is done at the service
> >> layer.
> >> >> >> So it is hard for developers and system admin to follow all
> the
> >> >> access
> >> >> >> and validations, to make sure all the checks are done
> correctly.
> >> To
> >> >> do
> >> >> >> it, developers need to be familiar with the various parts
of
> the
> >> CS
> >> >> >> code.
> >> >> >>
> >> >> >> 2.	Static: Majority of the DB access check and security
> >> validation
> >> >> >> is tightly coupled with the lower layer service class. This
> not
> >> only
> >> >> >> made the code hard to follow, also inherently made the policy
> >> static.
> >> >> >> This makes it impossible for system admin to apply a different
> >> >> security
> >> >> >> policy adaptively. If one wants to adopt a different policy,
> not
> >> >> only
> >> >> >> he needs to understand code scattered around, he also needs
to
> >> >> rebuild
> >> >> >> the CS after any changes.  Any security role modification
is
> not
> >> >> >> dynamic.
> >> >> >>
> >> >> >> 3.	Performance implication: Access check done at the lower
> >> service
> >> >> >> layer makes the error code path long.
> >> >> >>
> >> >> >> 4.	Docs generated, CLI and APIs are loosely coupled.
> >> >> >>
> >> >> >> 5.	Over the wire(OTW) entity is not well defined. For example,
> >> to
> >> >> >> listVMsCmd, it involves multiple DB access, with the large
> amount
> >> of
> >> >> >> data showing NICs, Vols, Secondary storage etc, the command
> can
> >> take
> >> >> >> quite a while.
> >> >> >>
> >> >> >> 6.	Admin and user have basically the same end-point access.
> >> >> >>
> >> >> >>        The goal of the API refactoring is aiming to tackle
> these
> >> >> >> problems:
> >> >> >>
> >> >> >> 1.	We would pull security checks, DB access checks, any
> >> related
> >> >> >> checks up from service layer/orchestration engine to the API
> >> layer
> >> >> as
> >> >> >> much as possible. This makes the necessary checking done more
> >> >> >> centralized and easy to follow. Conceptually the cloud
> >> orchestration
> >> >> >> engine layer handles the orchestration, the security check
and
> >> the
> >> >> >> access check should have done before reaching this layer.
This
> >> has
> >> >> >> performance benefits, since checks are done earlier instead
of
> >> >> reaching
> >> >> >> deep in the code path; this also has the benefits of a clear
> >> >> >> architecture. The API layer does the necessary access check
> and
> >> role
> >> >> >> based authentication, makes it easier and dynamic for future
> >> policy
> >> >> >> change. New policy can be added easily and dynamically as
a
> >> plugin
> >> >> to
> >> >> >> the system.
> >> >> >>
> >> >> >> 2.	The ACL and security checks also are written as adapter
> >> plugins,
> >> >> >> hence make them dynamic. Users and developers and easily adapt
> >> new
> >> >> >> policy if needed. This made the code more modular and more
> >> adaptive.
> >> >> >>
> >> >> >> 3.	Help improves performance: Instead of finding access error
> >> layer
> >> >> >> into the service layer, by doing 1, we would do possible
> checks
> >> >> early
> >> >> >> in the code path, which helps stop the wrong access earlier
in
> >> the
> >> >> code
> >> >> >> path.
> >> >> >>
> >> >> >> 4.	API layer is more tightly coupled with Doc generation,
and
> >> CLI.
> >> >> >> Related commands are grouped together, and the new @Doc
> >> annotation
> >> >> will
> >> >> >> help show the related commands in document.
> >> >> >>
> >> >> >> 5.	We define new view objects as response objects, avoiding
> >> big DB
> >> >> >> joins at run time.
> >> >> >>
> >> >> >> 6.	Separate the admin and user APIs. This is for developers
to
> >> >> >> understand the code, which should be accessible by users,
> which
> >> >> should
> >> >> >> only be handled by Admin.  Hence developers will have better
> >> grasp
> >> >> of
> >> >> >> the role and pay attention to the new code added. It also
> helps
> >> the
> >> >> >> document generated.
> >> >> >>
> >> >> >>        Notes:
> >> >> >> 1.	For end users, the new APIs after refactoring looks pretty
> >> much
> >> >> >> the same. One big change is the ID, we will always use UUID
in
> >> the
> >> >> over
> >> >> >> the wire APIs. The UUID can be created by Cloudstack, or can
> be
> >> >> >> provided by users (we call it Xid - external ID). Every UUID
> >> should
> >> >> be
> >> >> >> unique in the cloudstack system.
> >> >> >>
> >> >> >> 2.	The API layer provides a translation from the UUID to the
> >> >> >> internal DB_ID to the DB entity, but this translation is done
> >> >> >> internally. Outside users will never see DB_ID. Before the
> >> response
> >> >> is
> >> >> >> sending back by CS,  API layer replaces the internal DB_ID
> with
> >> the
> >> >> >> UUID.
> >> >> >>
> >> >> >> 3.	In original FS document, the annotation of entityType in
> >> >> >> @Parameter points to a resource class, this is replaced by
a
> >> >> response
> >> >> >> class. So entityType points to a response object, and the
> >> response
> >> >> >> class has a one-to-one mapping from the response to the
> physical
> >> >> entity
> >> >> >> itself. This translation work is done by the API layer and
the
> >> >> >> entityManagerImpl.
> >> >> >>
> >> >> >> 4.	The packages for the new API commands are all moved from
> >> the
> >> >> >> current com.cloud.api.commands to new location:
> >> >> >> org.apache.cloudstack.api.commands.user.[group name]
> >> >> >> org.apache.cloudstack.api.commands.admin.[group_name]
> >> >> >>
> >> >> >> The responses are also moved to new location at
> >> >> >> org.apache.cloudstack.api.response
> >> >> >>
> >> >> >> More implementation details can refer to the FS document.
We
> will
> >> >> also
> >> >> >> update the document along the way. The code is branch from
> master
> >> at
> >> >> >> api_refactoring.  Since the change is not minimum, we would
> like
> >> the
> >> >> >> community to know and give feedback.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> -Fang
> >> >> >>
> >> >> >>
> >> >
> >


Mime
View raw message