heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Fischer <j...@joshfischer.io>
Subject Proposing Changes To ECO
Date Sun, 21 Jan 2018 14:21:29 GMT
All,

While working with Karthik, we have discovered that the way ECO handles
topology configuration will not work for all configuration types.  To be
specific, setting individual component's ram will not work.  We will also
have to keep in mind container size that contains the components.  My
proposal is this:

Create a standardized way to allow for the  configuring of  component ram
size in the "config" section of the  eco yaml file.  This would be a list
of key value pairs that mapped the "id" of a component to an allocated ram
size in MB.  An example is below:

config:
  topology.workers: 2
  topology.component.rammap:
    - "some-id": 1234
    - "other-id": 6789

However the  above implementation may be unclear when it comes to
understanding what unit of measurement is implicitly specified and/or
expected.  Or we could do something like below.

config:
  topology.workers: 2
  topology.component.rammap:
    - spec:
       id: "component-1"
       size: 1234
       type: MB  // Megabytes
    - spec:
       id: "component-2"
       size: 6789
       type: GB // GigaBytes
    - spec:
       id: "component-3"
       size: 123456789
       type: B // Bytes


If a mapping is not specified for a component, we can just assume Heron's
defaults.  We could then dynamically calculate the container size based off
of the number components and their corresponding allocated resources for
simplicity of use for the user, but still allow them to specify a custom
set of resources to a container like below

topology.container.disk: 1234
topology.container.ram: 3456
topology.container.cpu: 2


It may be best if I reused the ByteAmount object to calculate resource size
to remain consistent with the other Heron APIs.  Any concerns or
improvements to this approach I am missing?

Please Advise,

Josh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message