cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koushik Das <>
Subject Command sequence logic in agent code
Date Wed, 23 Oct 2013 12:56:37 GMT
I was looking at the command sequencing logic in the agent code.

Each agent maintains a sequence that gets initialised based on following logic

    private static final Random s_rand = new Random(System.currentTimeMillis());
    _nextSequence = s_rand.nextInt(Short.MAX_VALUE) << 48;

For every command that gets processed by the agent the sequence is incremented by 1. If commands
are to be executed in sequence then they are queued up based on this sequence

    protected synchronized void addRequest(Request req) {
        int index = findRequest(req);
        assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
        _requests.add(-index - 1, req);

The above works fine in case of a single MS scenario. In case of a clustered MS setup things
change slightly.

The command can originate at any MS and based on the ownership of the agent, it gets forwarded
to the correct MS which then handles the command. Now command sequences are local to individual
agents in MS. In this case the originating MS agent tags the request with a sequence. This
gets forwarded to the owning MS and based on if 'executedInSequence' flag is set, gets added
to the list based on the sequence number. Now here lies the problem, commands are not inserted
in the order in which they arrive but based on the sequence number. In case of a forwarded
command the sequence is different from the local sequence. If the starting sequence of forwarded
commands is much less than that of the locally generated commands then there is a possibility
of local commands getting starved if there is a steady arrival of forwarded commands. Similarly
it can also happen the other way round. Also if the the starting sequence for a agent in local
and peer MS is not spread far apart then there may be overlaps and a new request will override
the old one.

Not sure if anyone encountered any issues due to this. The correct way looks like to implement
the queue model rather than doing a add based on the above code.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message