hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hank Cohen <hank.co...@altior.com>
Subject RE: HADOOP in Production
Date Wed, 03 Oct 2012 03:01:26 GMT
Good points.

I'm not trying to be exhaustive in the discussion of real time systems.
My only intent was to point out the difference between real time and fast response.
There are lots of real time requirements that do not require a particularly fast response
but the response needs to be on time.  Timing errors can introduce noise at a pretty astounding
Real fast often uses the same technology as real time but the consequences of missing a deadline
are different.

As for the high velocity traders, I'm an advocate of the Robin Hood Tax.
A little friction might help the world.

Hank Cohen

From: Ted Dunning [mailto:tdunning@maprtech.com]
Sent: Tuesday, October 02, 2012 4:13 PM
To: user@hadoop.apache.org
Subject: Re: HADOOP in Production

On Tue, Oct 2, 2012 at 7:05 PM, Hank Cohen <hank.cohen@altior.com<mailto:hank.cohen@altior.com>>
There is an important difference between real time and real fast

Real time means that system response must meet a fixed schedule.
Real fast just means sooner is better.

Good thought, but real-time can also include a fixed schedule and a specified list of exceptional
conditions which would prevent meeting the schedule.

It may also include a fixed schedule that must be met some fraction of the time (usually very
near 100% of the time).

Without providing exceptions, you basically force the designer to lie about how reliable their
system is.

Real time systems always have hard schedules. The schedule could be in microseconds to control
a laser for making masks for semiconductor manufacturing, milliseconds to control the ignition
in your car or flight controls on an F-22 or  seconds for even slower moving processes.  In
real time system missing the schedule can mean that very bad things happen: planes fall from
the sky, your laser printer fries it's imaging drum, factories explode etc.

It can mean that.  But if you specify the exceptional situations you can specifically mitigate
for them.

Most transaction processing is happy with real fast
The folks doing high velocity trading are pretty close to real time but they probably will
be happy with real fast.
If real fast systems miss a schedule then someone loses money.

Yeah.  And if you talk to these guys, they know the difference and ask for real-time.

The reason that RTOS type operating systems are popular for real time applications is that
they don't allow operations to spend indeterminate amounts of time in uninterruptable states.
 Java will never qualify as a real time system because it has garbage collection and garbage
collection can lock up a system for an indefinite amount of time while it goes through marking
and counting.

You are behind the times on a few counts.

- Java's collectors don't "count".

- Java can be real-time:


- Garbage collection can be deterministic and real-time:


View raw message