From user-return-1783-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 3 03:02:07 2012 Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E693E90F8 for ; Wed, 3 Oct 2012 03:02:07 +0000 (UTC) Received: (qmail 4145 invoked by uid 500); 3 Oct 2012 03:02:03 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 3790 invoked by uid 500); 3 Oct 2012 03:02:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 3758 invoked by uid 99); 3 Oct 2012 03:02:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Oct 2012 03:02:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hank.cohen@altior.com designates 216.32.180.185 as permitted sender) Received: from [216.32.180.185] (HELO co1outboundpool.messaging.microsoft.com) (216.32.180.185) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Oct 2012 03:01:53 +0000 Received: from mail73-co1-R.bigfish.com (10.243.78.246) by CO1EHSOBE002.bigfish.com (10.243.66.65) with Microsoft SMTP Server id 14.1.225.23; Wed, 3 Oct 2012 03:01:31 +0000 Received: from mail73-co1 (localhost [127.0.0.1]) by mail73-co1-R.bigfish.com (Postfix) with ESMTP id 6B48B4800B1 for ; Wed, 3 Oct 2012 03:01:31 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.236.213;KIP:(null);UIP:(null);IPV:NLI;H:BY2PRD0711HT005.namprd07.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -22 X-BigFish: VPS-22(zz98dI9371Ic85fhd799hzz1202h1d1ah1d2ahzz17326ah8275bh8275dh7fcILz2fh2a8h668h839hd25hf0ah107ah1288h12a5h12bdh137ah1441h1155h) Received-SPF: pass (mail73-co1: domain of altior.com designates 157.56.236.213 as permitted sender) client-ip=157.56.236.213; envelope-from=hank.cohen@altior.com; helo=BY2PRD0711HT005.namprd07.prod.outlook.com ;.outlook.com ; Received: from mail73-co1 (localhost.localdomain [127.0.0.1]) by mail73-co1 (MessageSwitch) id 1349233288339403_13382; Wed, 3 Oct 2012 03:01:28 +0000 (UTC) Received: from CO1EHSMHS009.bigfish.com (unknown [10.243.78.242]) by mail73-co1.bigfish.com (Postfix) with ESMTP id 50F25580042 for ; Wed, 3 Oct 2012 03:01:28 +0000 (UTC) Received: from BY2PRD0711HT005.namprd07.prod.outlook.com (157.56.236.213) by CO1EHSMHS009.bigfish.com (10.243.66.19) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 3 Oct 2012 03:01:27 +0000 Received: from BY2PRD0711MB428.namprd07.prod.outlook.com ([169.254.8.226]) by BY2PRD0711HT005.namprd07.prod.outlook.com ([10.255.88.168]) with mapi id 14.16.0207.007; Wed, 3 Oct 2012 03:01:27 +0000 From: Hank Cohen To: "user@hadoop.apache.org" Subject: RE: HADOOP in Production Thread-Topic: HADOOP in Production Thread-Index: AQHNn9ntbY5vTITTi0Svxrjbi11gXJel59aAgAAEkQCAAF4cEIAAXLiAgAA76WA= Date: Wed, 3 Oct 2012 03:01:26 +0000 Message-ID: <029C75A3482BE64594E21FC09BB19F411385F608@BY2PRD0711MB428.namprd07.prod.outlook.com> References: <029C75A3482BE64594E21FC09BB19F411385F342@BY2PRD0711MB428.namprd07.prod.outlook.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [50.192.254.25] Content-Type: multipart/alternative; boundary="_000_029C75A3482BE64594E21FC09BB19F411385F608BY2PRD0711MB428_" MIME-Version: 1.0 X-OriginatorOrg: altior.com --_000_029C75A3482BE64594E21FC09BB19F411385F608BY2PRD0711MB428_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Good points. I'm not trying to be exhaustive in the discussion of real time systems. My only intent was to point out the difference between real time and fast r= esponse. There are lots of real time requirements that do not require a particularly= fast response but the response needs to be on time. Timing errors can int= roduce noise at a pretty astounding rate. Real fast often uses the same technology as real time but the consequences = of missing a deadline are different. As for the high velocity traders, I'm an advocate of the Robin Hood Tax. A little friction might help the world. Hank Cohen From: Ted Dunning [mailto:tdunning@maprtech.com] Sent: Tuesday, October 02, 2012 4:13 PM To: user@hadoop.apache.org Subject: Re: HADOOP in Production On Tue, Oct 2, 2012 at 7:05 PM, Hank Cohen > wrote: There is an important difference between real time and real fast Real time means that system response must meet a fixed schedule. Real fast just means sooner is better. Good thought, but real-time can also include a fixed schedule and a specifi= ed list of exceptional conditions which would prevent meeting the schedule. It may also include a fixed schedule that must be met some fraction of the = time (usually very near 100% of the time). Without providing exceptions, you basically force the designer to lie about= how reliable their system is. Real time systems always have hard schedules. The schedule could be in micr= oseconds to control a laser for making masks for semiconductor manufacturin= g, milliseconds to control the ignition in your car or flight controls on a= n F-22 or seconds for even slower moving processes. In real time system m= issing the schedule can mean that very bad things happen: planes fall from = the sky, your laser printer fries it's imaging drum, factories explode etc. It can mean that. But if you specify the exceptional situations you can sp= ecifically mitigate for them. Most transaction processing is happy with real fast The folks doing high velocity trading are pretty close to real time but the= y probably will be happy with real fast. If real fast systems miss a schedule then someone loses money. Yeah. And if you talk to these guys, they know the difference and ask for = real-time. The reason that RTOS type operating systems are popular for real time appli= cations is that they don't allow operations to spend indeterminate amounts = of time in uninterruptable states. Java will never qualify as a real time = system because it has garbage collection and garbage collection can lock up= a system for an indefinite amount of time while it goes through marking an= d counting. You are behind the times on a few counts. - Java's collectors don't "count". - Java can be real-time: http://rtjava.blogspot.co.uk/2009/07/real-time-java-vms.html - Garbage collection can be deterministic and real-time: http://www.ibm.com/developerworks/java/library/j-rtj4/index.html http://docs.oracle.com/cd/E13221_01/wlrt/docs30/intro_wlrt/tuning.html --_000_029C75A3482BE64594E21FC09BB19F411385F608BY2PRD0711MB428_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Good points. 

 <= /p>

I’m not trying to b= e exhaustive in the discussion of real time systems. 

My only intent was to poi= nt out the difference between real time and fast response.

There are lots of real ti= me requirements that do not require a particularly fast response but the re= sponse needs to be on time.  Timing errors can introduce noise at a pretty astounding rate.

Real fast often uses the = same technology as real time but the consequences of missing a deadline are= different.

 <= /p>

As for the high velocity = traders, I’m an advocate of the Robin Hood Tax.

A little friction might h= elp the world.

 <= /p>

Hank Cohen

 <= /p>

From: Ted Dunn= ing [mailto:tdunning@maprtech.com]
Sent: Tuesday, October 02, 2012 4:13 PM
To: user@hadoop.apache.org
Subject: Re: HADOOP in Production

 

 

On Tue, Oct 2, 2012 at 7:05 PM, Hank Cohen <hank.cohen@altior.com= > wrote:

There is an important difference between real time a= nd real fast

Real time means that system response must meet a fixed schedule.
Real fast just means sooner is better.

 

Good thought, but real-time can also include a fixed= schedule and a specified list of exceptional conditions which would preven= t meeting the schedule.

 

It may also include a fixed schedule that must be me= t some fraction of the time (usually very near 100% of the time).

 

Without providing exceptions, you basically force th= e designer to lie about how reliable their system is.  

 

Real time systems always have hard schedules. The sc= hedule could be in microseconds to control a laser for making masks for sem= iconductor manufacturing, milliseconds to control the ignition in your car = or flight controls on an F-22 or  seconds for even slower moving processes.  In real time system missing the sc= hedule can mean that very bad things happen: planes fall from the sky, your= laser printer fries it's imaging drum, factories explode etc.

 

It can mean that.  But if you specify the excep= tional situations you can specifically mitigate for them.

 

Most transaction processing is happy with real fast<= br> The folks doing high velocity trading are pretty close to real time but the= y probably will be happy with real fast.
If real fast systems miss a schedule then someone loses money.

 

Yeah.  And if you talk to these guys, they know= the difference and ask for real-time.

 

 

The reason that RTOS type operating systems are popu= lar for real time applications is that they don't allow operations to spend= indeterminate amounts of time in uninterruptable states.  Java will n= ever qualify as a real time system because it has garbage collection and garbage collection can lock up a system for = an indefinite amount of time while it goes through marking and counting.

 

You are behind the times on a few counts.=

 

- Java's collectors don't "count".

 

- Java can be real-time:

 

 

- Garbage collection can be deterministic and real-t= ime:

 

 

--_000_029C75A3482BE64594E21FC09BB19F411385F608BY2PRD0711MB428_--