From user-return-62510-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Mon Oct 15 17:34:20 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 427E3180647 for ; Mon, 15 Oct 2018 17:34:19 +0200 (CEST) Received: (qmail 9656 invoked by uid 500); 15 Oct 2018 15:34:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 9646 invoked by uid 99); 15 Oct 2018 15:34:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Oct 2018 15:34:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 02EBEC578B for ; Mon, 15 Oct 2018 15:34:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.998 X-Spam-Level: * X-Spam-Status: No, score=1.998 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=epsilon.onmicrosoft.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id l9YoJ6ExFssM for ; Mon, 15 Oct 2018 15:34:14 +0000 (UTC) Received: from mail-ua1-f99.google.com (mail-ua1-f99.google.com [209.85.222.99]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D91EE5F19C for ; Mon, 15 Oct 2018 15:34:13 +0000 (UTC) Received: by mail-ua1-f99.google.com with SMTP id c10so2159085uao.12 for ; Mon, 15 Oct 2018 08:34:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:thread-topic :thread-index:date:message-id:references:in-reply-to:accept-language :content-language:spamdiagnosticoutput:spamdiagnosticmetadata :mime-version; bh=SLNuN/22R87GKc1O7UjbNzZkc6MsmpdGxTl/F4P1CS4=; b=IbrsydnpadaYQdZ2bu2LZ6o5ZpzZunyYgTm/wBMIomSh8Pg1+517ydKGtcJdk4Z6LP RpoCvRRq9bAW63+ZudiYloK0bcoB6eMMzVoZdw86OMJXxVCQK8h6dKIF6wNjqlNzaYKM PzF1E3/0Bvtl3HSEttyPWV0sywpsP+VkoD2SU27TK8ldknXQGuNdB2H3agIgnQJJYvb0 BjtwrnPIvfW30TDzB6HsuEw+/hYfxFyVLaOBPkUW1OMSf0A0MeNeWKd58WGcy0tBOT80 KJ9Veweslzc9BReDdZSn3kH/r9Qh3dfrCHGAefzthnETKAiqOb91DNTjcS6o1zRAR4MM xZvw== X-Gm-Message-State: ABuFfojlkEUAbqPy+puB9/E/L64Dn9zKvM8EdOJatFYKYG1GgRzQBGTo StA91ep5+5BVqz3T+PpVwCI5MGZRa7JRBGU3ZkN8S3QHh2tXmA== X-Google-Smtp-Source: ACcGV63C0ftNQuxwOfVJ3/4TmPNAmqkW7eaoYm5edLYohAZI/w4FOF8QhSkOxRZwcUb21F/192Sed9YeCzxX X-Received: by 2002:ab0:598b:: with SMTP id g11mr7427123uad.131.1539617647347; Mon, 15 Oct 2018 08:34:07 -0700 (PDT) Received: from PE1CPRELAY1.epsilon.com ([159.127.85.31]) by smtp-relay.gmail.com with ESMTPS id t9-v6sm737263vkt.3.2018.10.15.08.34.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Oct 2018 08:34:07 -0700 (PDT) X-Relaying-Domain: epsilon.com Received: from unknown (HELO pc1wepsiexdag07.corp.epsilon.com) ([10.212.6.26]) by PE1CPRELAY1.epsilon.com with ESMTP/TLS/AES256-SHA; 15 Oct 2018 10:34:05 -0500 Received: from pc1wepsiexdag07.corp.epsilon.com (10.212.6.26) by pc1wepsiexdag07.corp.epsilon.com (10.212.6.26) with Microsoft SMTP Server (TLS) id 15.0.1293.2; Mon, 15 Oct 2018 10:34:05 -0500 Received: from NAM05-CO1-obe.outbound.protection.outlook.com (10.212.6.126) by pc1wepsiexdag07.corp.epsilon.com (10.212.6.26) with Microsoft SMTP Server (TLS) id 15.0.1293.2 via Frontend Transport; Mon, 15 Oct 2018 10:34:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Epsilon.onmicrosoft.com; s=selector1-epsilon-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SLNuN/22R87GKc1O7UjbNzZkc6MsmpdGxTl/F4P1CS4=; b=AOtt1Nz02hM1T0FFMO/2cFmEPzs67wzkneZDs0sZVcfL492viILIaaDnm492zo2YYuYrLWaCDcia+tm8mdRyljzvl4v1+zER/cvGZRumjSPn8Qh6LLmlyUABWJfXRLSvoQcrw+1rmLXtK/gg9towsW5xcCy6gniNDsXrWBG/Ixs= Received: from CY4PR02MB3350.namprd02.prod.outlook.com (10.165.89.145) by CY4PR02MB2279.namprd02.prod.outlook.com (10.169.181.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1228.25; Mon, 15 Oct 2018 15:34:04 +0000 Received: from CY4PR02MB3350.namprd02.prod.outlook.com ([fe80::c12c:bbd8:a293:f84c]) by CY4PR02MB3350.namprd02.prod.outlook.com ([fe80::c12c:bbd8:a293:f84c%5]) with mapi id 15.20.1228.020; Mon, 15 Oct 2018 15:34:04 +0000 From: "Naik, Ninad" To: "user@cassandra.apache.org" Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) Thread-Topic: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) Thread-Index: AQHUYm0elD2+mLElLUyySMjUUgiy5aUcRQuAgAEor/iAAZuNgIABaoZv Date: Mon, 15 Oct 2018 15:34:03 +0000 Message-ID: References: <2C851B7C-8E53-4413-AB0E-249682051A2E@yahoo.com> ,<038DA035-E09D-4704-B659-EABCADE822D6@gmail.com> In-Reply-To: <038DA035-E09D-4704-B659-EABCADE822D6@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [159.127.199.1] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY4PR02MB2279;6:aRqjo7uUzKEsCwYEEuV2Nr2mzW4kk6a4Jy5YiiWoDFCz/Rb1pRia4+dif+OonTQOmLlrPKfQQchegnlyEQCXmv+i/ulA5+kaE19xJrUKFcUGVhltY1tP4/Tlyr9u6plBpsXPf9a24c5MJm08QOCVYF8aWs3zfv4NiSA21QT8W86ytjyU5942VdU4pj/5LwkJmaL8elnXfoDguiH4oUvcAOiSfNF6Cjl7FWvu9g3hWQciQWA+8ux0Hd0qmHgh5un9ZwwZYgK3t92UW8sh4RHx7yDrlsQswtF8AXXhVRhfL2qmBKTbyxQQgrxpTdcB6MYATHZACHmblKWfSGbhH8Or1JlQzXIWltQqn0RZnYK31t+JX9AoU/9LI55hcaMx1PIlvyEXv+mQD853IqmtXuz0IG42drz3rNa1aKW1+1bvKWLFTaf8HlGGP0s6sMBJboUPTUMBDdpk6udRVC152yN4Gg==;5:0j01mYZ+A3HTdYE0K/NmIfxKza5hNfZt8gQ8gEl0fPFnQcatmlCcac1ZRBAvYWz2VIsh6EscGJiKA0IV3tPeFeQnUs6UYF9SNo5pdEr6wxwnMJamPZ2X1tWDkK60jge5hw7UAWr4o3Nyaaot6tx+jjP5idIhr7mokx02VjNmHA4=;7:5lb7NB8xxFR0GO6Gn+gcFWUIN1uAqE5O4V16N12X0NJ1z7HEWohZTmGGIpGRVLQ53o00kRFYsqSeeloe8O8fqTBdypIRjYw+Ljg8ZMSWlYdrIMoPSY2WAhQaVLcbkiLg248d+c54B4zAaZQnysRKBipQBBwiKC4DwHdvFwKpNfVr4E6b0cz75pYwKhZCfHrx6BBCZs6N5q2lhM2mdsNNZGhRHgVLxYvkgpDF6y7tmKrs6ddmKf2ZYynhX/tZkLz2 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 856cce4a-488c-44df-e783-08d632b3a355 x-microsoft-antispam: BCL:0;PCL:0;RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020);SRVR:CY4PR02MB2279; x-ms-traffictypediagnostic: CY4PR02MB2279: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(85827821059158)(134217032509453)(98273658137915); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3231355)(944501410)(52105095)(3002001)(10201501046)(93006095)(93001095)(149066)(150057)(6041310)(20161123560045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123562045)(201708071742011)(7699051)(76991067);SRVR:CY4PR02MB2279;BCL:0;PCL:0;RULEID:;SRVR:CY4PR02MB2279; x-forefront-prvs: 0826B2F01B x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(396003)(136003)(376002)(346002)(366004)(39860400002)(189003)(199004)(33656002)(102836004)(2900100001)(186003)(26005)(6346003)(14454004)(97736004)(1730700003)(8676002)(81156014)(81166006)(606006)(105586002)(106356001)(8936002)(6306002)(6246003)(14444005)(5024004)(19627405001)(9686003)(236005)(54896002)(256004)(229853002)(5640700003)(55016002)(6436002)(68736007)(11346002)(446003)(316002)(53936002)(486006)(476003)(93886005)(6916009)(99286004)(6606003)(2351001)(2501003)(6116002)(3846002)(7696005)(5250100002)(5660300001)(66066001)(2906002)(7736002)(86362001)(25786009)(74316002)(71200400001)(478600001)(76176011)(71190400001)(53546011)(6506007)(966005)(3667265003);DIR:OUT;SFP:1101;SCL:1;SRVR:CY4PR02MB2279;H:CY4PR02MB3350.namprd02.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: epsilon.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=ninad.naik@epsilon.com; x-microsoft-antispam-message-info: taE6Rzspuz4RFedu12Fk650f9o+PXXIpD38SWWTnzgzhUJVjvNsICMdWt0114GiZwOpnbrbRtbdVU3vVZESz70R1RBwrvCA2l7nAPjP4P+pCHfU2XH3Hmro8yL10VPVBA3pZSIZ22bsC0c7w1zQXhozjCksj6lohdbLrISFFlCL4KhGtRS1sFB/JzAPr4BqGILRGRWrP/4lKdxT+6C904M8n3X/jWF7PAAHKuVk8LjDtXhIi3I6tUZFc1qNjoIEOJnbgnet2QkLaGmQ+po+PR6cFzwJ2DLJeYaMYwcRdTRJSphKB0c6YVTTSwL9TgvIIGqmaR0pn5fYloq77KLaCZSRe7RrdE0ST7Y0PosXwUhA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_CY4PR02MB33502CA56A103D86235C00B498FD0CY4PR02MB3350namp_" MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 856cce4a-488c-44df-e783-08d632b3a355 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Oct 2018 15:34:03.8193 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f603fe28-5be2-4b8f-b03c-98d0600b37f8 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR02MB2279 X-OriginatorOrg: epsilon.com --_000_CY4PR02MB33502CA56A103D86235C00B498FD0CY4PR02MB3350namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Thanks Jeff. We're not doing deletes, but I will take a look at this jira. ________________________________ From: Jeff Jirsa Sent: Sunday, October 14, 2018 12:55:17 PM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use ca= ution when clicking links or opening attachments. ] If this is 2.1 AND you do deletes AND you have a non-zero number of failed = writes (timeouts), it=92s possibly short reads 3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), i= t won=92t be backported to 2.1 because it=92s a significant change to how r= eads are executed -- Jeff Jirsa On Oct 13, 2018, at 7:24 PM, Naik, Ninad > wrote: Thanks Maitrayee. I should have mentioned this as one of the things we veri= fied. The clocks on cassandra nodes are in sync. ________________________________ From: maitrayee shah > Sent: Friday, October 12, 2018 6:40:25 PM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use ca= ution when clicking links or opening attachments. ] We have seen inconsistent read if the clock on the nodes are not in sync. Thank you Sent from my iPhone On Oct 12, 2018, at 1:50 PM, Naik, Ninad > wrote: Hello, We're seeing inconsistent data while doing reads on cassandra. Here are the= details: It's is a wide column table. The columns can be added my multiple machines,= and read by multiple machines. The time between writes and reads are in mi= nutes, but sometimes can be in seconds. Writes happen every 2 minutes. Now, while reading we're seeing the following cases of inconsistent reads: * One column was added. If a read was done after the column was added (= 20 secs to 2 minutes after the write), Cassandra returns no data. As if the= key doesn't exist. If the application retries, it gets the data. * A few columns exist for a row key. And a new column 'n' was added. Ag= ain, a read happens a few minutes after the write. This time, only the late= st column 'n' is returned. In this case the app doesn't know that the data = is incomplete so it doesn't retry. If we manually retry, we see all the col= umns. * A few columns exist for a row key. And a new column 'n' is added. Whe= n a read happens after the write, all columns but 'n' are returned. Here's what we've verified: * Both writes and reads are using 'LOCAL_QUORUM' consistency level. * The replication is within local data center. No remote data center is= involved in the read or write. * During the inconsistent reads, none of the nodes are undergoing GC pa= uses * There are no errors in cassandra logs * Reads always happen after the writes. A few other details: Cassandra version: 2.1.9 DataStax java driver version:= 2.1.10.2 Replication Factor: 3 We don't see this problem in lower environments. We have seen this happen o= nce or twice last year, but since last few days it's happening quite freque= ntly. On an average 2 inconsistent reads every minute. Here's how the table definition looks like: CREATE TABLE "MY_TABLE" ( key text, sub_key text, value text, PRIMARY KEY ((key), sub_key) ) WITH bloom_filter_fp_chance=3D0.010000 AND caching=3D'{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment=3D'' AND dclocal_read_repair_chance=3D0.100000 AND gc_grace_seconds=3D864000 AND read_repair_chance=3D0.000000 AND default_time_to_live=3D0 AND speculative_retry=3D'ALWAYS' AND memtable_flush_period_in_ms=3D0 AND compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND compression=3D{'sstable_compression': 'LZ4Compressor'}; Please point us in the right direction. Thanks ! The information contained in this e-mail message and any attachments may be= privileged and confidential. If the reader of this message is not the inte= nded recipient or an agent responsible for delivering it to the intended re= cipient, you are hereby notified that any review, dissemination, distributi= on or copying of this communication is strictly prohibited. If you have rec= eived this communication in error, please notify the sender immediately by = replying to this e-mail and delete the message and any attachments from you= r computer. The information contained in this e-mail message and any attachments may be= privileged and confidential. If the reader of this message is not the inte= nded recipient or an agent responsible for delivering it to the intended re= cipient, you are hereby notified that any review, dissemination, distributi= on or copying of this communication is strictly prohibited. If you have rec= eived this communication in error, please notify the sender immediately by = replying to this e-mail and delete the message and any attachments from you= r computer. The information contained in this e-mail message and any attachments may be= privileged and confidential. If the reader of this message is not the inte= nded recipient or an agent responsible for delivering it to the intended re= cipient, you are hereby notified that any review, dissemination, distributi= on or copying of this communication is strictly prohibited. If you have rec= eived this communication in error, please notify the sender immediately by = replying to this e-mail and delete the message and any attachments from you= r computer. --_000_CY4PR02MB33502CA56A103D86235C00B498FD0CY4PR02MB3350namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Thanks Jeff. We're not doing dele= tes, but I will take a look at this jira. 

From: Jeff Jirsa <jjirsa= @gmail.com>
Sent: Sunday, October 14, 2018 12:55:17 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
 

[ This email has been sent from a source extern= al to Epsilon. Please use caution when clicking links or opening attachment= s. ]

If this is 2.1 AND you do deletes AND you have a non-zero number of fa= iled writes (timeouts), it=92s possibly short reads

3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872=  ), it won=92t be backported to 2.1 because it=92s a significant chang= e to how reads are executed


-- 
Jeff Jirsa


On Oct 13, 2018, at 7:24 PM, Naik, Ninad <ninad.naik@epsilon.com> wrote:

Thanks Maitrayee. I should have = mentioned this as one of the things we verified. The clocks on cassandra no= des are in sync. 

From: maitrayee shah <= kooljava2@yahoo.com.INVALID<= /a>>
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apac= he.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
 

[ This email has been sent from a source extern= al to Epsilon. Please use caution when clicking links or opening attachment= s. ]

We have seen inconsistent read if the clock on the nodes are not in sy= nc. 


Thank you 

Sent from my iPhone

On Oct 12, 2018, at 1:50 PM, Naik, Ninad <ninad.naik@epsilon.com> wrote:

Hello= ,

We're seeing incons= istent data while doing reads on cassandra. Here are the details:

It's is a wide column table. The columns can be added my multiple machines,= and read by multiple machines. The time between writes and reads are in mi= nutes, but sometimes can be in seconds. Writes happen every 2 minutes.

Now, while reading we're seeing the following cases of inconsistent reads:<= /p>

  • One column was added. If a read was done after the column was added (20 sec= s to 2 minutes after the write), Cassandra returns no data. As if the key d= oesn't exist. If the application retries, it gets the data.
  • A few columns exist for a row key. And a new column 'n' was added. Again, a= read happens a few minutes after the write. This time, only the latest col= umn 'n' is returned. In this case the app doesn't know that the data is inc= omplete so it doesn't retry. If we manually retry, we see all the columns.
  • A few columns exist for a row key. And a new column 'n' is added. When a re= ad happens after the write, all columns but 'n' are returned.

Here's what we've verified:

  • Both writes and reads are using 'LOCAL_QUORUM' consistency level.
  • The replication is within local data center. No remote data center is invol= ved in the read or write.
  • During the inconsistent reads, none of the nodes are undergoing GC pauses
  • There are no errors in cassandra logs
  • Reads always happen after the writes.

A few other details: Cassandra version: 2.1.9 DataStax java driver version:= 2.1.10.2 Replication Factor: 3

We don't see this problem in lower environments. We have seen this happen o= nce or twice last year, but since last few days it's happening quite freque= ntly. On an average 2 inconsistent reads every minute.

Here's how the table definition looks like:

CREATE TABLE "MY_TABLE" (
  key text,
  sub_key text,
  value text,
  PRIMARY KEY ((key), sub_key)
) WITH
  bloom_filter_fp_chance=3D0.010000 AND
  caching=3D'{"keys":"ALL", "rows_per_partition&qu=
ot;:"NONE"}' AND
  comment=3D'' AND
  dclocal_read_repair_chance=3D0.100000 AND
  gc_grace_seconds=3D864000 AND
  read_repair_chance=3D0.000000 AND
  default_time_to_live=3D0 AND
  speculative_retry=3D'ALWAYS' AND
  memtable_flush_period_in_ms=3D0 AND
  compaction=3D{'class': 'SizeTieredCompactionStrategy'} AND
  compression=3D{'sstable_compression': 'LZ4Compressor'};

Please point us in the right direction. Thanks !

 

The information co= ntained in this e-mail message and any attachments may be privileged and co= nfidential. If the reader of this message is not the intended recipient or = an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dis= semination, distribution or copying of this communication is strictly prohi= bited. If you have received this communication in error, please notify the = sender immediately by replying to this e-mail and delete the message and any attachments from your computer.=

 

The information cont= ained in this e-mail message and any attachments may be privileged and conf= idential. If the reader of this message is not the intended recipient or an= agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissem= ination, distribution or copying of this communication is strictly prohibit= ed. If you have received this communication in error, please notify the sen= der immediately by replying to this e-mail and delete the message and any attachments from your computer.

 

The information conta= ined in this e-mail message and any attachments may be privileged and confi= dential. If the reader of this message is not the intended recipient or an = agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissem= ination, distribution or copying of this communication is strictly prohibit= ed. If you have received this communication in error, please notify the sen= der immediately by replying to this e-mail and delete the message and any attachments from your computer.

--_000_CY4PR02MB33502CA56A103D86235C00B498FD0CY4PR02MB3350namp_--