Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 85435 invoked from network); 17 Apr 2011 14:48:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Apr 2011 14:48:31 -0000 Received: (qmail 41769 invoked by uid 500); 17 Apr 2011 14:48:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41736 invoked by uid 500); 17 Apr 2011 14:48:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41728 invoked by uid 99); 17 Apr 2011 14:48:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Apr 2011 14:48:29 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.214.44 is neither permitted nor denied by domain of oberman@civicscience.com) Received: from [209.85.214.44] (HELO mail-bw0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Apr 2011 14:48:25 +0000 Received: by bwz13 with SMTP id 13so3835416bwz.31 for ; Sun, 17 Apr 2011 07:48:03 -0700 (PDT) Received: by 10.204.20.142 with SMTP id f14mr3366398bkb.155.1303051683380; Sun, 17 Apr 2011 07:48:03 -0700 (PDT) References: <8747B84D-8AAB-4DDB-A4B5-215EC7962B03@cmu.edu> <64E44C90-0AAB-40E3-B972-FD94C3D6C02F@cmu.edu> <86974EFE-F65B-47DB-86E7-0BDEB24E65FB@cmu.edu> From: William Oberman In-Reply-To: <86974EFE-F65B-47DB-86E7-0BDEB24E65FB@cmu.edu> Mime-Version: 1.0 (iPad Mail 8H7) Date: Sun, 17 Apr 2011 10:48:44 -0400 Message-ID: <1288264559807268502@unknownmsgid> Subject: Re: Consistency model To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=00032555505ab6400c04a11e5cf0 --00032555505ab6400c04a11e5cf0 Content-Type: text/plain; charset=ISO-8859-1 I'm pretty new to all of this, and I'm in the process of building my mental model of Cassandra, but I'm still feeling better about this thread. The way I figure it: 1. I'm trying to mutate the state of a key's column from A to B from a thread somewhere (quorum) 2. I'm trying to read the state of a key from a thread somewhere else (quorum) If #1 succeeds I'm guaranteed to see B. If #1 fails (with an exception) I'll see either A or B. I think I was concerned about that, and wanted to see A in #2 until success in #1. But, I wanted to get to state B, and if #1 retries until guaranteed success, do I care if I set B earlier than I expected? I'm thinking no. I guess in terms of distributed algorithms/reasoning about systems, I'm feeling ok with this level of guarantee (again, given the failed write tells the client code of the undefined state). On Apr 17, 2011, at 10:10 AM, James Cipar wrote: I'm pretty new to Cassandra, but I've also written a client in C++ using the thrift API directly. From what I've seen, wrapping writes in a retry loop is pretty much mandatory because if you are pushing a lot of data around, you're basically guaranteed to have TimedOutExceptions. I suppose what I'm getting at is: if you don't have consistency in the case of a TimedOutException, you don't have consistency for any high-throughput application. Is there a solution to this that I am missing? On Apr 17, 2011, at 9:42 AM, William Oberman wrote: At first I was concerned and was going to +1 on a fix, but I think I was confused on one detail and I'd like to confirm it. -An unsuccessful write implies readers can see either the old or new value ? The trick is using a library, it sounds like there is a period of time a write is unsuccessful but you don't know about it (as the retry is internal). But, (assuming writes are idempotent) QUORUM is actually consistent from successful writes to successful reads... right? On Sun, Apr 17, 2011 at 1:53 AM, Jonathan Ellis wrote: > Tyler is correct, because Cassandra doesn't wait until repair writes > are acked before the answer is returned. This is something we can fix. > > On Sun, Apr 17, 2011 at 12:05 AM, Sean Bridges > wrote: > > Tyler, your answer seems to contradict this email by Jonathan Ellis > > [1]. In it Jonathan says, > > > > "The important guarantee this gives you is that once one quorum read > > sees the new value, all others will too. You can't see the newest > > version, then see an older version on a subsequent write [sic, I > > assume he meant read], which is the characteristic of non-strong > > consistency" > > > > Jonathan also says, > > > > "{X, Y} and {X, Z} are equivalent: one node with the write, and one > > without. The read will recognize that X's version needs to be sent to > > Z, and the write will be complete. This read and all subsequent ones > > will see the write. (Z [sic, I assume he meant Y] will be replicated > > to asynchronously via read repair.)" > > > > To me, the statement "this read and all subsequent ones will see the > > write" implies that the new value must be committed to Y or Z before > > the read can return. If not, the statement must be false. > > > > Sean > > > > > > [1] : > http://mail-archives.apache.org/mod_mbox/cassandra-user/201102.mbox/%3CAANLkTimEGp8H87mGs_BxZKNCk-A59whXF-Xx58HcAWZm@mail.gmail.com%3E > > > > Sean > > > > On Sat, Apr 16, 2011 at 7:44 PM, Tyler Hobbs wrote: > >> Here's what's probably happening: > >> > >> I'm assuming RF=3 and QUORUM writes/reads here. I'll call the replicas > A, > >> B, and C. > >> > >> 1. Writer process writes sequence number 1 and everything works fine. > A, > >> B, and C all have sequence number 1. > >> 2. Writer process writes sequence number 2. Replica A writes > successfully, > >> B and C fail to respond in time, and a TimedOutException is returned. > >> pycassa waits to retry the operation. > >> 3. Reader process reads, gets a response from A and B. When the row > from A > >> and B is merged, sequence number 2 is the newest and is returned. A > read > >> repair is pushed to B and C, but they don't yet update their data. > >> 4. Reader process reads again, gets a response from B and C (before > they've > >> repaired). These both report sequence number 1, so that's returned to > the > >> client. This is were you get a decreasing sequence number. > >> 5. pycassa eventually retries the write; B and C eventually repair > their > >> data. Either way, both B and C shortly have sequence number 2. > >> > >> I've left out some of the details of read repair, and this scenario > could > >> happen in several slightly different ways, but it should give you an > idea of > >> what's happening. > >> > >> On Sat, Apr 16, 2011 at 8:35 PM, James Cipar wrote: > >>> > >>> Here it is. There is some setup code and global variable definitions > that > >>> I left out of the previous code, but they are pretty similar to the > setup > >>> code here. > >>> import pycassa > >>> import random > >>> import time > >>> consistency_level = > pycassa.cassandra.ttypes.ConsistencyLevel.QUORUM > >>> duration = 600 > >>> sleeptime = 0.0 > >>> hostlist = 'worker-hostlist' > >>> def read_servers(fn): > >>> f = open(fn) > >>> servers = [] > >>> for line in f: > >>> servers.append(line.strip()) > >>> f.close() > >>> return servers > >>> servers = read_servers(hostlist) > >>> start_time = time.time() > >>> seqnum = -1 > >>> timestamp = 0 > >>> while time.time() < start_time + duration: > >>> target_server = random.sample(servers, 1)[0] > >>> target_server = '%s:9160'%target_server > >>> try: > >>> pool = pycassa.connect('Keyspace1', [target_server]) > >>> cf = pycassa.ColumnFamily(pool, 'Standard1') > >>> row = cf.get('foo', > read_consistency_level=consistency_level) > >>> pool.dispose() > >>> except: > >>> time.sleep(sleeptime) > >>> continue > >>> sq = int(row['seqnum']) > >>> ts = float(row['timestamp']) > >>> if sq < seqnum: > >>> print 'Row changed: %i %f -> %i %f'%(seqnum, timestamp, sq, > >>> ts) > >>> seqnum = sq > >>> timestamp = ts > >>> if sleeptime > 0.0: > >>> time.sleep(sleeptime) > >>> > >>> > >>> > >>> On Apr 16, 2011, at 5:20 PM, Tyler Hobbs wrote: > >>> > >>> James, > >>> > >>> Would you mind sharing your reader process code as well? > >>> > >>> On Fri, Apr 15, 2011 at 1:14 PM, James Cipar wrote: > >>>> > >>>> I've been experimenting with the consistency model of Cassandra, and I > >>>> found something that seems a bit unexpected. In my experiment, I have > 2 > >>>> processes, a reader and a writer, each accessing a Cassandra cluster > with a > >>>> replication factor greater than 1. In addition, sometimes I generate > >>>> background traffic to simulate a busy cluster by uploading a large > data file > >>>> to another table. > >>>> > >>>> The writer executes a loop where it writes a single row that contains > >>>> just an sequentially increasing sequence number and a timestamp. In > python > >>>> this looks something like: > >>>> > >>>> while time.time() < start_time + duration: > >>>> target_server = random.sample(servers, 1)[0] > >>>> target_server = '%s:9160'%target_server > >>>> > >>>> row = {'seqnum':str(seqnum), 'timestamp':str(time.time())} > >>>> seqnum += 1 > >>>> # print 'uploading to server %s, %s'%(target_server, row) > >>>> > >>>> pool = pycassa.connect('Keyspace1', [target_server]) > >>>> cf = pycassa.ColumnFamily(pool, 'Standard1') > >>>> cf.insert('foo', row, > write_consistency_level=consistency_level) > >>>> pool.dispose() > >>>> > >>>> if sleeptime > 0.0: > >>>> time.sleep(sleeptime) > >>>> > >>>> > >>>> The reader simply executes a loop reading this row and reporting > whenever > >>>> a sequence number is *less* than the previous sequence number. As > expected, > >>>> with consistency_level=ConsistencyLevel.ONE there are many > inconsistencies, > >>>> especially with a high replication factor. > >>>> > >>>> What is unexpected is that I still detect inconsistencies when it is > set > >>>> at ConsistencyLevel.QUORUM. This is unexpected because the > documentation > >>>> seems to imply that QUORUM will give consistent results. With > background > >>>> traffic the average difference in timestamps was 0.6s, and the maximum > was > >>>> >3.5s. This means that a client sees a version of the row, and can > >>>> subsequently see another version of the row that is 3.5s older than > the > >>>> previous. > >>>> > >>>> What I imagine is happening is this, but I'd like someone who knows > that > >>>> they're talking about to tell me if it's actually the case: > >>>> > >>>> I think Cassandra is not using an atomic commit protocol to commit to > the > >>>> quorum of servers chosen when the write is made. This means that at > some > >>>> point in the middle of the write, some subset of the quorum have seen > the > >>>> write, while others have not. At this time, there is a quorum of > servers > >>>> that have not seen the update, so depending on which quorum the client > reads > >>>> from, it may or may not see the update. > >>>> > >>>> Of course, I understand that the client is not *choosing* a bad quorum > to > >>>> read from, it is just the first `q` servers to respond, but in this > case it > >>>> is effectively random and sometimes an bad quorum is "chosen". > >>>> > >>>> Does anyone have any other insight into what is going on here? > >>> > >>> > >>> -- > >>> Tyler Hobbs > >>> Software Engineer, DataStax > >>> Maintainer of the pycassa Cassandra Python client library > >>> > >>> > >> > >> > >> > >> -- > >> Tyler Hobbs > >> Software Engineer, DataStax > >> Maintainer of the pycassa Cassandra Python client library > >> > >> > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) oberman@civicscience.com --00032555505ab6400c04a11e5cf0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I'm pretty new to all of this, and= I'm in the process of building my mental model of Cassandra, but I'= ;m still feeling better about this thread. The way I figure it:
1. I'm trying to mutate the state of a key's column from A to B fro= m a thread somewhere (quorum)
2. I'm trying to read the state= of a key from a thread somewhere else (quorum)

If #1 succeeds I'm guaranteed to see B. If #1 fails (with an exception)= I'll see either A or B. I think I was concerned about that, and wanted= to see A in #2 until success in #1. =A0But, I wanted to get to state B, an= d if #1 retries until guaranteed success, do I care if I set B earlier than= I expected? =A0I'm thinking no.=A0

I guess in terms of distributed algorithms/reasoning ab= out systems, I'm feeling ok with this level of guarantee (again, given = the failed write tells the client code of the undefined state).=A0

On Apr 17, 2011, at 10:10 AM, James Cipar <jcipar@cmu.edu> wrote:

I'm pretty new to Cassandra, but I've also writt= en a client in C++ using the thrift API directly. =A0From what I've see= n, wrapping writes in a retry loop is pretty much mandatory because if you = are pushing a lot of data around, you're basically guaranteed to have T= imedOutExceptions. =A0I suppose what I'm getting at is: if you don'= t have consistency in the case of a TimedOutException, you don't have c= onsistency for any high-throughput application. =A0Is there a solution to t= his that I am missing?


On Apr 17, 2011, at 9:42 AM, William Oberman wr= ote:

At first I was concerned and was going to +1 =A0on a fix, but I think I wa= s confused on one detail and I'd like to confirm it.
-An unsuccessful write implies readers can see either the old or new value<= /div>
?

The trick is using a library, it sounds like there is a= period of time a write is unsuccessful but you don't know about it (as= the retry is internal). =A0But, (assuming writes are idempotent) QUORUM is= actually consistent from successful writes to successful reads... right?

On Sun, Apr 17, 2011 at 1:53 = AM, Jonathan Ellis <jbellis@gmail.com> wrote:
Tyler is correct, because Cassandra doesn't wait until repair writes are acked before the answer is returned. This is something we can fix.

On Sun, Apr 17, 2011 at 12:05 AM, Sean Bridges <sean.bridges@gm= ail.com> wrote:
> Tyler, your answer seems to contradict this email by Jonathan Ellis > [1]. =A0In it Jonathan says,
>
> "The important guarantee this gives you is that once one quorum r= ead
> sees the new value, all others will too. =A0 You can't see the new= est
> version, then see an older version on a subsequent write [sic, I
> assume he meant read], which is the characteristic of non-strong
> consistency"
>
> Jonathan also says,
>
> "{X, Y} and {X, Z} are equivalent: one node with the write, and o= ne
> without. The read will recognize that X's version needs to be sent= to
> Z, and the write will be complete. =A0This read and all subsequent one= s
> will see the write. =A0(Z [sic, I assume he meant Y] will be replicate= d
> to asynchronously via read repair.)"
>
> To me, the statement "this read and all subsequent ones will see = the
> write" implies that the new value must be committed to Y or Z bef= ore
> the read can return. =A0If not, the statement must be false.
>
> Sean
>
>
> [1] : http://mail-archives.apache.org/mod_mbox/cassandra-u= ser/201102.mbox/%3CAANLkTimEGp8H87mGs_BxZKNCk-A59whXF-Xx58HcAWZm@mail.gmail= .com%3E
>
> Sean
>
> On Sat, Apr 16, 2011 at 7:44 PM, Tyler Hobbs <tyler@datastax.com> wrote:
>> Here's what's probably happening:
>>
>> I'm assuming RF=3D3 and QUORUM writes/reads here.=A0 I'll = call the replicas A,
>> B, and C.
>>
>> 1.=A0 Writer process writes sequence number 1 and everything works= fine.=A0 A,
>> B, and C all have sequence number 1.
>> 2.=A0 Writer process writes sequence number 2.=A0 Replica A writes= successfully,
>> B and C fail to respond in time, and a TimedOutException is return= ed.
>> pycassa waits to retry the operation.
>> 3.=A0 Reader process reads, gets a response from A and B.=A0 When = the row from A
>> and B is merged, sequence number 2 is the newest and is returned.= =A0 A read
>> repair is pushed to B and C, but they don't yet update their d= ata.
>> 4.=A0 Reader process reads again, gets a response from B and C (be= fore they've
>> repaired).=A0 These both report sequence number 1, so that's r= eturned to the
>> client.=A0 This is were you get a decreasing sequence number.
>> 5.=A0 pycassa eventually retries the write; B and C eventually rep= air their
>> data.=A0 Either way, both B and C shortly have sequence number 2.<= br> >>
>> I've left out some of the details of read repair, and this sce= nario could
>> happen in several slightly different ways, but it should give you = an idea of
>> what's happening.
>>
>> On Sat, Apr 16, 2011 at 8:35 PM, James Cipar <jcipar@cmu.edu&g= t; wrote:
>>>
>>> Here it is. =A0There is some setup code and global variable de= finitions that
>>> I left out of the previous code, but they are pretty similar t= o the setup
>>> code here.
>>> =A0=A0 =A0import pycassa
>>> =A0=A0 =A0import random
>>> =A0=A0 =A0import time
>>> =A0=A0 =A0consistency_level =3D pycassa.cassandra.ttypes.Consi= stencyLevel.QUORUM
>>> =A0=A0 =A0duration =3D 600
>>> =A0=A0 =A0sleeptime =3D 0.0
>>> =A0=A0 =A0hostlist =3D 'worker-hostlist'
>>> =A0=A0 =A0def read_servers(fn):
>>> =A0=A0 =A0 =A0 =A0f =3D open(fn)
>>> =A0=A0 =A0 =A0 =A0servers =3D []
>>> =A0=A0 =A0 =A0 =A0for line in f:
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0servers.append(line.strip())
>>> =A0=A0 =A0 =A0 =A0f.close()
>>> =A0=A0 =A0 =A0 =A0return servers
>>> =A0=A0 =A0servers =3D read_servers(hostlist)
>>> =A0=A0 =A0start_time =3D time.time()
>>> =A0=A0 =A0seqnum =3D -1
>>> =A0=A0 =A0timestamp =3D 0
>>> =A0=A0 =A0while time.time() < start_time + duration:
>>> =A0=A0 =A0 =A0 =A0target_server =3D random.sample(servers, 1)[= 0]
>>> =A0=A0 =A0 =A0 =A0target_server =3D '%s:9160'%target_s= erver
>>> =A0=A0 =A0 =A0 =A0try:
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0pool =3D pycassa.connect('Keyspa= ce1', [target_server])
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0cf =3D pycassa.ColumnFamily(pool, &#= 39;Standard1')
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0row =3D cf.get('foo', read_c= onsistency_level=3Dconsistency_level)
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0pool.dispose()
>>> =A0=A0 =A0 =A0 =A0except:
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0time.sleep(sleeptime)
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0continue
>>> =A0=A0 =A0 =A0 =A0sq =3D int(row['seqnum'])
>>> =A0=A0 =A0 =A0 =A0ts =3D float(row['timestamp'])
>>> =A0=A0 =A0 =A0 =A0if sq < seqnum:
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0print 'Row changed: %i %f -> = %i %f'%(seqnum, timestamp, sq,
>>> ts)
>>> =A0=A0 =A0 =A0 =A0seqnum =3D sq
>>> =A0=A0 =A0 =A0 =A0timestamp =3D ts
>>> =A0=A0 =A0 =A0 =A0if sleeptime > 0.0:
>>> =A0=A0 =A0 =A0 =A0 =A0 =A0time.sleep(sleeptime)
>>>
>>>
>>>
>>> On Apr 16, 2011, at 5:20 PM, Tyler Hobbs wrote:
>>>
>>> James,
>>>
>>> Would you mind sharing your reader process code as well?
>>>
>>> On Fri, Apr 15, 2011 at 1:14 PM, James Cipar <jcipar@cmu.edu> wrote:
>>>>
>>>> I've been experimenting with the consistency model of = Cassandra, and I
>>>> found something that seems a bit unexpected. =A0In my expe= riment, I have 2
>>>> processes, a reader and a writer, each accessing a Cassand= ra cluster with a
>>>> replication factor greater than 1. =A0In addition, sometim= es I generate
>>>> background traffic to simulate a busy cluster by uploading= a large data file
>>>> to another table.
>>>>
>>>> The writer executes a loop where it writes a single row th= at contains
>>>> just an sequentially increasing sequence number and a time= stamp. =A0In python
>>>> this looks something like:
>>>>
>>>> =A0 =A0while time.time() < start_time + duration:
>>>> =A0 =A0 =A0 =A0target_server =3D random.sample(servers, 1)= [0]
>>>> =A0 =A0 =A0 =A0target_server =3D '%s:9160'%target_= server
>>>>
>>>> =A0 =A0 =A0 =A0row =3D {'seqnum':str(seqnum), '= ;timestamp':str(time.time())}
>>>> =A0 =A0 =A0 =A0seqnum +=3D 1
>>>> =A0 =A0 =A0 =A0# print 'uploading to server %s, %s'= ;%(target_server, row)
>>>>
>>>> =A0 =A0 =A0 =A0pool =3D pycassa.connect('Keyspace1'= ;, [target_server])
>>>> =A0 =A0 =A0 =A0cf =3D pycassa.ColumnFamily(pool, 'Stan= dard1')
>>>> =A0 =A0 =A0 =A0cf.insert('foo', row, write_consist= ency_level=3Dconsistency_level)
>>>> =A0 =A0 =A0 =A0pool.dispose()
>>>>
>>>> =A0 =A0 =A0 =A0if sleeptime > 0.0:
>>>> =A0 =A0 =A0 =A0 =A0 =A0time.sleep(sleeptime)
>>>>
>>>>
>>>> The reader simply executes a loop reading this row and rep= orting whenever
>>>> a sequence number is *less* than the previous sequence num= ber. =A0As expected,
>>>> with consistency_level=3DConsistencyLevel.ONE there are ma= ny inconsistencies,
>>>> especially with a high replication factor.
>>>>
>>>> What is unexpected is that I still detect inconsistencies = when it is set
>>>> at ConsistencyLevel.QUORUM. =A0This is unexpected because = the documentation
>>>> seems to imply that QUORUM will give consistent results. = =A0With background
>>>> traffic the average difference in timestamps was 0.6s, and= the maximum was
>>>> >3.5s. =A0This means that a client sees a version of th= e row, and can
>>>> subsequently see another version of the row that is 3.5s o= lder than the
>>>> previous.
>>>>
>>>> What I imagine is happening is this, but I'd like some= one who knows that
>>>> they're talking about to tell me if it's actually = the case:
>>>>
>>>> I think Cassandra is not using an atomic commit protocol t= o commit to the
>>>> quorum of servers chosen when the write is made. =A0This m= eans that at some
>>>> point in the middle of the write, some subset of the quoru= m have seen the
>>>> write, while others have not. =A0At this time, there is a = quorum of servers
>>>> that have not seen the update, so depending on which quoru= m the client reads
>>>> from, it may or may not see the update.
>>>>
>>>> Of course, I understand that the client is not *choosing* = a bad quorum to
>>>> read from, it is just the first `q` servers to respond, bu= t in this case it
>>>> is effectively random and sometimes an bad quorum is "= ;chosen".
>>>>
>>>> Does anyone have any other insight into what is going on h= ere?
>>>
>>>
>>> --
>>> Tyler Hobbs
>>> Software Engineer, DataStax
>>> Maintainer of the pycassa Cassandra Python client library
>>>
>>>
>>
>>
>>
>> --
>> Tyler Hobbs
>> Software Engineer, DataStax
>> Maintainer of the pycassa Cassandra Python client library
>>
>>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com



--
Will Oberman
= Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 152= 01
(M) 412-480-7835
(E) <= a href=3D"mailto:oberman@civicscience.com">oberman@civicscience.com=

--00032555505ab6400c04a11e5cf0--