hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vrodio...@carrieriq.com>
Subject Master server abort
Date Thu, 11 Jul 2013 18:10:53 GMT
This is happening in  one of our small QA cluster.
HBase 0.94.6.1 (CDH 4.3.0)

1 master + 5 RS. Zk quorum is 1 (on master node)

We can not start the cluster:

In a log file I find some ERROR's and FATALs . FATAL's come first followed by ERRORs (this
is important):

FATALs:

2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region SMALL_GOLDENROD_2012-IDPROFILES,31,1363783108271.77a4640bfaecc907e0ea3535a16c56a8.
that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region TEST_MM5550_INDEX-UPLOADS,9C,1363689771995.2ee2e6b81ee44ff790abf38275698d45.
that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
2013-07-10 19:42:00,376 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort:
loaded coprocessors are: []
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region SMALL_GOLDENROD_2012-IDPROFILES,F8,1363783108280.a0b1b6d003df84ca1404af942bcc9fbc.
that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region TEST_MM5550_INDEX-UPLOADS,E0,1363689771998.36db41b10c86ac537542104c87950709.
that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region SMALL_GOLDENROD_2012-IDPROFILES,FD,1363783108280.95ac4fb83f1bc2753aca0f6a914f6ff2.
that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region TEST_MM5550_INDEX-UPLOADS,E5,1363689771999.65440a41f85b9dd70afd669280491363.
that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region SMALL_GOLDENROD_2012-IDPROFILES,34,1363783108271.979f840723771588cf65910183ecf55c.
that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region TEST_MM5550_INDEX-UPLOADS,A0,1363689771995.45f193f3dcfcfa76e705b5fa020e4309.
that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
2013-07-10 19:42:00,377 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
.. Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state : packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
.. Cannot transit it to OFFLINE.
        at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
        at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-07-10 19:42:00,377 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master
has opened the region XXX-IDPROFILES,36,1363783108271.65e59eae29b9d61d294d2dfcc3f08bd9. that
was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,379 INFO org.apache.hadoop.hbase.master.HMaster: Aborting


All errors in a log files are logged after the master started abort operation.

There are two types of ERRORs:

Type 1:

2013-07-10 19:42:01,084 ERROR org.apache.hadoop.hbase.master.handler.OpenedRegionHandler:
The znode of region TRIAL-UPLOADS,1C,1360234840368.a6944572c8e4a2447d3dcaa1d5029500. could
not be deleted.

Type 2:

2013-07-10 19:42:01,085 ERROR org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[OpenedRegionHandler-sjc1-eng-qa01.carrieriq.com,60000,1373485277977-8205] because the executor
is missing. Is this process shutting down?


But again, I think that these errors are results of abort operation.

So, the question is what can the reason of a master sudden shutdown and where should I look
for clues?

WARNINGs preceding first FATAL:

2013-07-10 19:41:18,196 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated.
Instead, use mapreduce.task.attempt.id
2013-07-10 19:41:18,385 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/backup-masters/sjc1-eng-qa01.carrieriq.com,60000,1373485277977 already deleted, and
this is not a retry
2013-07-10 19:41:18,942 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-07-10 19:41:19,208 WARN org.apache.hadoop.conf.Configuration: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
2013-07-10 19:41:19,494 WARN org.apache.hadoop.hbase.master.snapshot.SnapshotManager: Couldn't
delete working snapshot directory: hdfs://sjc1-eng-qa01.carrieriq.com:9000/hbase/.snapshot/.tmp
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-07-10 19:41:27,787 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/root-region-server already deleted, and this is not a retry
2013-07-10 19:41:27,818 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/-ROOT- already deleted, and this is not a retry
2013-07-10 19:41:29,562 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/.META. already deleted, and this is not a retry
2013-07-10 19:41:32,846 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1359838920000 already deleted, and this is not
a retry
2013-07-10 19:41:32,909 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1360281600000 already deleted, and this is not
a retry
2013-07-10 19:41:32,966 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_OUTPUT_68_TS-DIMENSIONS-1360281540000 already deleted, and this is not
a retry
2013-07-10 19:41:33,011 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_SMOKE_98-DIMENSIONS-1358380800000 already deleted, and this is not a retry
2013-07-10 19:41:33,042 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_SMOKE_TS-DIMENSIONS-1358121540000 already deleted, and this is not a retry
2013-07-10 19:41:33,070 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1359590340000 already deleted, and this is not
a retry
2013-07-10 19:41:33,107 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1360281540000 already deleted, and this is not
a retry
2013-07-10 19:41:33,128 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/CB_SMALL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,184 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/CB_SMALL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,226 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1327881540000 already deleted, and this is not a
retry
2013-07-10 19:41:33,245 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1359503940000 already deleted, and this is not a
retry
2013-07-10 19:41:33,264 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICERATIO_FILTER_111_TS-DIMENSIONS-1358467200000 already deleted, and this
is not a retry
2013-07-10 19:41:33,288 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICE_LIST_TS-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,306 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICE_RATIO_0_7_TS-DIMENSIONS-1358467200000 already deleted, and this is not
a retry
2013-07-10 19:41:33,323 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICE_RATIO_109_TS-DIMENSIONS-1357603200000 already deleted, and this is not
a retry
2013-07-10 19:41:33,338 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICE_RATIO_109_TS2-DIMENSIONS-1357603200000 already deleted, and this is not
a retry
2013-07-10 19:41:33,352 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/DEVICE_RATIO_109_TS3-DIMENSIONS-1358467200000 already deleted, and this is not
a retry
2013-07-10 19:41:33,364 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/GENERATED_UB-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,390 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/GENERATED_UB-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,416 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/GOLDENROD_PERF_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,444 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/GOLDENROD_PERF_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,467 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INCOMING_UPLOADBINARIES_101229-IDPROFILES already deleted, and this is not a
retry
2013-07-10 19:41:33,491 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INCOMING_UPLOADBINARIES_101229-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,514 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INCOMING_UPLOADBINARIES_VE-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,535 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INCOMING_UPLOADBINARIES_VE-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,557 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INDEX_TEST3-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,578 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/INDEX_TEST3-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,596 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/IQADataStore already deleted, and this is not a retry
2013-07-10 19:41:33,603 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/MM_5550_TEST-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,619 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/MM_5550_TEST-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,634 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/MetaTable already deleted, and this is not a retry
2013-07-10 19:41:33,644 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/NO_FILTER_111_TS-DIMENSIONS-1358467200000 already deleted, and this is not a
retry
2013-07-10 19:41:33,654 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/RTI_331_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,669 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/RTI_331_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,683 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/RTI_SEBY_TEST-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,702 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/RTI_SEBY_TEST-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,716 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/SMALL_GOLDENROD_2012-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,730 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/SMALL_GOLDENROD_2012-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,744 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/SYNTHETIC_UB-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,757 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/SYNTHETIC_UB-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,771 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TEST_INDEX_TOOL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,785 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TEST_INDEX_TOOL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,810 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TEST_MM5550_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,823 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TEST_MM5550_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,837 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TMO_2012_DATASET-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,850 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TMO_2012_DATASET-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,863 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TRIAL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,876 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/TRIAL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,889 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
/hbase/table/packageindex already deleted, and this is not a retry
2013-07-10 19:41:56,349 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED
for region 61e36043936d733a370c8f6dddb83566 from server sjc1-eng-qa04.carrieriq.com,60020,1373485278882
but region was in  the state INDEX_TEST3-IDPROFILES,16,1363283705765.61e36043936d733a370c8f6dddb83566.
state=OPEN, ts=1373485312867, server=sjc1-eng-qa04.carrieriq.com,60020,1373485278882 and not
in expected OFFLINE, PENDING_OPEN or OPENING states

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
Sent: Thursday, July 11, 2013 10:55 AM
To: user@hbase.apache.org
Subject: Re: G1 before/after GC time graph

Correct!

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jul 11, 2013 1:18 PM, "Asaf Mesika" <asaf.mesika@gmail.com> wrote:

> This means you can safely run Hadoop and Hbase on jvm 7?
> We were just considering switching in production to java 7.
>
> On Thursday, July 11, 2013, Azuryy Yu wrote:
>
> > Otis,
> >
> > I will do this test, maybe on the end of this month. because I haven't
> big
> > memory server for test now util the end of this month.
> >
> >
> >
> >
> > On Thu, Jul 11, 2013 at 9:38 AM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Probably not any time in the near future.  We could do it, plop SPM in
> > > it and watch, but don't have time for it now.
> > >
> > > If anyone wants to give it a shot and use SPM to capture various
> > > JVM/HBase/server metrics, let me know if help around SPM is needed,
> > > longer trials, or whatever.
> > >
> > > Otis
> > > --
> > > HBase Performance Monitoring -- http://sematext.com/spm
> > >
> > >
> > >
> > > On Wed, Jul 10, 2013 at 5:27 PM, Kevin O'dell <
> kevin.odell@cloudera.com>
> > > wrote:
> > > > Anyway you can test this with a 32GB heap? :)
> > > >
> > > >
> > > > On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > >> But, very importantly:
> > > >>
> > > >> CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
> > > >> Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ
> > > >>
> > > >> So in practical terms, use something like -XX:MaxGCPauseMillis=200
> > > >> -XX:NewRatio=4 to force short GC pauses, but count on having more
of
> > > >> them and CPU sweating more.  This gives you smaller pauses, no big
> STW
> > > >> pauses, but be prepared to pay in CPU cycles.
> > > >>
> > > >> HTH
> > > >>
> > > >> Otis
> > > >> --
> > > >> HBase Performance Monitoring -- http://sematext.com/spm
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
> > > >> <otis.gospodnetic@gmail.com> wrote:
> > > >> > Not big in this case - 1800m
> > > >> >
> > > >> > Otis
> > > >> > --
> > > >> > HBase Performance Monitoring -- http://sematext.com/spm
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
> > > >> > <vrodionov@carrieriq.com> wrote:
> > > >> >>
> > > >> >> What is  -Xmx heap size?
> > > >> >>
> > > >> >> Best regards,
> > > >> >> Vladimir Rodionov
> > > >> >> Principal Platform Engineer
> > > >> >> Carrier IQ, www.carrieriq.com
> > > >> >> e-mail: vrodionov@carrieriq.com
> > > >> >>
> > > >> >> ________________________________________
> > > >> >> From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
> > > >> >> Sent: Wednesday, July 10, 2013 12:00 PM
> > > >> >> To: user@hbase.apache.org
> > > >> >> Subject: G1 before/after GC time graph
> > > >> >>
> > > >> >> Hi,
> > > >> >>
> > > >> >> There was talk of G1 and HBase here over the past few days.
 I
> > wanted
> > > >> >> to share a fresh and telling SPM graph showing G1 + some
other
> > params
> > > >> >> in action:
> > > >> >>
> > > >> >> Result first: https://apps.sematext.com/spm/s/wN0hKVixxX
> > > >> >>
> > > >> >> This is what we had before:
> > > >> >>   -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
> > > >> >> -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25
> > > >> >>
> > > >> >> And then we added:
> > > >> >>
>

Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.

Mime
View raw message