hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Luo <j...@merkleinc.com>
Subject RE: CopyTable fails on copying between two secured clusters
Date Thu, 08 Sep 2016 23:45:26 GMT
I don't think they are pointing to different locations. Both of them should be /hbase-secure.

However, the debugger shows that ConnectionManager#retrieveClusterId are called twice, the
first time regards to the source cluster, which works fine, and
watcher.clusterIdZNode=/hbase-secure/hbaseid, and it is correct.

The second time for the remote cluster, watcher.clusterIdZNode=/hbase/hbaseid, which should
be incorrect.

What I am suspecting is ZooKeeperWatcher, method setNodeNames. It reads:

  private void setNodeNames(Configuration conf) {
    baseZNode = conf.get(HConstants.ZOOKEEPER_ZNODE_PARENT,
        HConstants.DEFAULT_ZOOKEEPER_ZNODE_PARENT);

I am not sure the conf is corrected fetched from the remote cluster. If not, the default value
is given, which is /hbase and incorrect.

By the way, below is the root znodes for zookeepers:

The source cluster:
 [hbase-secure, hiveserver2, hive, hbase-unsecure, templeton-hadoop, hadoop-ha, zookeeper]

The target cluster:
[hbase-secure, hive, hiveserver2, hbase-unsecure, hadoop-ha, zookeeper]

-----Original Message-----
From: Esteban Gutierrez [mailto:esteban@cloudera.com]
Sent: Thursday, September 08, 2016 1:02 PM
To: user@hbase.apache.org
Subject: Re: CopyTable fails on copying between two secured clusters

Is it possible that in your destination cluster zookeeper.znode.parent points to a different
location than /hbase ? If both clusters are under the same kerberos realm then there is no
need to worry about zookeeper.security.auth_to_local.



--
Cloudera, Inc.


On Thu, Sep 8, 2016 at 10:50 AM, Frank Luo <jluo@merkleinc.com> wrote:

> Thanks Esteban for replying.
>
> The Kerberos realm is shared between the two clusters.
>
>  I searched zookeeper config and couldn't find the rule, so where it
> is set?
>
> Having said that, I looked at parameters passed to getData call, and
> it doesn't look like security related.
>
> PS. I am using hbase 1.1.2.
>
> Here is the log:
>
> com.merkleinc.cr.hbase_maintenance.tableexport.CopyTableTest,testCopyT
> able Connected to the target VM, address: '127.0.0.1:50669',
> transport:
> 'socket'
> 0    [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil  -
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 247  [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to
> load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 1492 [main] INFO  org.apache.hadoop.security.UserGroupInformation  -
> Login successful for user tidsvc_exphbase@PCLC0.MERKLE.LOCAL using
> keytab file \Users\jluo\tidsvc_exphbase.keytab
> 11462 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 164493 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 167720 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 172108 [main] WARN
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory
> - The short-circuit local reads feature cannot be used because UNIX
> Domain sockets are not available on Windows.
> 224735 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 225165 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x4dd5d6bb connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 225190 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> host.name=JLUOWIN7.pclc0.merkle.local
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.version=1.7.0_51
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.vendor=Oracle Corporation
> 225191 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.home=C:\work\jdk1.7.0_51\jre
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.io.tmpdir=C:\Users\jluo\AppData\Local\Temp\
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:java.compiler=<NA>
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> os.name=Windows 7
> 225193 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:os.arch=amd64
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:os.version=6.1
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client environment:
> user.name=jluo
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:user.home=C:\Users\jluo
> 225194 [main] INFO  org.apache.zookeeper.ZooKeeper  - Client
> environment:user.dir=C:\work\project\c2\trunk\misc\hbase_maintenance
> 225197 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=hconnection-0x4dd5d6bb0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 225442 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 225490 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 225544 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 231758 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> 231758 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> client connection, connectString=hqhd02nm01.pclc0.merkle.local:2181,
> hqhd02nm02.pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181
> sessionTimeout=120000 watcher=TokenUtil-getAuthToken0x0,
> quorum=hqhd02nm01.pclc0.merkle.local:2181,hqhd02nm02.
> pclc0.merkle.local:2181,hqhd02ed01.pclc0.merkle.local:2181,
> baseZNode=/hbase-secure
> 231876 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 253405 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, initiating session
> 257914 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02nm02.pclc0.merkle.local/10.129.8.14:2181, sessionid =
> 0x357067c54424116, negotiated timeout = 120000
> 374820 [main-SendThread(hqhd02nm02.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 96993ms for sessionid 0x357067c5442410e, closing
> socket connection and attempting reconnect
> 374923 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x357067c54424116 closed
> 374924 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 375053 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 375095 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, initiating session
> 375136 [main-SendThread(hqhd02ed01.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqhd02ed01.pclc0.merkle.local/10.129.8.11:2181, sessionid =
> 0x357067c5442410e, negotiated timeout = 120000
> 376538 [main] INFO
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> tion
> - Closing zookeeper sessionid=0x357067c5442410e
> 376581 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x357067c5442410e closed
> 376581 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 376746 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376749 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation
> - io.bytes.per.checksum is deprecated. Instead, use
> dfs.bytes-per-checksum
> 376839 [main] WARN  org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil
> - hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 376841 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation
> - io.bytes.per.checksum is deprecated. Instead, use
> dfs.bytes-per-checksum
> 376843 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=hconnection-0x613cf3c5 connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 376843 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> client connection,
> connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=hconnection-0x613cf3c50x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> baseZNode=/hbase
> 377008 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 377057 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, initiating session
> 377098 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqetghdpadm91.pclc0.merkle.local/10.129.40.11:2181, sessionid =
> 0x2565c552d296982, negotiated timeout = 40000
> 377140 [main] INFO  org.apache.hadoop.hbase.client.ZooKeeperRegistry
> - ClusterId read in ZooKeeper is null
> 377141 [main] INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper
> ensemble=hqetghdpadm91.pclc0.merkle.local:2181,hqetghdpadm90.pclc0.merkle.
> local:2181
> 377141 [main] INFO  org.apache.zookeeper.ZooKeeper  - Initiating
> client connection,
> connectString=hqetghdpadm91.pclc0.merkle.local:2181,
> hqetghdpadm90.pclc0.merkle.local:2181 sessionTimeout=120000
> watcher=TokenUtil-getAuthToken0x0, quorum=hqetghdpadm91.pclc0.
> merkle.local:2181,hqetghdpadm90.pclc0.merkle.local:2181,
> baseZNode=/hbase
> 377189 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 399494 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 399541 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Session establishment complete on
> server hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, sessionid =
> 0x1565c552cfd6b03, negotiated timeout = 40000
> 1036441 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 636835ms for sessionid 0x1565c552cfd6b03, closing
> socket connection and attempting reconnect
> 1036441 [main-SendThread(hqetghdpadm91.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Client session timed out, have not
> heard from server in 636900ms for sessionid 0x2565c552d296982, closing
> socket connection and attempting reconnect
> 1236709 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 1236761 [main-SendThread(hqetghdpadm90.pclc0.merkle.local:2181)] INFO
> org.apache.zookeeper.ClientCnxn  - Socket connection established to
> hqetghdpadm90.pclc0.merkle.local/10.129.40.10:2181, initiating session
> 1236762 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x1565c552cfd6b03 closed
> 1236762 [main] INFO
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementa
> tion
> - Closing zookeeper sessionid=0x2565c552d296982
> 1236762 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
> 1236908 [main] INFO  org.apache.zookeeper.ZooKeeper  - Session:
> 0x2565c552d296982 closed
> 1236909 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn  -
> EventThread shut down
>
> java.io.IOException: Failed to get cluster ID
>
>         at org.apache.hadoop.hbase.security.token.TokenUtil.
> getAuthToken(TokenUtil.java:365)
>         at org.apache.hadoop.hbase.security.token.TokenUtil.
> addTokenForJob(TokenUtil.java:327)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initCredentials(TableMapReduceUtil.java:451)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:658)
>         at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.
> initTableReducerJob(TableMapReduceUtil.java:591)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.
> createSubmittableJob(CopyTable.java:168)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.run(
> CopyTable.java:348)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.hadoop.hbase.mapreduce.CopyTable.main(
> CopyTable.java:341)
>         at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.export(CopyTableTest.java:44)
>         at com.merkleinc.cr.hbase_maintenance.tableexport.
> CopyTableTest.testCopyTable(CopyTableTest.java:25)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> FrameworkMethod.java:50)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(
> ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> FrameworkMethod.java:47)
>         at org.junit.internal.runners.statements.InvokeMethod.
> evaluate(InvokeMethod.java:17)
>         at com.intellij.junit4.JUnit4TestRunnerUtil$
> IgnoreIgnoredTestJUnit4ClassRunner.runChild(JUnit4TestRunnerUtil.java:349)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> BlockJUnit4ClassRunner.java:57)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>         at org.junit.runners.ParentRunner.runChildren(
> ParentRunner.java:288)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>         at org.junit.runners.ParentRunner$2.evaluate(
> ParentRunner.java:268)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>         at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(
> JUnit4IdeaTestRunner.java:69)
>         at com.intellij.rt.execution.junit.JUnitStarter.
> prepareStreamsAndStart(JUnitStarter.java:234)
>         at com.intellij.rt.execution.junit.JUnitStarter.main(
> JUnitStarter.java:74)
>
> Disconnected from the target VM, address: '127.0.0.1:50669', transport:
> 'socket'
>
> Process finished with exit code -1
>
> -----Original Message-----
> From: Esteban Gutierrez [mailto:esteban@cloudera.com]
> Sent: Thursday, September 08, 2016 12:37 PM
> To: user@hbase.apache.org
> Subject: Re: CopyTable fails on copying between two secured clusters
>
> Frank,
>
> Do you have a cross realm setup? if yes, do you have ZK to use the
> proper rules via the -Dzookeeper.security.auth_to_local flag? If you
> could share additional logs that would be helpful for us.
>
> Thanks!
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Thu, Sep 8, 2016 at 10:32 AM, Frank Luo <jluo@merkleinc.com> wrote:
>
> > I couldn’t manage to get the CopyTable to work between two secured
> > clusters and hope someone can shed some light.
> >
> > I have a table created on both clusters, and I am running the
> > CopyTable command from the source cluster.
> >
> > The parameters are the following:
> > "--peer.adr=zookeeper1, zookeeper2:2181:/hbase",
> > "--new.name=TargetTable", "SourceTable"
> >
> > I have ensured that the user has all permissions on both tables.
> > When I run it, I got:
> >
> > =====================
> > java.io.IOException: Failed to get cluster ID
> >
> >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > getAuthToken(TokenUtil.java:365)
> >                at org.apache.hadoop.hbase.security.token.TokenUtil.
> > addTokenForJob(TokenUtil.java:327)
> > =====================
> >
> > Debugging into it, I found that it is trying to get the clusterId
> > from my target cluster using address of zookeeper1/2, and the code
> > below returns
> > null:
> >
> >
> > =====================
> >
> >         ZKUtil.getData(watcher, watcher.clusterIdZNode);
> > =====================
> >
> > Any hints please?
> >
> >
> >
> > Access the Q2 2016 Digital Marketing Report for a fresh set of
> > trends and benchmarks in digital
> > marketing<http://www2.merkleinc.com/l/47252/2016-07-
> > 26/47gt7c>
> >
> > Download our latest report titled “The Case for Change: Exploring
> > the Myths of Customer-Centric Transformation”
> > <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c>
> >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this
> > email in error, please notify the sender immediately and then delete
> > it. If you are not the intended recipient, you must not keep, use,
> > disclose, copy or distribute this email without the author’s prior permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss
> > or damage caused by software viruses. The information contained in
> > this communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Access the Q2 2016 Digital Marketing Report for a fresh set of trends
> and benchmarks in digital
> marketing<http://www2.merkleinc.com/l/47252/2016-07-
> 26/47gt7c>
>
> Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation”
> <http://www2.merkleinc.com/l/ 47252/2016-08-04/4b9p7c> This email and
> any attachments transmitted with it are intended for use by the
> intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are
> not the intended recipient, you must not keep, use, disclose, copy or
> distribute this email without the author’s prior permission. We take
> precautions to minimize the risk of transmitting software viruses, but
> we advise you to perform your own virus checks on any attachment to
> this message. We cannot accept liability for any loss or damage caused
> by software viruses. The information contained in this communication
> may be confidential and may be subject to the attorney-client privilege.
>
Access the Q2 2016 Digital Marketing Report for a fresh set of trends and benchmarks in digital
marketing<http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c>

Download our latest report titled “The Case for Change: Exploring the Myths of Customer-Centric
Transformation” <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c>
This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.
Mime
View raw message