hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Using Spark on Hive with Hive also using Spark as its execution engine
Date Sun, 29 May 2016 19:19:14 GMT
Very interesting do you plan also a test with TEZ?

> On 29 May 2016, at 13:40, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
> 
> Hi,
> 
> I did another study of Hive using Spark engine compared to Hive with MR.
> 
> Basically took the original table imported using Sqoop and created and populated a new
ORC table partitioned by year and month into 48 partitions as follows:
> 
> <sales_partition.PNG>
> ​ 
> Connections use JDBC via beeline. Now for each partition using MR it takes an average
of 17 minutes as seen below for each PARTITION..  Now that is just an individual partition
and there are 48 partitions.
> 
> In contrast doing the same operation with Spark engine took 10 minutes all inclusive.
I just gave up on MR. You can see the StartTime and FinishTime from below
> 
> <image.png>
> 
> This is by no means indicate that Spark is much better than MR but shows that some very
good results can ve achieved using Spark engine.
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
>> On 24 May 2016 at 08:03, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
>> Hi,
>> 
>> We use Hive as the database and use Spark as an all purpose query tool.
>> 
>> Whether Hive is the write database for purpose or one is better off with something
like Phoenix on Hbase, well the answer is it depends and your mileage varies. 
>> 
>> So fit for purpose.
>> 
>> Ideally what wants is to use the fastest  method to get the results. How fast we
confine it to our SLA agreements in production and that helps us from unnecessary further
work as we technologists like to play around.
>> 
>> So in short, we use Spark most of the time and use Hive as the backend engine for
data storage, mainly ORC tables.
>> 
>> We use Hive on Spark and with Hive 2 on Spark 1.3.1 for now we have a combination
that works. Granted it helps to use Hive 2 on Spark 1.6.1 but at the moment it is one of my
projects.
>> 
>> We do not use any vendor's products as it enables us to move away  from being tied
down after years of SAP, Oracle and MS dependency to yet another vendor. Besides there is
some politics going on with one promoting Tez and another Spark as a backend. That is fine
but obviously we prefer an independent assessment ourselves.
>> 
>> My gut feeling is that one needs to look at the use case. Recently we had to import
a very large table from Oracle to Hive and decided to use Spark 1.6.1 with Hive 2 on Spark
1.3.1 and that worked fine. We just used JDBC connection with temp table and it was good.
We could have used sqoop but decided to settle for Spark so it all depends on use case.
>> 
>> HTH
>> 
>> 
>> 
>> Dr Mich Talebzadeh
>>  
>> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>  
>> http://talebzadehmich.wordpress.com
>>  
>> 
>>> On 24 May 2016 at 03:11, ayan guha <guha.ayan@gmail.com> wrote:
>>> Hi
>>> 
>>> Thanks for very useful stats. 
>>> 
>>> Did you have any benchmark for using Spark as backend engine for Hive vs using
Spark thrift server (and run spark code for hive queries)? We are using later but it will
be very useful to remove thriftserver, if we can. 
>>> 
>>>> On Tue, May 24, 2016 at 9:51 AM, Jörn Franke <jornfranke@gmail.com>
wrote:
>>>> 
>>>> Hi Mich,
>>>> 
>>>> I think these comparisons are useful. One interesting aspect could be hardware
scalability in this context. Additionally different type of computations. Furthermore, one
could compare Spark and Tez+llap as execution engines. I have the gut feeling that  each one
can be justified by different use cases.
>>>> Nevertheless, there should be always a disclaimer for such comparisons, because
Spark and Hive are not good for a lot of concurrent lookups of single rows. They are not good
for frequently write small amounts of data (eg sensor data). Here hbase could be more interesting.
Other use cases can justify graph databases, such as Titan, or text analytics/ data matching
using Solr on Hadoop.
>>>> Finally, even if you have a lot of data you need to think if you always have
to process everything. For instance, I have found valid use cases in practice where we decided
to evaluate 10 machine learning models in parallel on only a sample of data and only evaluate
the "winning" model of the total of data.
>>>> 
>>>> As always it depends :) 
>>>> 
>>>> Best regards
>>>> 
>>>> P.s.: at least Hortonworks has in their distribution spark 1.5 with hive
1.2 and spark 1.6 with hive 1.2. Maybe they have somewhere described how to manage bringing
both together. You may check also Apache Bigtop (vendor neutral distribution) on how they
managed to bring both together.
>>>> 
>>>>> On 23 May 2016, at 01:42, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:
>>>>> 
>>>>> Hi,
>>>>>  
>>>>> I have done a number of extensive tests using Spark-shell with Hive DB
and ORC tables.
>>>>>  
>>>>> Now one issue that we typically face is and I quote:
>>>>>  
>>>>> Spark is fast as it uses Memory and DAG. Great but when we save data
it is not fast enough
>>>>> 
>>>>> OK but there is a solution now. If you use Spark with Hive and you are
on a descent version of Hive >= 0.14, then you can also deploy Spark as execution engine
for Hive. That will make your application run pretty fast as you no longer rely on the old
Map-Reduce for Hive engine. In a nutshell what you are gaining speed in both querying and
storage.
>>>>>  
>>>>> I have made some comparisons on this set-up and I am sure some of you
will find it useful.
>>>>>  
>>>>> The version of Spark I use for Spark queries (Spark as query tool) is
1.6.
>>>>> The version of Hive I use in Hive 2
>>>>> The version of Spark I use as Hive execution engine is 1.3.1 It works
and frankly Spark 1.3.1 as an execution engine is adequate (until we sort out the Hadoop libraries
mismatch).
>>>>>  
>>>>> An example I am using Hive on Spark engine to find the min and max of
IDs for a table with 1 billion rows:
>>>>>  
>>>>> 0: jdbc:hive2://rhes564:10010/default>  select min(id), max(id),avg(id),
stddev(id) from oraclehadoop.dummy;
>>>>> Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006
>>>>>  
>>>>>  
>>>>> Starting Spark Job = 5e092ef9-d798-4952-b156-74df49da9151
>>>>>  
>>>>> INFO  : Completed compiling command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006);
Time taken: 1.911 seconds
>>>>> INFO  : Executing command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006):
select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>>>> INFO  : Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006
>>>>> INFO  : Total jobs = 1
>>>>> INFO  : Launching Job 1 out of 1
>>>>> INFO  : Starting task [Stage-1:MAPRED] in serial mode
>>>>>  
>>>>> Query Hive on Spark job[0] stages:
>>>>> 0
>>>>> 1
>>>>> Status: Running (Hive on Spark job[0])
>>>>> Job Progress Format
>>>>> CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
[StageCost]
>>>>> 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1
>>>>> 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1
>>>>> 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1
>>>>> 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22    Stage-1_0: 0/1
>>>>> INFO  :
>>>>> Query Hive on Spark job[0] stages:
>>>>> INFO  : 0
>>>>> INFO  : 1
>>>>> INFO  :
>>>>> Status: Running (Hive on Spark job[0])
>>>>> INFO  : Job Progress Format
>>>>> CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
[StageCost]
>>>>> INFO  : 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1
>>>>> INFO  : 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1
>>>>> INFO  : 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22    Stage-1_0: 0/1
>>>>> INFO  : 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22    Stage-1_0: 0/1
>>>>> 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished       Stage-1_0: 0(+1)/1
>>>>> 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished       Stage-1_0: 1/1
Finished
>>>>> Status: Finished successfully in 53.25 seconds
>>>>> OK
>>>>> INFO  : 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished       Stage-1_0:
0(+1)/1
>>>>> INFO  : 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished       Stage-1_0:
1/1 Finished
>>>>> INFO  : Status: Finished successfully in 53.25 seconds
>>>>> INFO  : Completed executing command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006);
Time taken: 56.337 seconds
>>>>> INFO  : OK
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> | c0  |     c1     |      c2       |          c3           |
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> | 1   | 100000000  | 5.00000005E7  | 2.8867513459481288E7  |
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> 1 row selected (58.529 seconds)
>>>>>  
>>>>> 58 seconds first run with cold cache is pretty good
>>>>>  
>>>>> And let us compare it with running the same query on map-reduce engine
>>>>>  
>>>>> : jdbc:hive2://rhes564:10010/default> set hive.execution.engine=mr;
>>>>> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X
releases.
>>>>> No rows affected (0.007 seconds)
>>>>> 0: jdbc:hive2://rhes564:10010/default>  select min(id), max(id),avg(id),
stddev(id) from oraclehadoop.dummy;
>>>>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available
in the future versions. Consider using a different execution engine (i.e. spark, tez) or using
Hive 1.X releases.
>>>>> Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc
>>>>> Total jobs = 1
>>>>> Launching Job 1 out of 1
>>>>> Number of reduce tasks determined at compile time: 1
>>>>> In order to change the average load for a reducer (in bytes):
>>>>>   set hive.exec.reducers.bytes.per.reducer=<number>
>>>>> In order to limit the maximum number of reducers:
>>>>>   set hive.exec.reducers.max=<number>
>>>>> In order to set a constant number of reducers:
>>>>>   set mapreduce.job.reduces=<number>
>>>>> Starting Job = job_1463956731753_0005, Tracking URL = http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>>>> Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job  -kill job_1463956731753_0005
>>>>> Hadoop job information for Stage-1: number of mappers: 22; number of
reducers: 1
>>>>> 2016-05-23 00:26:38,127 Stage-1 map = 0%,  reduce = 0%
>>>>> INFO  : Compiling command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc):
select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>>>> INFO  : Semantic Analysis Completed
>>>>> INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:c0,
type:int, comment:null), FieldSchema(name:c1, type:int, comment:null), FieldSchema(name:c2,
type:double, comment:null), FieldSchema(name:c3, type:double, comment:null)], properties:null)
>>>>> INFO  : Completed compiling command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc);
Time taken: 0.144 seconds
>>>>> INFO  : Executing command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc):
select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>>>> WARN  : Hive-on-MR is deprecated in Hive 2 and may not be available in
the future versions. Consider using a different execution engine (i.e. spark, tez) or using
Hive 1.X releases.
>>>>> INFO  : WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available
in the future versions. Consider using a different execution engine (i.e. spark, tez) or using
Hive 1.X releases.
>>>>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available
in the future versions. Consider using a different execution engine (i.e. spark, tez) or using
Hive 1.X releases.
>>>>> INFO  : Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc
>>>>> INFO  : Total jobs = 1
>>>>> INFO  : Launching Job 1 out of 1
>>>>> INFO  : Starting task [Stage-1:MAPRED] in serial mode
>>>>> INFO  : Number of reduce tasks determined at compile time: 1
>>>>> INFO  : In order to change the average load for a reducer (in bytes):
>>>>> INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
>>>>> INFO  : In order to limit the maximum number of reducers:
>>>>> INFO  :   set hive.exec.reducers.max=<number>
>>>>> INFO  : In order to set a constant number of reducers:
>>>>> INFO  :   set mapreduce.job.reduces=<number>
>>>>> WARN  : Hadoop command-line option parsing not performed. Implement the
Tool interface and execute your application with ToolRunner to remedy this.
>>>>> INFO  : number of splits:22
>>>>> INFO  : Submitting tokens for job: job_1463956731753_0005
>>>>> INFO  : The url to track the job: http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>>>> INFO  : Starting Job = job_1463956731753_0005, Tracking URL = http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>>>> INFO  : Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job  -kill
job_1463956731753_0005
>>>>> INFO  : Hadoop job information for Stage-1: number of mappers: 22; number
of reducers: 1
>>>>> INFO  : 2016-05-23 00:26:38,127 Stage-1 map = 0%,  reduce = 0%
>>>>> 2016-05-23 00:26:44,367 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU
4.56 sec
>>>>> INFO  : 2016-05-23 00:26:44,367 Stage-1 map = 5%,  reduce = 0%, Cumulative
CPU 4.56 sec
>>>>> 2016-05-23 00:26:50,558 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU
9.17 sec
>>>>> INFO  : 2016-05-23 00:26:50,558 Stage-1 map = 9%,  reduce = 0%, Cumulative
CPU 9.17 sec
>>>>> 2016-05-23 00:26:56,747 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU
14.04 sec
>>>>> INFO  : 2016-05-23 00:26:56,747 Stage-1 map = 14%,  reduce = 0%, Cumulative
CPU 14.04 sec
>>>>> 2016-05-23 00:27:02,944 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU
18.64 sec
>>>>> INFO  : 2016-05-23 00:27:02,944 Stage-1 map = 18%,  reduce = 0%, Cumulative
CPU 18.64 sec
>>>>> 2016-05-23 00:27:08,105 Stage-1 map = 23%,  reduce = 0%, Cumulative CPU
23.25 sec
>>>>> INFO  : 2016-05-23 00:27:08,105 Stage-1 map = 23%,  reduce = 0%, Cumulative
CPU 23.25 sec
>>>>> 2016-05-23 00:27:14,298 Stage-1 map = 27%,  reduce = 0%, Cumulative CPU
27.84 sec
>>>>> INFO  : 2016-05-23 00:27:14,298 Stage-1 map = 27%,  reduce = 0%, Cumulative
CPU 27.84 sec
>>>>> 2016-05-23 00:27:20,484 Stage-1 map = 32%,  reduce = 0%, Cumulative CPU
32.56 sec
>>>>> INFO  : 2016-05-23 00:27:20,484 Stage-1 map = 32%,  reduce = 0%, Cumulative
CPU 32.56 sec
>>>>> 2016-05-23 00:27:26,659 Stage-1 map = 36%,  reduce = 0%, Cumulative CPU
37.1 sec
>>>>> INFO  : 2016-05-23 00:27:26,659 Stage-1 map = 36%,  reduce = 0%, Cumulative
CPU 37.1 sec
>>>>> 2016-05-23 00:27:32,839 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU
41.74 sec
>>>>> INFO  : 2016-05-23 00:27:32,839 Stage-1 map = 41%,  reduce = 0%, Cumulative
CPU 41.74 sec
>>>>> 2016-05-23 00:27:39,003 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU
46.32 sec
>>>>> INFO  : 2016-05-23 00:27:39,003 Stage-1 map = 45%,  reduce = 0%, Cumulative
CPU 46.32 sec
>>>>> 2016-05-23 00:27:45,173 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU
50.93 sec
>>>>> 2016-05-23 00:27:50,316 Stage-1 map = 55%,  reduce = 0%, Cumulative CPU
55.55 sec
>>>>> INFO  : 2016-05-23 00:27:45,173 Stage-1 map = 50%,  reduce = 0%, Cumulative
CPU 50.93 sec
>>>>> INFO  : 2016-05-23 00:27:50,316 Stage-1 map = 55%,  reduce = 0%, Cumulative
CPU 55.55 sec
>>>>> 2016-05-23 00:27:56,482 Stage-1 map = 59%,  reduce = 0%, Cumulative CPU
60.25 sec
>>>>> INFO  : 2016-05-23 00:27:56,482 Stage-1 map = 59%,  reduce = 0%, Cumulative
CPU 60.25 sec
>>>>> 2016-05-23 00:28:02,642 Stage-1 map = 64%,  reduce = 0%, Cumulative CPU
64.86 sec
>>>>> INFO  : 2016-05-23 00:28:02,642 Stage-1 map = 64%,  reduce = 0%, Cumulative
CPU 64.86 sec
>>>>> 2016-05-23 00:28:08,814 Stage-1 map = 68%,  reduce = 0%, Cumulative CPU
69.41 sec
>>>>> INFO  : 2016-05-23 00:28:08,814 Stage-1 map = 68%,  reduce = 0%, Cumulative
CPU 69.41 sec
>>>>> 2016-05-23 00:28:14,977 Stage-1 map = 73%,  reduce = 0%, Cumulative CPU
74.06 sec
>>>>> INFO  : 2016-05-23 00:28:14,977 Stage-1 map = 73%,  reduce = 0%, Cumulative
CPU 74.06 sec
>>>>> 2016-05-23 00:28:21,134 Stage-1 map = 77%,  reduce = 0%, Cumulative CPU
78.72 sec
>>>>> INFO  : 2016-05-23 00:28:21,134 Stage-1 map = 77%,  reduce = 0%, Cumulative
CPU 78.72 sec
>>>>> 2016-05-23 00:28:27,282 Stage-1 map = 82%,  reduce = 0%, Cumulative CPU
83.32 sec
>>>>> INFO  : 2016-05-23 00:28:27,282 Stage-1 map = 82%,  reduce = 0%, Cumulative
CPU 83.32 sec
>>>>> 2016-05-23 00:28:33,437 Stage-1 map = 86%,  reduce = 0%, Cumulative CPU
87.9 sec
>>>>> INFO  : 2016-05-23 00:28:33,437 Stage-1 map = 86%,  reduce = 0%, Cumulative
CPU 87.9 sec
>>>>> 2016-05-23 00:28:38,579 Stage-1 map = 91%,  reduce = 0%, Cumulative CPU
92.52 sec
>>>>> INFO  : 2016-05-23 00:28:38,579 Stage-1 map = 91%,  reduce = 0%, Cumulative
CPU 92.52 sec
>>>>> 2016-05-23 00:28:44,759 Stage-1 map = 95%,  reduce = 0%, Cumulative CPU
97.35 sec
>>>>> INFO  : 2016-05-23 00:28:44,759 Stage-1 map = 95%,  reduce = 0%, Cumulative
CPU 97.35 sec
>>>>> 2016-05-23 00:28:49,915 Stage-1 map = 100%,  reduce = 0%, Cumulative
CPU 99.6 sec
>>>>> INFO  : 2016-05-23 00:28:49,915 Stage-1 map = 100%,  reduce = 0%, Cumulative
CPU 99.6 sec
>>>>> 2016-05-23 00:28:54,043 Stage-1 map = 100%,  reduce = 100%, Cumulative
CPU 101.4 sec
>>>>> MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec
>>>>> Ended Job = job_1463956731753_0005
>>>>> MapReduce Jobs Launched:
>>>>> Stage-Stage-1: Map: 22  Reduce: 1   Cumulative CPU: 101.4 sec   HDFS
Read: 5318569 HDFS Write: 46 SUCCESS
>>>>> Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec
>>>>> OK
>>>>> INFO  : 2016-05-23 00:28:54,043 Stage-1 map = 100%,  reduce = 100%, Cumulative
CPU 101.4 sec
>>>>> INFO  : MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400
msec
>>>>> INFO  : Ended Job = job_1463956731753_0005
>>>>> INFO  : MapReduce Jobs Launched:
>>>>> INFO  : Stage-Stage-1: Map: 22  Reduce: 1   Cumulative CPU: 101.4 sec
  HDFS Read: 5318569 HDFS Write: 46 SUCCESS
>>>>> INFO  : Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec
>>>>> INFO  : Completed executing command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc);
Time taken: 142.525 seconds
>>>>> INFO  : OK
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> | c0  |     c1     |      c2       |          c3           |
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> | 1   | 100000000  | 5.00000005E7  | 2.8867513459481288E7  |
>>>>> +-----+------------+---------------+-----------------------+--+
>>>>> 1 row selected (142.744 seconds)
>>>>>  
>>>>> OK Hive on map-reduce engine took 142 seconds compared to 58 seconds
with Hive on Spark. So you can obviously gain pretty well by using Hive on Spark.
>>>>>  
>>>>> Please also note that I did not use any vendor's build for this purpose.
I compiled Spark 1.3.1 myself.
>>>>>  
>>>>> HTH
>>>>>  
>>>>>  
>>>>> Dr Mich Talebzadeh
>>>>>  
>>>>> LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>  
>>>>> http://talebzadehmich.wordpress.com/
>>> 
>>> 
>>> 
>>> -- 
>>> Best Regards,
>>> Ayan Guha
> 

Mime
View raw message