hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Date Sun, 10 Sep 2017 02:02:48 GMT
To your question, I think it is misplaced (at least to me). Go back and read what I wrote you
just today. 

As for the backup feature, once the work for HBASE-15227 is committed and it is resolved,
I think we'd be interested in trying it out, and until then I am not personally interested
in its  inclusion in a release, but that is just me. 


> On Sep 9, 2017, at 5:58 PM, Vladimir Rodionov <vladrodionov@gmail.com> wrote:
> 
> That was I thought. Thanks. Can you tell me that you are not considering AM
> v2 as an unfinished and untested feature? The question to Stack as well.
> 
> On Sat, Sep 9, 2017 at 5:53 PM, Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
> 
>> No all I have to do is pay attention to words you have written yourself in
>> emails and on JIRA. Don't argue with us not to believe our lying eyes,
>> consider finishing the work. I'll be happy to try it out when you indicate
>> it can work if anything happens to fail on the cluster at the time. Until
>> then there are a lot of other things need doing first.
>> 
>> 
>> On Sep 9, 2017, at 5:25 PM, Vladimir Rodionov <vladrodionov@gmail.com>
>> wrote:
>> 
>>>>> but the impression we have is it is unfinished and untested.
>>> To make a conclusion that "feature is not finished and tested"  you have
>>> had to test it at least.
>>> Andrew, If you have discovered issues, why wouldn't you open bug JIRAs?
>>> 
>>> -Vlad
>>> 
>>> On Sat, Sep 9, 2017 at 4:40 PM, Andrew Purtell <andrew.purtell@gmail.com
>>> 
>>> wrote:
>>> 
>>>> For what it's worth, I think AMv2 is the main reason to have a 2.0 in
>> the
>>>> first place, so I would both agree it needs a lot more testing and yet I
>>>> would want us to have a 2.0 release as the vehicle for getting that to
>>>> happen. For other features without testing from a number of parties or
>> at
>>>> scale the value proposition is less clear and it's fine by me for the
>> RM to
>>>> set them aside for future releases.
>>>> 
>>>> Also, I can relay that there is some interest where I work in utilizing
>>>> HBASE-7912 but the impression we have is it is unfinished and untested.
>> So
>>>> for now we are ignoring it and continuing with home grown solutions.
>> Part
>>>> of the problem is fault tolerance was left to the last phase(s) and yet
>> it
>>>> is an essential property for adoption for serious work. The best way to
>>>> resolve this IMHO is for the developers of this feature to complete
>> those
>>>> unfinished JIRAs, especially concerning resilience to failures.
>>>> 
>>>> 
>>>>> On Sep 9, 2017, at 4:11 PM, Vladimir Rodionov <vladrodionov@gmail.com>
>>>> wrote:
>>>>> 
>>>>> Hmm, the next on your list (of kicked out from branch v2) should be AM
>>>> v2 I
>>>>> presume?
>>>>> 
>>>>> -Vlad
>>>>> 
>>>>>> On Sat, Sep 9, 2017 at 4:04 PM, stack <saint.ack@gmail.com>
wrote:
>>>>>> 
>>>>>> In spite of repeated requests for eng summary of state of this feature
>>>> --
>>>>>> summary of what is in 2.0, what is not, what the capabilities are,
how
>>>> well
>>>>>> it has been tested and at what scale -- all I get, when the requests
>> are
>>>>>> not ignored, are pointers to lists of ill-describing jiras and some
>>>> pending
>>>>>> user facing doc update.
>>>>>> 
>>>>>> For other features, mob or region server groups, I know that they
have
>>>> been
>>>>>> running at scale in production for as much as a year and more. I
have
>>>> some
>>>>>> confidence these items basically work.  For backup/restore I have
no
>>>> such
>>>>>> sense even after spending time in review and trying to use the
>> feature.
>>>>>> 
>>>>>> As release manager, I have say over what makes it into a release.
>>>> Unless
>>>>>> the work is done to convince me that backup/restore is more than
a
>> lump
>>>> of
>>>>>> code and a few unit tests that can pass on some fellows laptop, I
am
>>>> going
>>>>>> to kick it out of branch-2.  Let the feature harden more in master
>>>> branch
>>>>>> before it ships in a release.
>>>>>> 
>>>>>> S
>>>>>> 
>>>>>> On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodionov@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>>>> Have I grasped the state of things correctly, Vlad?
>>>>>>> 
>>>>>>> Josh, the only thing which is still pending is doc update. All
other
>>>>>>> features are good to have but not a blockers for 2.0 release.
>>>>>>> 
>>>>>>> -Vlad
>>>>>>> 
>>>>>>> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
>>>>>> vladrodionov@gmail.com
>>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>>>> What testing and at what
>>>>>>>>>> scale has testing been done?
>>>>>>>> 
>>>>>>>> Do we have have that for other features?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
>>>>>>> vladrodionov@gmail.com
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>>>> It asks: "How do I figure what of backup/restore
feature is going
>>>>>> to
>>>>>>>>> be in
>>>>>>>>>>> hbase-2.0.0?
>>>>>>>>> 
>>>>>>>>> Hmm, wait for doc update.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>>>> 
>>>>>>>>>> HBASE-14414 is a JIRA with a list of random seeming
issues w/
>>>>>>>>>> non-descript
>>>>>>>>>> summaries: "Add nonce support to TableBackupProcedure,
BackupID
>> must
>>>>>>>>>> include backup set name, ...". The last comment in
that issue is
>>>> from
>>>>>>>>>> July.
>>>>>>>>>> It asks: "How do I figure what of backup/restore
feature is going
>> to
>>>>>> be
>>>>>>>>>> in
>>>>>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
>>>>>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
>>>>>> name=vrodionov
>>>>>>>>>>> ."
>>>>>>>>>> to which there is no answer.  Doc update is TODO.
>>>>>>>>>> 
>>>>>>>>>> Where is the summary of the capability in hbase-2?
What testing
>> and
>>>>>> at
>>>>>>>>>> what
>>>>>>>>>> scale has testing been done? Is this 'stable or experimental'?
If
>> I
>>>>>>> can't
>>>>>>>>>> get basic info on this feature though I ask repeatedly,
what hope
>>>>>> does
>>>>>>>>>> the
>>>>>>>>>> poor old operator have?
>>>>>>>>>> 
>>>>>>>>>> St.Ack
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov
<
>>>>>>>>>> vladrodionov@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> HBASE-14414
>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <stack@duboce.net>
wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Where do I go to get the current status of
this feature? Looking
>>>>>> in
>>>>>>>>>> JIRA
>>>>>>>>>>> I
>>>>>>>>>>>> see loads of issues open against backup including
some against
>>>>>>>>>>> hbase-2.0.0
>>>>>>>>>>>> and no progress being made that I can discern.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> S
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack
<stack@duboce.net>
>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack
<stack@duboce.net>
>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM,
Vladimir Rodionov <
>>>>>>>>>>>>>> vladrodionov@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> and/or he answered most
of the review feedback
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> No, questions are still open,
but I do not see any blockers
>>>>>> and
>>>>>>>>>> we
>>>>>>>>>>> have
>>>>>>>>>>>>>>> HBASE-16940 to address these
questions.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Agree. No blockers but stuff that
should be dealt with (No one
>>>>>>>>>> will
>>>>>>>>>>> pay
>>>>>>>>>>>>>> me any attention once merge goes
in -- smile).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> Let me clarify the above. I want review
addressed before merge
>>>>>>>>>> happens.
>>>>>>>>>>>>> Sorry if any confusion.
>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:04
PM, Devaraj Das <
>>>>>>>>>> ddas@hortonworks.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Stack, hats off to you
for spending so much time on
>>>>>> this!
>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>> From
>>>>>>>>>>>>>>>> my understanding, Vlad has
raised follow-up jiras for the
>>>>>>>>>> issues
>>>>>>>>>>> you
>>>>>>>>>>>>>>>> raised, and/or he answered
most of the review feedback. So,
>>>>>>> do
>>>>>>>>>> you
>>>>>>>>>>>>>>> think we
>>>>>>>>>>>>>>>> could do a merge vote now?
>>>>>>>>>>>>>>>> Devaraj.
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Vladimir Rodionov <vladrodionov@gmail.com>
>>>>>>>>>>>>>>>> Sent: Monday, November 21,
2016 8:34 PM
>>>>>>>>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>>>>>>>>> Subject: Re: [DISCUSSION]
Merge Backup / Restore - Branch
>>>>>>>>>>> HBASE-7912
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have spent a good
bit of time reviewing and testing
>>>>>> this
>>>>>>>>>>>> feature.
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> like my review and
concerns addressed and I'd like it to
>>>>>>> be
>>>>>>>>>>> clear
>>>>>>>>>>>>>>> how;
>>>>>>>>>>>>>>>>>> either explicit follow-on
issues, pointers to where in
>>>>>> the
>>>>>>>>>> patch
>>>>>>>>>>>> or
>>>>>>>>>>>>>>> doc
>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> remarks have been
catered to, etc. Until then, I am
>>>>>>> against
>>>>>>>>>>>> commit.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Stack, mega patch review
comments will be addressed in the
>>>>>>>>>>> dedicated
>>>>>>>>>>>>>>> JIRA:
>>>>>>>>>>>>>>>> HBASE-16940
>>>>>>>>>>>>>>>> I have open several other
JIRAs to address your other
>>>>>>> comments
>>>>>>>>>> (not
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> review board).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Details are here (end of
the thread):
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Let me know what else should
we do to move merge forward.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -Vlad
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Nov 18, 2016 at 4:54
PM, Stack <stack@duboce.net>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Nov 18, 2016
at 3:53 PM, Ted Yu <
>>>>>>> yuzhihong@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks, Matteo.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> bq. restore is not
clear if given an incremental id it
>>>>>>>>>> will do
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> full
>>>>>>>>>>>>>>>>>> restore from full
up to that point or if i need to
>>>>>> apply
>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>> everything
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The restore takes
into consideration of the dependent
>>>>>>>>>>> backup(s).
>>>>>>>>>>>>>>>>>> So there is no need
to apply preceding backup(s)
>>>>>>> manually.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I ask this question on
the issue. It is not clear from
>>>>>> the
>>>>>>>>>> usage
>>>>>>>>>>> or
>>>>>>>>>>>>>>> doc
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>> to run a restore from
incremental. Can you fix in doc and
>>>>>>>>>> usage
>>>>>>>>>>> how
>>>>>>>>>>>>>>> so I
>>>>>>>>>>>>>>>>> can be clear and try
it. Currently I am stuck verifying a
>>>>>>>>>> round
>>>>>>>>>>>> trip
>>>>>>>>>>>>>>>> backup
>>>>>>>>>>>>>>>>> restore made of incrementals.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Fri, Nov 18, 2016
at 3:48 PM, Matteo Bertozzi <
>>>>>>>>>>>>>>>>> theo.bertozzi@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I did one last
pass to the mega patch. I don't see
>>>>>>>>>> anything
>>>>>>>>>>>> major
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> should block
the merge.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> - most of the
code is isolated in the backup package
>>>>>>>>>>>>>>>>>>> - all the backup
code is client side
>>>>>>>>>>>>>>>>>>> - there are few
changes to the server side, mainly
>>>>>> for
>>>>>>>>>>>> cleaners,
>>>>>>>>>>>>>>> wal
>>>>>>>>>>>>>>>>>>> rolling and similar
(which is ok)
>>>>>>>>>>>>>>>>>>> - there is a
good number of tests, and an integration
>>>>>>>>>> test
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> the code seems
to have still some left overs from the
>>>>>>> old
>>>>>>>>>>>>>>>>> implementation,
>>>>>>>>>>>>>>>>>>> and some stuff
needs a cleanup. but I don't think
>>>>>> this
>>>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> argument to block
the merge. I think the guys will
>>>>>> keep
>>>>>>>>>>> working
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> they may also
get help of others once the patch is in
>>>>>>>>>> master.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I still have
my concerns about the current
>>>>>> limitations,
>>>>>>>>>> but
>>>>>>>>>>>>>>> these are
>>>>>>>>>>>>>>>>>>> things already
planned for phase 3, so some of this
>>>>>>>>>> stuff may
>>>>>>>>>>>>>>> even be
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> the final 2.0.
>>>>>>>>>>>>>>>>>>> but as long as
we have a "current limitations"
>>>>>> section
>>>>>>>>>> in the
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> guide
>>>>>>>>>>>>>>>>>>> mentioning important
stuff like the ones below, I'm
>>>>>> ok
>>>>>>>>>> with
>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>> - if you write
to the table with
>>>>>> Durability.SKIP_WALS
>>>>>>>>>> your
>>>>>>>>>>>> data
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> be in the incremental-backup
>>>>>>>>>>>>>>>>>>> - if you bulkload
files that data will not be in the
>>>>>>>>>>>> incremental
>>>>>>>>>>>>>>>>> backup
>>>>>>>>>>>>>>>>>>> (HBASE-14417)
>>>>>>>>>>>>>>>>>>> - the incremental
backup will not only contains the
>>>>>>>>>> data of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> table
>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> specified but
also the regions from other tables that
>>>>>>>>>> are on
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>> of RSs (HBASE-14141)
...maybe a note about security
>>>>>>>>>> around
>>>>>>>>>>> this
>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>> - the incremental
backup will not contains just the
>>>>>>>>>> "latest
>>>>>>>>>>>> row"
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> backup A and
B, but it will also contains all the
>>>>>>> updates
>>>>>>>>>>>>>>> occurred in
>>>>>>>>>>>>>>>>>>> between. but
the restore does not allow you to
>>>>>> restore
>>>>>>>>>> up to
>>>>>>>>>>> a
>>>>>>>>>>>>>>>> certain
>>>>>>>>>>>>>>>>>>> point in time,
the restore will always be up to the
>>>>>>>>>> "latest
>>>>>>>>>>>>>>> backup
>>>>>>>>>>>>>>>>>> point".
>>>>>>>>>>>>>>>>>>> - you should
limit the number of "incremental" up
>>>>>> to N
>>>>>>>>>> (or
>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>>>> SIZE),
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> avoid replay
time becoming the bottleneck.
>>>>>>> (HBASE-14135)
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I'll be ok even
with the above not being in the final
>>>>>>>>>> 2.0,
>>>>>>>>>>>>>>>>>>> but i'd like
to see as blocker for the final 2.0 (not
>>>>>>> the
>>>>>>>>>>>> merge)
>>>>>>>>>>>>>>>>>>> - the backup
code moved in an hbase-backup module
>>>>>>>>>>>>>>>>>>> - and some more
work around tools, especially to try
>>>>>>> to
>>>>>>>>>>> unify
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>> simple the backup
experience (simple example: in some
>>>>>>>>>> case
>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>> backup_id argument
in others a backupId argument. or
>>>>>>>>>> things
>>>>>>>>>>>>>>> like..
>>>>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>>>> is not clear
if given an incremental id it will do
>>>>>> the
>>>>>>>>>> full
>>>>>>>>>>>>>>> restore
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> full up to that
point or if i need to apply manually
>>>>>>>>>>>> everything).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> in conclusion,
I think we can open a merge vote. I'll
>>>>>>> be
>>>>>>>>>> +1
>>>>>>>>>>> on
>>>>>>>>>>>>>>> it,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> think we should
try to reject -1 with just a "code
>>>>>>>>>> cleanup"
>>>>>>>>>>>>>>>> motivation,
>>>>>>>>>>>>>>>>>>> since there will
still be work going on on the code
>>>>>>>>>> after the
>>>>>>>>>>>>>>> merge.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Matteo
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sun, Nov 6,
2016 at 10:54 PM, Devaraj Das <
>>>>>>>>>>>>>>> ddas@hortonworks.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Stack and
others, anything else on the patch? Merge
>>>>>>> to
>>>>>>>>>>> master
>>>>>>>>>>>>>>> now?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message