hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Hbase stops working
Date Thu, 26 Feb 2009 16:00:05 GMT
I notice in the above that you have file descriptors set to 1024 still.  I'd
suggest reading the 'Getting Started' section:
http://hadoop.apache.org/hbase/docs/r0.19.0/api/overview-summary.html#overview_description
.

How many regions do you have loaded?

On such a small machine, I'd be surprised if you got it all to run together.

St.Ack

On Thu, Feb 26, 2009 at 7:14 AM, Andrew McCall <andrew.mccall@goroam.net>wrote:

> Hi J-D,
>
> I'm running the HBase 0.19 release downloaded to the website on a pretty
> lacklustre linux machine; 1.6Ghz single core processor, 512MB ram with 1.5GB
> swap space enabled and 80GB hard disk. It's a testing machine, I've been
> using for development. It's running Hadoop core 0.19 in pseudo distributed
> mode and running a map/reduce reduce crawl using nutch and HBase. I've been
> planning to move to an EC3 cluster in the near future but I've been putting
> it off since everything seemed to come up and work on the testing box.
>
> It's entirely possible that it is a result of a swap, how do I tell? I've
> looked in all the logs for hadoop, hbase and my job and can't see that
> anything stopped logging around then, the logs all have entries for 9:34.
> I've also increased my lease timeout from the default to 600,000 ms in my
> hbase-site, which is below as a result of another issue. I did search of the
> mailing list for the other issue and answers suggested that the lease
> timeout should be the same as my map/reduce job timeout.
>
> <configuration>
>        <property>
>            <name>hbase.rootdir</name>
>            <value>hdfs://localhost:9000/hbase</value>
>            <description>The directory shared by region servers.
>            </description>
>          </property>
>          <property>
>            <name>hbase.regionserver.lease.period</name>
>            <value>600000</value>
>          </property>
> </configuration>
>
> Thanks for the help,
> Andrew
>
>
>
> On 26 Feb 2009, at 13:41, Jean-Daniel Cryans wrote:
>
>  Andrew,
>>
>> Oh yes sorry, your setup is in local mode so no region server in a
>> different
>> thread. My bad!
>>
>> That lease timeout probably means that you were swapping and a full jvm
>> garbage collecting occurred which blocked the main thread. Can you make
>> sure
>> that it did swap at that moment?
>>
>> It would also be helpful to get the usual info like hardware, OS, hbase
>> version, etc.
>>
>> Thx!
>>
>> J-D
>>
>> On Thu, Feb 26, 2009 at 8:35 AM, Andrew McCall <andrew.mccall@goroam.net
>> >wrote:
>>
>>  Hi J-D,
>>>
>>> I did check the region server logs and they didn't seem to be very useful
>>> at all:
>>>
>>> Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.net
>>> ulimit -n 1024
>>> 2009-02-23 22:11:58,817 WARN
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a
>>> distinct
>>> region server because hbase.master is set to 'local' mode
>>>
>>> Is all that is in them, once for each time I started HBase.
>>>
>>> I saw the shutdown message, which was preceded by this:
>>>
>>> 2009-02-24 09:34:41,579 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:
>>> 127.0.0.1:40076 lease expired
>>>
>>> I can't find any evidence it's related to my map/reduce task, it's the
>>> only
>>> one in the log files and it's suspiciously just before the process
>>> shutdown
>>> message - but, I'm new to this and honestly don't know what to make of
>>> it.
>>>
>>> Andrew
>>>
>>>
>>> On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote:
>>>
>>> Andrew,
>>>
>>>>
>>>> For some reason, your region server shut down around 9:36:
>>>>
>>>> 2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.
>>>>
>>>>
>>>>> master.RegionServerOperation: process shutdown of server
>>>>> 127.0.0.1:40076
>>>>> :
>>>>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,
>>>>>
>>>>>
>>>>
>>>> But just before that the master received a startup message from it. Did
>>>> you
>>>> take a look at your region server log? What does it say around that
>>>> time?
>>>>
>>>> Thx
>>>>
>>>> J-D
>>>>
>>>> On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <
>>>> andrew.mccall@goroam.net
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> I've been testing a nuch on HBase and I'm running into an issue with it
>>>>
>>>>> when it's been running for a while.
>>>>>
>>>>> I'm running hbase 0.19 on a single local host pseudo distributed hadoop
>>>>> 0.19 cluster.
>>>>>
>>>>> After some time I get the following in my logs:
>>>>>
>>>>> 2009-02-24 09:31:18,664 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:31:18,664 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:31:24,293 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:31:24,293 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:31:27,217 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:31:27,217 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:31:55,533 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:32:18,665 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:32:35,749 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:32:38,957 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:32:38,957 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:33:18,771 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:33:54,889 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:33:57,387 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:34:41,579 INFO
>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>> 127.0.0.1:40076 lease expired
>>>>> 2009-02-24 09:34:53,440 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:34:53,441 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:34:53,441 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:35:38,720 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:35:38,720 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:35:56,951 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:36:05,667 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>>>> MSG_CALL_SERVER_STARTUP
>>>>> 2009-02-24 09:36:05,668 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:36:05,884 INFO
>>>>> org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown
>>>>> of
>>>>> server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
>>>>> numberOfMetaRegions: 1,
>>>>> onlineMetaRegions.size(): 1
>>>>> 2009-02-24 09:36:17,689 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>> Splitting 1 log(s) in
>>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>>>>> 2009-02-24 09:36:25,034 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>>>> 076} complete
>>>>> 2009-02-24 09:36:25,034 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:36:36,598 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>>>> log
>>>>> file splitting completed for
>>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>>>>> 2009-02-24 09:36:56,953 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:37:05,279 ERROR
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server
>>>>> handler 4 on 40076, call openScanner([B@1ac5a78, [[B@6799dc,
>>>>> [B@1a66cfc,
>>>>> 9223372036854775807, null) from 127.0.0.1
>>>>> :50139: error: org.apache.hadoop.hbase.NotServingRegionException:
>>>>> .META.,,1
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>> 2009-02-24 09:37:07,051 WARN
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>>>>> 127.0.0.1:40076}
>>>>> org.apache.hadoop.hbase.NotServingRegionException:
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>>
>>>>>    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>>>>>    at
>>>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>>>>>    at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>>>>> 2009-02-24 09:37:07,230 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> All 1 .META. region(s) scanned
>>>>> 2009-02-24 09:37:50,642 INFO
>>>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>>>> Closed .META.,,1
>>>>> 2009-02-24 09:37:56,955 INFO
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>>>> startKey: <>, server: 127.0.0.1:40076}
>>>>> 2009-02-24 09:37:57,126 ERROR
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>>>> Server
>>>>> handler 2 on 40076, call openScanner([B@155bf4e, [[B@82b436,
>>>>> [B@1056757,
>>>>> 9223372036854775807, null) from 127.0.0.1
>>>>> :35804: error: org.apache.hadoop.hbase.NotServingRegionException:
>>>>> .META.,,1
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>> 2009-02-24 09:38:00,412 WARN
>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>>>>> 127.0.0.1:40076}
>>>>> org.apache.hadoop.hbase.NotServingRegionException:
>>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>>>    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>>>    at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>>>
>>>>>    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>>>    at
>>>>>
>>>>>
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>>>>>    at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>>>>>    at
>>>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>>>>>    at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>>>>>
>>>>> This just repeats forever, until I restart HBase.
>>>>>
>>>>> Any help would be greatly appreciated.
>>>>> Andrew
>>>>>
>>>>>
>>>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message