ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Harvey <dhar...@jobcase.com>
Subject Re: slow restart of ignite (with persistence enabled)
Date Tue, 08 May 2018 15:10:41 GMT
You have deactivated the cluster, e.g., using visor?

On Mon, May 7, 2018 at 9:08 PM, Randall Harmon <randall.harmon@sizmek.com>
wrote:

>
> What is the recommended way to signal Ignite to shutdown cleanly, so that
> WAL file doesn't have to be replayed (?) when Ignite restarts?  We are
> loading a large set of data from multiple tables, and we are also
> restarting ignite occasionally (new JARs for job running as ignite
> service).
>
> During this restart, even a single node cluster won't serve cmd=metadata
> on its api port until the WAL scanning(?) process is complete.
>
> (we've experience a similar situation with in-memory storage and clustered
> operation, but one thing at a time...)
>
> During the restart, iostat shows mostly lots of small reads:
>
> > sdb 422.00 4140.00 0.00 4140 0
>
> with occasional tiny writes:
>
> > sdb 462.00 4512.00 128.00 4512 128
>
>
> I'm looking for a way to have Ignite shut down cleanly, or another
> configuration option so that normal restart (SIGINT/SIGTERM) does not cause
> a long restart scenario.   Here's one case that shows 19 minutes of pause
> during a restart.  I've seen it take 40 to 60 minutes too (when 2 or 3
> times as much data has been loaded through it over the course of weeks).
>
> @400000005aeca93c09d3481c 14:40:50.160 [ker-#42] WARN
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
> - Logging at INFO level without checking if INFO level is enabled: Found
> last checkpoint marker [cpId=bcffa2cb-803c-4378-bf52-74b7dedf6285,
> pos=FileWALPointer [idx=658, fileOff=45873219, len=1111281]]
>
> @400000005aecadb123dbc964 14:59:51.600 [rker-#3] WARN
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor -
> Logging at INFO level without checking if INFO level is enabled: Received
> state change finish message: true
>
> Any help suggested so I can restart Ignite when I need to, and have it
> start responding in a reasonable amount of time?
>
> <property name="dataStorageConfiguration">
>     <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
>         <property name="storagePath" value="/srv/var/ignite-data" />
>         <property name="walPath" value="/srv/var/ignite-data/wal" />
>         <property name="walArchivePath" value="/srv/var/ignite-data/wal/archive" />
>
>         <property name="defaultDataRegionConfiguration">
>             <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
>                 <property name="persistenceEnabled" value="true"/>
>                 <property name="maxSize" value="#{5L * 1024 * 1024 * 1024}"/>
>             </bean>
>         </property>
>         <property name="writeThrottlingEnabled" value="true"/>
>     </bean>
> </property>
>
>
>
> Thanks,
> R
>
>

Disclaimer

The information contained in this communication from the sender is confidential. It is intended
solely for use by the recipient and others authorized to receive it. If you are not the recipient,
you are hereby notified that any disclosure, copying, distribution or taking action in relation
of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived
by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer
and more useful place for your human generated data. Specializing in; Security, archiving
and compliance. To find out more visit the Mimecast website.

Mime
View raw message