Mongodb crash after WiredTiger Cannot allocate memory error

Posted on

Question :

I am using MongoDB (3.2.6) as storage for my application and since a couple of weeks MongoDB started to crash usually every 1 or 2 weeks. The crash seems to come from an issue during WiredTiger memory allocation here is the full stacktrace of the last crash:

2016-10-09T18:25:36.389+0200 E STORAGE  [thread1] WiredTiger (12) [1476030336:325695][993:0x7f559c19a700], file:index-1108-8065571460375661294.wt, WT_SESSION.checkpoint: memory allocation of 18624 bytes failed: Cannot allocate memory
2016-10-09T18:25:36.547+0200 E STORAGE  [thread1] WiredTiger (12) [1476030336:422567][993:0x7f559c19a700], file:index-1108-8065571460375661294.wt, WT_SESSION.checkpoint: checkpoints cannot be dropped when in-use: Cannot allocate memory
2016-10-09T18:25:37.347+0200 I COMMAND  [conn10793] command admin.$cmd command: ping { ping: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:37 locks:{} protocol:op_query 208ms
2016-10-09T18:25:37.386+0200 E STORAGE  [thread1] WiredTiger (12) [1476030337:254969][993:0x7f559c19a700], checkpoint-server: checkpoint server error: Cannot allocate memory
2016-10-09T18:25:38.448+0200 E STORAGE  [thread1] WiredTiger (-31804) [1476030337:484838][993:0x7f559c19a700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-10-09T18:25:38.677+0200 I -        [conn10894] Fatal Assertion 28559
2016-10-09T18:25:38.994+0200 I -        [WTJournalFlusher] Fatal Assertion 28559
2016-10-09T18:25:38.994+0200 I -        [TTLMonitor] Fatal Assertion 28559
2016-10-09T18:25:40.978+0200 I -        [thread1] Fatal Assertion 28558
2016-10-09T18:25:41.207+0200 I -        [thread1] 

The server it’s running on has 7Gb RAM and is exclusively used for Mongo.

free -m
             total       used       free     shared    buffers     cached
Mem:          6811       6140        670          0         28        634
-/+ buffers/cache:       5478       1332
Swap:         2047       1009       1038

Several dbs are running on it with a total size around 925MB while WiredTiger cache status is :

"bytes currently in the cache" : 969927022,
"bytes read into cache" : 800513540,
"bytes written from cache" : 1349774879,
"maximum bytes configured" : 3221225472, 

According to the documentation, WiredTiger should use by default

60% of RAM minus 1 GB, or […]

Based on 7GB RAM we get ~ 3.2Gb so it seems good.

Now how can 925Mb databases can bust a 3.2Gb cache ? I know that we have to take the indexes into account but even by indexing every fields (which is not the case) it should not fill the 2+ Gb remaining ? Last crashed occurred yesterday so the provided stats are just after a fresh restart. Could it be that the way I’m using Mongo a memory leak occurs somewhere slowly increasing the memory consumption ? I may completely miss the point as I am new to Mongo so any leading advice or article would be a relief.

I found this issue : but it does not provide any further explanations.

EDIT 12/10:

3 days later the cache status is the following :

"bytes currently in the cache" : 1319874144,
"bytes read into cache" : 1054001990,
"bytes written from cache" : 8936911271,

Answer :

Potentially the unix box is forking the process from your application and causing a duplicate process to try to allocate the same amount of memory but crashing when it does so.

Leave a Reply

Your email address will not be published. Required fields are marked *