Mailu: ClamAV eats up all the memory

I use Mailu as a simple mail server to manage the mails for one of my domain. This application uses ClamAv as an antivirus solution. To host this application I use VPS with only 4 GB RAM. Few weeks ago my Zabbix started sending notifications once a day that I have OOM on the mail server. It was strange as the problem appeared only once per day in the +/- same time. I tried to investigate the problem and found that the reason was in ClamAV...

As usual I started form the logs and found the next information in "/var/log/message" log

I compared the time from these logs with the time of OOM notifications with Zabbix. I found that I received notifications each time when I have same problem in log on server

Next, I created simple logger to collect the stats of the docker containers to understand was clamd process the root cause of OOM or some other process consumed the memory, and clamd was killed as consumed more RAM than others.

Next, time when I observed the problem I checked the log and found that antivirus container consumed all the memory, so it confirmed that the main consumer is ClamAV process

Here we can observe that normal memory consumption for mailu-antivirus-1 container was around 37%, but in some moment it grows to 57% and then to 67% and then drops to 3%.

Other containers consumption in the same time

I checked configuration of ClamAV inside the antivirus container

Check freshclam.conf

Check clamd.conf

I didn't find any options that can lead to such behavior.

Then I found on the internet that in version 0.103.0 ClamAV added "Non-blocking database reloads" which can increase memory usage up to x2 during the virus databases update
- https://blog.clamav.net/2020/09/clamav-01030-released.html

Major changes

• clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.
  • Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads, as it will temporarily consume double the amount of memory. We added a new clamd config option ConcurrentDatabaseReload, which may be set to no.

This looked very similar to my case as I didn't have this option in the configuration which meant it was enabled by default.

As ClamAV release notes said it can be disabled by adding next option to clamd config

then you need to restart clamav

But I decided to keep previous configuration and increase swap file instead (as it was set to 0), so some other process can avoid problem if they need a bit more memory for short period.