I have a Cron workload running on a slightly under-provisioned server, and when it runs it eats up 80–100% of the available CPU and memory of my AWS t2.micro EC2 instance. The job often worked, but when it crashed, it would take the whole server down with it. My SSH sessions would freeze, and I would have to force stop the instance from the EC2 control panel to get it back online.

I spent a while looking for bugs in my code, but even after clean up and optimization it barely fits within the instance’s resource limits, and the instance still went down periodically.

It was more important to me that the instance stayed alive than that the workload never failed, because the workload would run again — but not if the instance was down.

On desktop Linux, I’ve found that system freezes tend to be the result of not having enough RAM, and in my experience are exacerbated by increasing swappiness.

I remembered something about setting vm.overcommit_memory settings. Maybe this would help it not run out of RAM?

AWS’s default is vm.overcommit_memory=0 (heuristic). Setting it to 1 enables overcommiting, and setting it to 2 disables it.

I set it, temporarily, with sudo sysctl vm.overcommit_memory=2, and then permanently by adding vm.overcommit_memory=2 to the end of /etc/sysctl.conf.

Once this is changed, vm.overcommit_ratio comes into effect. The default is 50%. With the default, a heavy workload wouldn’t run at all. (Because, as I understand it, generally far more memory is committed than is actually used.) I tried various settings. Anything below 150-200, and the workload would never run. And at 200, the workload would still bring down the instance.

If I knew more about the exact details of memory usage and commitment, maybe I would have been able to, but using guesswork, I didn’t find a sweet spot where my resource-intensive workload would run, but wouldn’t take down the host.

Eventually, I found this StackOverflow post, and added swap:

sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
sudo chmod 600 /var/swap.1
sudo /sbin/mkswap /var/swap.1
sudo /sbin/swapon /var/swap.1

Then, sudo vim /etc/fstab and add a new line at the end:

/var/swap.1 swap swap defaults 0 0

(AWS EC2 instances have no swap by default. This may be related to the fact that AWS used to charge for I/O, so swapping could rack up an unpleasant bill.)

My instance hasn’t gone down since, and the cronjob has run successfully every time.