BOINC on Ubuntu with time profiles: setting it up on the web server

In the first part of this series we explained why we make our development server’s idle capacity available to BOINC, and which projects we support with it. This part is the technical counterpart: what the setup on the web server actually looks like, which configuration decisions we made, and what went wrong before it was truly stable.

The server

We are working on a server with 8 vCPUs, 16 GB RAM, running Ubuntu 24.04 Noble. The machine is used exclusively for internal web development (Plesk interface, mostly WordPress development): no customer production workloads. That’s the precondition for BOINC being an option at all: any resources we hand over have to be fully available again for development work at any moment.

Installation

BOINC is available in the Ubuntu package repositories:

sudo apt update
sudo apt install boinc-client

The service starts automatically afterwards. Projects are managed through each platform’s web interface: create an account, get the account key, connect the client:

boinccmd --project_attach https://climateprediction.net

We currently run four projects: Climate Prediction, World Community Grid (project MCM1, cancer research), Rosetta@home and Einstein@home. Rosetta@home is a new addition since part one.

The resource problem

BOINC has its own configuration options for CPU and RAM usage. However, the defaults aren’t suitable for a server used for development. Left unrestricted, BOINC would use every available resource: that’s the design goal on desktop systems, but it’s unacceptable on a shared server.

Our requirement: BOINC may use only a little CPU during weekday working hours, considerably more at night and on weekends, but never so much that the server stalls when an important development task comes in. That takes two things: BOINC’s own prefs, which tell the software how much it’s allowed to claim, and a systemd cgroup limit underneath as a hard safety net.

Layer 1: BOINC prefs and time profiles

BOINC reads its resource limits from an XML file (global_prefs_override.xml) in its data directory. We maintain three such files in /etc/boinc-prefs/ and swap them automatically via a cron script.

Day profile (Mon–Fri, 07:00–21:00)

/etc/boinc-prefs/prefs-day.xml:

<global_preferences>
  <max_ncpus_pct>25.000000</max_ncpus_pct>
  <cpu_usage_limit>75.000000</cpu_usage_limit>
  <suspend_cpu_usage>30.000000</suspend_cpu_usage>
  <ram_max_used_busy_frac>0.03</ram_max_used_busy_frac>
  <ram_max_used_idle_frac>0.06</ram_max_used_idle_frac>
  <disk_max_used_gb>10.000000</disk_max_used_gb>
  <disk_min_free_gb>20.000000</disk_min_free_gb>
  <max_bytes_sec_up>500000</max_bytes_sec_up>
  <max_bytes_sec_down>500000</max_bytes_sec_down>
</global_preferences>

max_ncpus_pct=25 works out to roughly two cores on an eight-core machine. suspend_cpu_usage=30 makes BOINC pause as soon as overall server load goes above 30 %: this provides another buffer against unexpected resource competition.

Night and weekend profile

/etc/boinc-prefs/prefs-night.xml and prefs-weekend.xml are identical:

<global_preferences>
  <max_ncpus_pct>70.000000</max_ncpus_pct>
  <cpu_usage_limit>75.000000</cpu_usage_limit>
  <suspend_cpu_usage>30.000000</suspend_cpu_usage>
  <ram_max_used_busy_frac>0.06</ram_max_used_busy_frac>
  <ram_max_used_idle_frac>0.12</ram_max_used_idle_frac>
  <disk_max_used_gb>10.000000</disk_max_used_gb>
  <disk_min_free_gb>20.000000</disk_min_free_gb>
  <max_bytes_sec_up>500000</max_bytes_sec_up>
  <max_bytes_sec_down>500000</max_bytes_sec_down>
</global_preferences>

70% of eight cores is just over 5.5 cores: substantially more compute for the projects when nobody is actively working on the server.

The switch script

/usr/local/bin/boinc-switch-prefs.sh handles moving between profiles:

#!/bin/bash
BOINC_DIR="/var/lib/boinc-client"
PREFS_DIR="/etc/boinc-prefs"
HOUR=$(date +%H)
DAY=$(date +%u) # 1=Mon, 7=Sun

if [ "$DAY" -ge 6 ]; then
SRC="weekend"
elif [ "$HOUR" -ge 21 ] || [ "$HOUR" -lt 7 ]; then
SRC="night"
else
SRC="day"
fi

cp "$PREFS_DIR/prefs-$SRC.xml" "$BOINC_DIR/global_prefs_override.xml"
chown boinc:boinc "$BOINC_DIR/global_prefs_override.xml"
boinccmd --read_global_prefs_override

The script works out the current time context, copies the matching prefs file and tells BOINC to read it in immediately. It’s triggered by cron:

0 7 * * 1-5 /usr/local/bin/boinc-switch-prefs.sh # Mon–Fri day profile
0 21 * * 1-5 /usr/local/bin/boinc-switch-prefs.sh # Mon–Fri night profile
0 0 * * 6 /usr/local/bin/boinc-switch-prefs.sh # Sat 00:00 weekend
0 7 * * 1 /usr/local/bin/boinc-switch-prefs.sh # Mon 07:00 back to day

Layer 2: systemd cgroup as a hard limit

The BOINC prefs are a cooperative agreement: BOINC sticks to them as long as the software behaves correctly. For a hard boundary that holds regardless of how the BOINC client behaves, we set up a systemd drop-in.

/etc/systemd/system/boinc-client.service.d/limits.conf:

ini

[Service]
CPUQuota=75%
MemoryMax=8G
IOWeight=10

CPUQuota=75% caps CPU time independently of whatever BOINC calculates internally. MemoryMax=8G is the memory limit at the cgroup level: the kernel terminates BOINC processes before they can exceed it. IOWeight=10 gives the service a very low I/O priority, so disk access from deployments or database operations is never blocked by BOINC.

After changing this file:

sudo systemctl daemon-reload
sudo systemctl restart boinc-client

A pitfall with the RAM fractions

Before the setup was stable, we had problems with OOM kills: Einstein@home was being terminated by the kernel in the middle of the night. The configuration looked correct, so the fault had to lay deeper.

ram_max_used_busy_frac and ram_max_used_idle_frac in the BOINC prefs are not absolute megabyte values: they are fractions of the system’s total RAM. With 16 GB and a value set too high, BOINC had allowed itself a working set of nearly 4 GB, which is more than the cgroup limit at the time permitted. As a result, the kernel killed the process.

The fix had two parts: raise MemoryMax to 8G, and set the ram_max_used_*_frac values conservatively: low enough that BOINC allows itself less than the cgroup limit permits.

The prefs are the first layer: BOINC shouldn’t even attempt to use more RAM than intended. The cgroup limit is the second: if BOINC does take more anyway – through an update, a misconfiguration, or an unexpectedly memory-hungry work unit – the kernel steps in before the rest of the server is affected.

To illustrate with the current values: with 16 GB total RAM and ram_max_used_idle_frac=0.06, BOINC allows itself at most 0.06 × 16,384 MB = ~983 MB when idle. With ram_max_used_busy_frac=0.03 it’s ~491 MB under load. Both sit well below the cgroup limit of 8G, which is exactly the point.

The setup has run without trouble ever since. BOINC switches between profiles automatically, the systemd cgroup prevents uncontrolled memory growth, and the resources for development work remain untouched.