Cron, and the jobs that never ran

If you just want a crontab checked against these, paste it into our cron expression parser; this article is the why behind its findings. The syntax side, special characters and dialects live in the cron cheatsheet.

The timezone cron actually uses

Cron runs on the system timezone of the machine hosting the daemon, read from /etc/localtime. Not your user timezone, not TZ from your shell, not whatever the app config says. And practically every base image (debian, alpine, the official language images) is UTC, so the job you tested at 03:00 on a laptop in Vienna fires at 05:00 local once it is deployed. In summer. In winter it is 04:00, which is how people discover this in the first place.

Vixie-descendant crons and cronie support a CRON_TZ=Europe/Vienna line placed above the entries it should apply to. It is a real fix and it is per crontab, so different jobs can live in different zones. Two caveats: CRON_TZ is not POSIX, and it changes nothing about the environment your command runs in, so a Python script that calls datetime.now() still sees UTC. Set TZ separately if the command itself cares.

After changing the system timezone, restart the daemon. Cron caches the offset and a running daemon that thinks it is still in the old zone produces schedules nobody can explain.

DST: the 02:30 job that runs twice, or not at all

Everyone knows the problem: schedule something at 02:30 local, and on the March switch night 02:30 does not exist, on the October switch night it happens twice. What almost nobody knows is that Vixie cron and cronie handle this, and the documented rule is more specific than the folklore.

From crontab(5) and cron(8): for clock changes of less than three hours, if the time moves forward, jobs that would have run in the skipped interval run once soon after the change. If the time moves backward, jobs that fall into the repeated interval are not re-run. So on Debian, Ubuntu, RHEL and Fedora your 02:30 job runs exactly once on both switch nights. That is the good news.

The catch is in the exemption. This correction applies only to jobs that run at a particular time. Jobs with * in the hour or minute field, and @hourly, are deliberately left alone, on the reasoning that a job running every few minutes does not care about a one-hour jump. Which means:

The middle row is the one that bites. Put anything sensitive to double execution outside the 01:00 to 03:00 window, or run the daemon in UTC and do the local-time reasoning in the application. A job at 04:00 UTC has exactly 365 runs a year, every year, in every jurisdiction, and that is worth more than a schedule that reads nicely. The same reasoning applies one level up in your data model, which we go through in how to store timestamps without regrets.

Day-of-month vs day-of-week: the OR nobody expects

Cron has two day fields, and they do not combine the way every other field does. POSIX puts it plainly: if both the day-of-month and the day-of-week field are restricted (neither is *), the command runs when either matches.

So the classic Friday-the-13th expression:

That second one is the expensive version. Somebody wanted a monthly report, wrote what looked like "the 1st, on a Monday", and shipped a job that runs five times a month. Restricting both fields widens the schedule, it never narrows it.

There is no cron syntax that expresses AND. You schedule the broader field and test the narrower one in the command, for example 0 0 13 * * with a guard on the weekday. The cron parser computes both readings for any expression, the OR that will run and the AND that was meant, as days per year. Just watch the percent sign when you write the guard, which is the next trap.

The % that eats your command

In a crontab, % is not a character, it is syntax. crontab(5): a percent sign in the command, unless escaped with a backslash, is turned into a newline, and everything after the first % is fed to the command as standard input.

So this line:

The failure is loud in the sense that the command breaks, and silent in the sense that you will never see the error unless you sorted out mail or logging first (see below). Any date +%Y-%m-%d, any printf '%s\n', any awk or curl format string with a percent in it: escape every one of them. Honestly, the moment a crontab line contains a pipe, a redirect and a date format, move it into a shell script and put the script path in the crontab. Cron lines are a terrible place to write shell.

While we are on syntax that silently discards work: the crontab file has to end with a newline. Vixie cron treats a file whose last entry is terminated by EOF as broken, and depending on version and distribution you get either a "premature EOF" error or a last line that is simply never executed. It comes up constantly with generated crontabs, config management templates and anything assembled by a script that trims trailing whitespace. Add the empty line.

Works in my shell, not in cron

Cron does not source anything. No /etc/profile, no .bashrc, no .profile. Vixie cron sets SHELL=/bin/sh, LOGNAME and HOME from /etc/passwd, and a PATH that crontab(5) documents as /usr/bin:/bin. That is the whole environment.

Consequences, in rough order of how often they burn people:

• Your interpreter is missing. nvm, pyenv, rbenv and asdf all work by putting shims on PATH in a shell rc file. Cron never reads those files, so node and python either resolve to the system version or not at all. Use the absolute path to the real binary, the one readlink -f $(which node) prints.
• /bin/sh is not bash. On Debian and Ubuntu it has been dash since Ubuntu 6.10, so [[ ]], arrays and source are syntax errors. Either set SHELL=/bin/bash at the top of the crontab or put #!/bin/bash in the script and call the script.
• The working directory is $HOME. Relative paths in the script resolve somewhere you did not intend. cd first, or make every path absolute.
• No locale. LANG and LC_ALL are unset, so sorting, decimal separators and date formatting differ from your terminal. Worth knowing before you debug a report whose numbers changed shape.

Two more environment-adjacent traps. Files in /etc/cron.d and /etc/crontab have an extra field: the user the job runs as, between the schedule and the command. Paste a user crontab line in there and cron reads your command name as a username. And the drop-in directories run through run-parts, which by default only executes filenames made of letters, digits, underscores and hyphens. A script called backup.sh in /etc/cron.daily is skipped because of the dot, without a word in any log. Drop the extension.

The reflex fix for "cron does not run my script" is chmod 777. Don't; chmod +x is what you actually want, and we wrote down why 777 is worse than it looks.

Overlapping runs and flock

Cron has no idea whether the previous run finished. A job scheduled */5 that occasionally takes 20 minutes will happily have four copies running at once, and the day the database is slow you get a pile-up that makes it slower.

The standard answer is flock from util-linux, present on essentially every Linux box:

That third variant matters more than it looks. Without -E 0 (the conflict exit code flag), every skipped run exits non-zero, which means an alerting setup that watches exit codes pages you for the thing working as designed. Two things to know about flock: the lock lives on the file descriptor, so it is released automatically when the process dies, no stale lock files to clean up, and it is not dependable over NFS, so keep the lock file on local disk.

Where the error messages went

Cron mails anything a job writes to stdout or stderr to the crontab owner. On a 1995 workstation with a local MTA that was a decent design. On a container or a minimal cloud VM there is no MTA, so the output is discarded, and the only trace is a line in the log saying the command started. Not that it succeeded. Not what it printed.

This is why "the cron job silently does nothing" is the single most common cron report. The job is failing loudly, into a void. Three settings worth knowing:

• MAILTO=ops@example.com at the top of the crontab sends output somewhere real, assuming a working MTA.
• MAILTO="" disables mail for the entries below it. Useful for deliberately chatty jobs, dangerous as a blanket default because it also throws away errors.
• Redirecting yourself is the version we use: >> /var/log/job.log 2>&1. Order matters, 2>&1 >> file sends stderr to the old stdout and does not do what you want.

Note also that > /dev/null 2>&1, which the internet pastes into every example line, throws away the error messages too. It is the reason for half the mystery failures. If you keep it, add a dead-man switch: have the job ping a monitoring URL as its last step, and alert when the ping stops arriving. Cron cannot tell you that a job did not run, so something outside cron has to.

Kubernetes CronJob specifics

CronJob uses the same five-field syntax, so every trap above applies, plus a few of its own.

• Quote the schedule. In YAML a leading * is an alias indicator, so an unquoted schedule: */5 * * * * is a parse error, not a scheduling bug. Same family of problem as the Norway problem, where YAML decides your value is a boolean. If you are unsure how a manifest parses, run it through our YAML to JSON converter and look at what the parser actually produced.
• Timezone. Without .spec.timeZone the schedule is interpreted in the timezone of the kube-controller-manager, which is UTC on every managed control plane we have seen. The timeZone field takes an IANA name and went stable in Kubernetes 1.27.
• The 100 missed schedules rule. The controller counts how many schedules it missed since the last one it started. Past 100, it gives up on that CronJob permanently and logs one line: "Cannot determine if job needs to be started: too many missed start time (> 100)". Nothing restarts it. A controller outage, a long suspend or clock skew is enough to reach it, and with a one-minute schedule that takes 100 minutes.
• startingDeadlineSeconds. If set, the controller counts missed schedules within that window instead of since the last start, which is what keeps the 100 counter from filling up. Set it to a value that reflects how late a run may still be useful. Set it too low and jobs get skipped whenever the controller is briefly busy.
• concurrencyPolicy. The default is Allow, same overlap problem as plain cron. Forbid skips the new run, Replace kills the old one.
• At-least-once, not exactly-once. The Kubernetes docs say it directly: a CronJob may create two Jobs, or no Job, for a single schedule. Make the work idempotent, because the platform does not promise otherwise.

Cron questions from jobs that failed