feature: add install-time logtime timestamp precision choice (#325)

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 --- develop ---
 
+* feature: Add an install-time timestamp precision choice for the logtime column (issue #325). New MariaDB installs default to millisecond TIMESTAMP(3), which preserves sub-second ordering and still RANGE partition-prunes; MySQL installs are fixed to second precision because MySQL cannot prune a fractional partitioned column, and an explicit millisecond request on MySQL is rejected. Upgrades never migrate the schema automatically; switch an existing MariaDB install with the destructive cli/updatedb_convert_ms.php utility during a maintenance window. README documents forwarder configuration (rsyslog, Cisco, Juniper, Arista, syslog-ng) for delivering sub-second timestamps
 * feature: Add a manifest.json that drives upgrade-time "housecleaning": when genuinely upgrading from an older version, syslog_prune_files() (run from the shared syslog_check_upgrade() path so every version-aware entry point - the plugin upgrade hook, the re-enable path and cli/syslog_schema.php - cleans up) removes files and directories that older layouts shipped but that have since moved or been deleted (for example the old top-level database.php/functions.php now living under includes/), so a stale module can no longer redeclare a function and fatal the plugin on load after an upgrade. User-data paths (config.php, config_local.php) and VCS metadata (including .git/.md descendants nested under a tombstoned directory) are never touched, tombstones reached through a symlinked parent or that resolve outside the plugin are refused, a fresh install or same-version repair leaves the dev-only tests/ tree intact, and a CI check (tests/bin/validate-manifest.php) keeps the manifest's expected-file list from drifting
 * bug: Stop partition recovery from wedging with MySQL error 1493 ('VALUES LESS THAN value must be strictly increasing') when an existing partition boundary already covers the day being created. Both syslog_partition_check() and syslog_partition_create() decided a daily partition was missing purely by name, so a table carried over from an older TO_DAYS schema, or left inconsistent by an interrupted 4.2-to-4.11 upgrade, would repeatedly emit a REORGANIZE of dMaxValue that MySQL rejected, leaving writes parked in the dMaxValue safety partition. Both now compare the target boundary against the highest existing concrete boundary, so already-covered days no longer count as missing, no longer consume the bounded per-run recovery budget, and are treated as a successful no-op, while genuine gaps are still created (issue #434)
 * bug: Stop preloading every distinct program into the query-builder choice list; on a busy collector the unbounded SELECT over syslog_programs built a very large choice array that could exhaust the PHP memory limit and fatal (HTTP 503) when opening the Alert, Removal and Dashboard editors, while the Report editor (which does not use the filter builder) kept working. The program and program_id fields already resolve through the bounded AJAX suggestion endpoint, so only the small facility and priority enumerations are preloaded now

README.md

@@ -115,6 +115,82 @@ supported.  Other engines, including MyISAM, are no longer offered and fall
 back to InnoDB when legacy settings request them.  Enabling any Remote Data
 Collector option requires InnoDB; the setup advisor selects it automatically.
 
+## Millisecond Timestamps
+
+The `logtime` column can be stored either at second precision (a `TIMESTAMP`
+column) or at millisecond precision (a `TIMESTAMP(3)` column).  Millisecond
+precision preserves the sub-second ordering of high-rate messages.
+
+**Millisecond precision is only supported on MariaDB.**  MariaDB can still
+`RANGE` partition a `TIMESTAMP(3)` column by `UNIX_TIMESTAMP(logtime)` and prune
+partitions; MySQL rejects that partition function on a fractional column, so
+MySQL installs always use second precision.
+
+### Choosing the precision
+
+* **New installs** – the setup advisor shows a *Timestamp Precision* option.
+  On MariaDB it defaults to *Millisecond - TIMESTAMP(3)*; on MySQL it is fixed
+  to second precision.  Silent installs can set
+  `$syslog_install_options['logtime_precision'] = 'ms';` (or `'sec'`) in
+  `config.php` / `config_local.php`; a millisecond request on MySQL is rejected
+  with a log message rather than silently downgraded.
+* **Upgrades** – the existing schema is authoritative and is **never** migrated
+  automatically, even if the configured precision changes.  This avoids the
+  long, blocking table rewrite that an in-place precision change would require.
+* **Existing production systems** – switch precision only with the CLI utility
+  below, during a maintenance window, after accepting that stored messages are
+  destroyed.
+
+### Converting an existing MariaDB install
+
+```console
+php plugins/syslog/cli/updatedb_convert_ms.php
+```
+
+The utility is **MariaDB only** and **destructive**: it truncates the
+`syslog`, `syslog_incoming`, and `syslog_removed` tables (deleting all stored
+messages) and changes their `logtime` column to `TIMESTAMP(3)`. The replication
+outbox (`syslog_replication_output`) is widened in place so any undelivered rows
+are preserved, and the sequence high-water marks of the truncated tables are kept
+so remote replication identities are never reused. Alert, removal, and report
+rules and all settings are preserved. The conversion is all-or-nothing and
+re-runnable: the stored precision is only recorded once every target column is
+`TIMESTAMP(3)`. It prompts for confirmation; pass `--confirm` to skip the prompt
+in automation, and `--help` for usage.  Take a database backup first.
+
+### Delivering sub-second timestamps from your log forwarders
+
+Storing a `TIMESTAMP(3)` column only helps when the forwarder actually writes a
+sub-second value into `logtime`.  The default rsyslog template in
+[Cacti Configuration for RSYSLOG](#cacti-configuration-for-rsyslog) uses
+`date-mysql`, which has no fractional part.  To capture milliseconds, write a
+high-precision timestamp instead.  For an rsyslog `ommysql` template, use the
+RFC3339 timestamp trimmed to a MySQL compatible `YYYY-MM-DD HH:MM:SS.mmm`:
+
+```console
+$template cacti_syslog_ms,"INSERT INTO syslog_incoming(facility_id, priority_id, program, logtime, host, message) \
+  values (%syslogfacility%, %syslogpriority%, '%programname%', \
+  REPLACE(SUBSTRING('%timereported:::date-rfc3339%',1,23),'T',' '), '%HOSTNAME%', TRIM('%msg%'))", SQL
+```
+
+Use `timereported` (the sender's own timestamp) when you want the device's
+sub-second time; use `timegenerated` for the rsyslog receipt time.  The
+sender-reported time is only as precise as the device emits, so enable
+millisecond timestamps on the originating devices:
+
+| Platform | Directive |
+| --- | --- |
+| Cisco IOS / IOS-XE | `service timestamps log datetime msec` (add `localtime show-timezone year` as needed) |
+| Cisco NX-OS | `logging timestamp milliseconds` |
+| Juniper Junos | `set system syslog time-format millisecond` (optionally `... time-format year millisecond`) |
+| Arista EOS | `logging format timestamp high-resolution` |
+| syslog-ng (forwarding) | `ts-format(iso)` with `frac-digits(3)` on the destination |
+| rsyslog relay (RFC5424) | forward with the `RSYSLOG_SyslogProtocol23Format` template, which carries fractional seconds |
+
+For devices that cannot emit sub-second timestamps, `timegenerated` on the Cacti
+collector still records millisecond receipt times, which preserves ordering for
+messages arriving in the same second.
+
 ## Important Version 4.0 Release Notes
 
 In prior releases of Syslog, the Individual Alert Method would send an Email,
@@ -469,6 +545,11 @@ master process for its workers.
 `cli/syslog_schema.php` takes `--audit`, `--repair`, or `--upgrade`, as
 described under Installation.
 
+`cli/updatedb_convert_ms.php` converts an existing MariaDB install to
+millisecond (`TIMESTAMP(3)`) timestamps.  It is destructive (it truncates the
+message tables) and takes `--confirm` and `--help`; see
+[Millisecond Timestamps](#millisecond-timestamps).
+
 ## Development and Testing
 
 The commands below are the ones the CI workflow runs, from the plugin directory

cli/updatedb_convert_ms.php

@@ -0,0 +1,225 @@
+<?php
+/*
+ +-------------------------------------------------------------------------+
+ | Copyright (C) 2004-2026 The Cacti Group                                 |
+ |                                                                         |
+ | This program is free software; you can redistribute it and/or           |
+ | modify it under the terms of the GNU General Public License             |
+ | as published by the Free Software Foundation; either version 2          |
+ | of the License, or (at your option) any later version.                  |
+ |                                                                         |
+ | This program is distributed in the hope that it will be useful,         |
+ | but WITHOUT ANY WARRANTY; without even the implied warranty of          |
+ | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the           |
+ | GNU General Public License for more details.                            |
+ +-------------------------------------------------------------------------+
+ | Cacti: The Complete RRDTool-based Graphing Solution                     |
+ +-------------------------------------------------------------------------+
+ | This code is designed, written, and maintained by the Cacti Group. See  |
+ | about.php and/or the AUTHORS file for specific developer information.   |
+ +-------------------------------------------------------------------------+
+ | http://www.cacti.net/                                                   |
+ +-------------------------------------------------------------------------+
+*/
+
+include(__DIR__ . '/../../../include/cli_check.php');
+require_once(dirname(__DIR__) . '/setup.php');
+require_once(dirname(__DIR__) . '/includes/schema.php');
+
+// process calling arguments
+$parms = $_SERVER['argv'];
+array_shift($parms);
+
+$confirm = false;
+
+foreach ($parms as $arg) {
+	switch ($arg) {
+		case '--confirm':
+			$confirm = true;
+
+			break;
+		case '--help':
+		case '-H':
+		case '-h':
+			updatedb_convert_ms_display_help();
+			exit(0);
+		default:
+			fwrite(STDERR, "ERROR: Unknown argument: $arg" . PHP_EOL);
+			updatedb_convert_ms_display_help();
+			exit(1);
+	}
+}
+
+syslog_determine_config();
+
+if (!defined('SYSLOG_CONFIG')) {
+	fwrite(STDERR, 'ERROR: Syslog database configuration is missing.' . PHP_EOL);
+	exit(1);
+}
+
+if (!syslog_connect()) {
+	fwrite(STDERR, 'ERROR: Unable to connect to the Syslog database.' . PHP_EOL);
+	exit(1);
+}
+
+exit(updatedb_convert_ms_run($confirm) ? 0 : 1);
+
+/**
+ * Convert the Syslog message pipeline tables to millisecond (TIMESTAMP(3)) logtime.
+ *
+ * This is a destructive, MariaDB only operation.  The message tables are
+ * truncated before their logtime column is widened, because the historical
+ * second-granularity rows cannot be migrated in place and the switch is
+ * intended for a maintenance window.  The replication outbox is widened in
+ * place instead, so undelivered rows are preserved, and its new rows then match
+ * the millisecond incoming timestamps on archival.  MySQL is rejected outright
+ * because it cannot RANGE partition-prune a fractional logtime column.
+ *
+ * The conversion is all-or-nothing: every TRUNCATE/ALTER is checked and the
+ * stored precision setting is only recorded once every target column is
+ * TIMESTAMP(3), so a partial failure can simply be re-run.
+ *
+ * @param bool $confirm Whether --confirm bypassed the prompt.
+ *
+ * @return bool Whether the conversion completed successfully.
+ */
+function updatedb_convert_ms_run(bool $confirm): bool {
+	global $syslogdb_default;
+
+	if (!syslog_db_is_mariadb()) {
+		fwrite(STDERR, 'ERROR: Millisecond (TIMESTAMP(3)) timestamps require MariaDB.  This server is MySQL, which cannot partition-prune a fractional logtime column.  No changes were made.' . PHP_EOL);
+
+		return false;
+	}
+
+	// Tables whose historical rows are discarded before the column is widened.
+	$truncate_tables = ['syslog', 'syslog_incoming', 'syslog_removed'];
+
+	// The outbox is widened in place so undelivered replication rows survive.
+	$alter_only_tables = ['syslog_replication_output'];
+
+	$all_tables = array_merge($truncate_tables, $alter_only_tables);
+
+	// Report which targets still need conversion.  Checking every target (not
+	// just syslog) lets a partial previous run be retried.
+	$pending = [];
+
+	foreach ($all_tables as $table) {
+		if (syslog_db_table_exists($table, false) && syslog_detect_logtime_precision($table) !== 'ms') {
+			$pending[] = $table;
+		}
+	}
+
+	if (!$pending) {
+		print 'The Syslog tables already use millisecond (TIMESTAMP(3)) timestamps.  Nothing to do.' . PHP_EOL;
+
+		return true;
+	}
+
+	print '================================ WARNING ================================' . PHP_EOL;
+	print 'This will convert the Syslog message pipeline to millisecond timestamps.' . PHP_EOL;
+	print 'It is DESTRUCTIVE: these tables will be TRUNCATED (all stored messages' . PHP_EOL;
+	print 'deleted) and their schema changed to TIMESTAMP(3):' . PHP_EOL;
+	print '    ' . implode(', ', $truncate_tables) . PHP_EOL;
+	print 'The replication outbox (' . implode(', ', $alter_only_tables) . ') is widened in' . PHP_EOL;
+	print 'place, preserving any undelivered rows.  Alert, removal and report rules' . PHP_EOL;
+	print 'are NOT affected.  Take a backup first.' . PHP_EOL;
+	print '========================================================================' . PHP_EOL;
+
+	if (!$confirm) {
+		print 'Type "yes" to proceed, anything else to abort: ';
+
+		$answer = trim((string) fgets(STDIN));
+
+		if (strtolower($answer) !== 'yes') {
+			print 'Aborted; no changes were made.' . PHP_EOL;
+
+			return false;
+		}
+	}
+
+	$column = syslog_logtime_sql_type('ms');
+
+	// Preserve the AUTO_INCREMENT high-water marks before truncating so new
+	// rows never reuse sequence values that remote replication identities
+	// (source_event_id, receipts) still reference.
+	$auto_increment = [];
+
+	foreach ($truncate_tables as $table) {
+		if (!syslog_db_table_exists($table, false)) {
+			continue;
+		}
+
+		$next = syslog_db_fetch_cell_prepared(
+			'SELECT AUTO_INCREMENT FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?',
+			[$syslogdb_default, $table],
+			'',
+			false
+		);
+
+		$auto_increment[$table] = (int) $next;
+	}
+
+	$ok = true;
+
+	foreach ($truncate_tables as $table) {
+		if (!syslog_db_table_exists($table, false)) {
+			continue;
+		}
+
+		$ok = syslog_db_execute("TRUNCATE TABLE `$syslogdb_default`.`$table`") && $ok;
+	}
+
+	foreach ($all_tables as $table) {
+		if (!syslog_db_table_exists($table, false)) {
+			continue;
+		}
+
+		$ok = syslog_db_execute("ALTER TABLE `$syslogdb_default`.`$table` MODIFY COLUMN logtime $column") && $ok;
+	}
+
+	foreach ($auto_increment as $table => $next) {
+		if ($next > 1) {
+			$ok = syslog_db_execute("ALTER TABLE `$syslogdb_default`.`$table` AUTO_INCREMENT = $next") && $ok;
+		}
+	}
+
+	// Record success only when every target column is actually TIMESTAMP(3).
+	foreach ($all_tables as $table) {
+		if (syslog_db_table_exists($table, false) && syslog_detect_logtime_precision($table) !== 'ms') {
+			$ok = false;
+		}
+	}
+
+	if (!$ok) {
+		fwrite(STDERR, 'ERROR: Conversion did not fully complete; the logtime precision setting was NOT changed.  Re-run this utility after resolving the database error (check the Cacti log).' . PHP_EOL);
+
+		return false;
+	}
+
+	set_config_option('syslog_install_logtime_precision', 'ms');
+
+	cacti_log('SYSLOG: Converted message pipeline tables to millisecond (TIMESTAMP(3)) logtime via cli/updatedb_convert_ms.php', false, 'SYSLOG');
+
+	print 'Conversion complete; Syslog now stores millisecond timestamps.' . PHP_EOL;
+	print 'Ensure your log forwarders emit sub-second timestamps (see README.md).' . PHP_EOL;
+
+	return true;
+}
+
+/**
+ * Display the millisecond conversion utility usage and options.
+ *
+ * @return void
+ */
+function updatedb_convert_ms_display_help(): void {
+	print 'Usage: php updatedb_convert_ms.php [--confirm]' . PHP_EOL;
+	print '  --confirm  Skip the interactive confirmation prompt.' . PHP_EOL;
+	print '  --help     Display this help.' . PHP_EOL;
+	print PHP_EOL;
+	print 'Converts the Syslog message pipeline to millisecond TIMESTAMP(3) logtime.' . PHP_EOL;
+	print 'MariaDB only.  TRUNCATES syslog, syslog_incoming and syslog_removed' . PHP_EOL;
+	print '(deleting stored messages) and widens syslog_replication_output in place;' . PHP_EOL;
+	print 'rules and settings are preserved.' . PHP_EOL;
+}
+

config.php.dist

@@ -76,6 +76,14 @@ $syslog_replication_main_db_ssl_ca   = $database_ssl_ca;
 //$syslog_install_options['mode']         = 'install';
 //$syslog_install_options['id']           = 'syslog';
 
+/* Timestamp precision for the logtime column on a NEW install:
+ *   'ms'  - millisecond TIMESTAMP(3), preserves sub-second ordering (MariaDB only, default)
+ *   'sec' - second granularity TIMESTAMP
+ * Millisecond precision is only supported on MariaDB; a 'ms' request on MySQL is
+ * rejected.  On a production system this cannot be changed in place later without
+ * destroying the stored messages (see cli/updatedb_convert_ms.php). */
+//$syslog_install_options['logtime_precision'] = 'ms';
+
 /* field in the incoming table */
 $syslog_incoming_config['priorityField']      = 'priority_id';
 $syslog_incoming_config['facilityField']      = 'facility_id';

config_local.php.dist

@@ -88,6 +88,14 @@ $syslog_install_options['days']         = '30';
 $syslog_install_options['mode']         = 'install';
 $syslog_install_options['id']           = 'syslog';
 
+/* Timestamp precision for the logtime column on a NEW install:
+ *   'ms'  - millisecond TIMESTAMP(3), preserves sub-second ordering (MariaDB only, default)
+ *   'sec' - second granularity TIMESTAMP
+ * Millisecond precision is only supported on MariaDB; a 'ms' request on MySQL is
+ * rejected.  On a production system this cannot be changed in place later without
+ * destroying the stored messages (see cli/updatedb_convert_ms.php). */
+//$syslog_install_options['logtime_precision'] = 'ms';
+
 /**
  * Optional Section:
  *