***************************************** * What's New in Synchronet Version 3.20 * * (ChangeLog since v3.19b Jan 2, 2022) * ***************************************** General ~~~~~~~ o New userbase (see https://wiki.synchro.net/history:newuserbase for details) o New primary configuration files: http://wiki.synchro.net/history:newcfgfiles o Improved UTF-8 terminal support - UNICODE characters sent to Terminal Server are automatically converted to CP437 (if possible) in most scenarios - Auto-detection of zero-width character column width in UTF-8 terminals (some UTF-8 terminals display an empty cell, others do not) - Spying on UTF-8 terminals displays CP437 equivalents (much prettier) via sbbsctrl and umonitor applications o New statistics file formats - */dsts.dab (daily statistics and running totals) -> */dsts.ini - */csts.dab (cumulative statistics / log) -> */csts.tab - DSTSEDIT (daily stats editor) goes away: just edit the dsts.ini file o New event time file format - ctrl/time.dab and qnet.dab are now converted to ctrl/time.ini o Increase maximum length of configured file/directory paths and command-lines from 63 to 100 characters o Internal codes (prefixes and suffixes) have doubled in maximum length from 8 to 16 characters (or 32 characters for prefixed-codes) o New IRC service (ircd v2.0) o New command-line specifier: %- to represent the user's chat handle/call-sign o New configurable date display/input formats (e.g. Year first) and configurable date value separators (e.g. '.', '-', ' ' instead of '/') Security ~~~~~~~~ o New cryptographic library (cryptlib) version 3.4.7 for TLS and SSH - with many custom patches for bug fixes and Synchronet-specific features o Set the minimum TLS version for the FTP server and TLS services to TLS 1.2 o Disallow new user aliases from matching an existing user's real name o Implement duplicate new-user email address checking (optional) - Set SCFG->System->New User Prompts->Force Unique E-mail/NetMail to "Yes" o New 'O' restriction is automatically applied to accounts with a duplicate real name (when allowed by the sysop) and prevents such users from posting or sending mail with their real name o Change the semantics of the "Allow Sysop Logins" setting in SCFG->System: an account with sysop access (i.e. level 90+) can still login, but any action that normally requires the system password will not be allowed. This includes the sysop-actions available in the FTP server when authenticating with : as the password. The sysop-user can still authenticate (and login), but none of those sysop-actions will be available to them. Renamed/moved to SCFG->System->Security Options->Toggles->Allow Sysop Access o Filter files (e.g. text/*.can) now have optional expiration date and other metadata - Configured via the sbbs.ini: LoginAttemptFilterDuration (default: 0/infinite) For details, https://wiki.synchro.net/config:filter_files#trash_can_files Customization ~~~~~~~~~~~~~ o New preferred method of text.dat string sysop-customization: ctrl/text.ini See https://wiki.synchro.net/custom:text.ini for details o Baja: Make !INCLUDE directives filename case-insensitive - some old command shells/mods wouldn't compile on *nix systems otherwise o New stock JavaScript modules: - msgscancfg.js and filescancfg.js Replaces logic previously copy/pasted into multiple Baja command shells - user_settings.js Replaces hard-coded (in C++) user defaults/settings menu o New @-codes: - BUILD_DATE - BUILD_TIME - CPS - OS_CPU - TRUNCATE - TRUNCOFF - GETDIM - LINEDELAY[:n] - MSG_TO_FIRST - MSG_FROM_FIRST - PROT - PROTNAME - TERMTYPE - TERMROWS - TERMCOLS - AUTOTERM - ANSI - ASCII - COLOR - ICE - RIP - PETSCII - SWAPDEL - UTF8 - MOUSE - UPAUSE - SPIN - PAUSESPIN - EXPERT - HOTKEYS - MSGCLS - REMSUBS - FILEDESC - FILEFLAG - AUTOHANG - AUTOLOGON - QUIET - ASKNSCAN - ASKSSCAN - ANFSCAN - EDITOR - SHELL - Every string from the text.dat file, using the string ID o @-codes can now be included in text centered with the CENTER @-code Note: this centering logic does not know the expanded-size of the @-code, so use a fixed-length @-code (e.g. with padding) or use the 'C' @-code format modifier instead o @-codes with format modifiers are now supported in JS bbs.atcode() method, previously, only the basic '-L' and '-R' modifiers were partially supported o Support @-code expansion in certin text.dat/ini strings: NodeLoggedOnAtNbps NodeLoggedOff Regarding RegardingByOn RegardingByToOn o Fixed negative MSG_SCORE @-code value on 64-bit *nix builds o Updated text.dat strings: - BulkUploadDescPrompt - now takes a string (%s) for the file size in bytes - Regarding* -can now use @-codes instead of printf-specifiers - UseerDefaultsHotKey - is now blank ("") by default (suppressed) - YNQP - no longer used, replaced with first char of Yes/No/Quit/Password o Mnemonic strings now support @-codes immediately following the tilde (e.g. ~@Yes@) to use the first character of a dynamically-replaced (e.g. localized/translated) text string as a command key o New text.dat strings: - HashingFile - HashedFile - YouCantUseThatNetmail - NewUserValEmailSubj - InactivityAlert - FreeDownloadUserMsg - None - Which - Next - Prev - Quit - Language - LANG - UserDefaultsLanguage - PasswordChar - NodeConnectionSFTP - QWKSettingsWrapText - UnknownUploader - FREE o New text.dat strings for received Fido and Internet mail - InternetMailReceived - InternetMailForwarded - FidoNetMailReceived - WithAttachment - FidoEchoMailReceived For The Millionaire (RIP?), closing issue #254 The first string (suppressed with %.0s) is an optional date/time stamp. This also deprecates (removes support for) the [mail] NewMailNotice and ForwardNotice keys in sbbs.ini Windows ~~~~~~ o Synchronet runs on Windows XP again Fixed 'The procedure entry point inet_ntop could not be located in the dynamic link library WS2_32.dll' o Ignore VDD WriteFile() failures if the child process has terminated If the child process (e.g. door game) has terminated, don't log errors if/when WriteFile() to the mailslot fails. This would be expected as the mailslot is created/owned-by sbbsexec.dll which would also terminate along with the process, thus closing the mailslot o Suppress "VDD Open failed" warning if child process terminated o Many improvements to the Virtual UART Driver as a result of SVDM development: - Report CTS status in FOSSIL driver (high when outbuf has some space) - Emulate FIFO enablement - Support RTS flow control - Report "DCD change" in MSR correctly o Synchronet Control Panel (sbbsctrl.exe) now has a "Log Events to Disk" option, enabling writing of event thread log messages to data/events*.log o New useredit.exe (all new C++ version, but using the old Delphi forms) o scfg.exe and echocfg.exe are now Graphics/GDI applications (use -iw option if you want to use the old Windows "console" output mode) Configuration Utility (SCFG) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ o Displays path/filename of config file actively being edited, not date/time o Supports cycling through like-items using left/right arrow keys e.g. comparing/glancing through grps, subs, libs, and dirs, is really easy! o New "Servers" menu: Almost all settings in the sbbs.ini file are now configurable here o New "Initial Setup Wizard" runs automatically on new-install or via '-w' opt Similar to the SBBSCTRL:Configuration Wizard in function, but TUI, not GUI o Moved Node->Toggle Options->Allow Login by User Number, Login by Real Name, Always Prompt for Password, to (new) System->Security Options o Moved Node->Advanced Options->Validation user, Notification User, and Notification Error Level to (new) System->Notifications menu o Moved Node->Advanced Options->Inactivity Warning and Inactivity Disconnection to System->Advanced->User Inactivity Warning (now a percentage) and Maximum User Inactivity o Removed Node->Advanced Options->Semaphore Frequency and Statistics Frequency o Adding nodes prompts to automatically adjust the Terminal Server LastNode value to include the new (higher) node number o Moved security settings from "System" menu to (new) System->Security Options - Password - Users Can Change Password - Days to Preserve Deleted Users - Maximum Days of Inactivity - New User Password (now prompted when setting Security->Open to New Users) - Security Level Values, Expired Account Values, and Quick-Validation Values - Toggle Options->Allow Sysop Logins, Display/Log Passwords Locally, Users Expire when Out-of-time, Require Sys Pass during Login, and Closed to New Users o Added System->New User Values->QWK Packet Settings o Added New User Chat options to System->New User Values->Default Toggles: Multinode Chat Echo, Multinode Chat Actions, Pageable for Chat, Node Activity Messages, and Split-Screen Private Chat o Moved System->New User Values->Question Toggles to System->New User Prompts o Added System->Advanced->Maximum Log File Size (enable/control log rotation) o Added System->Loadable Modules->Send Feedback, Chat Section, List Users, Scan Dirs, List Files, View File Info, Batch Transfer - *All* configured loadable modules can now include command-line options o File Directory->Toggle->Template for New replaced with File Library-> Directory Defaults... "Clone Options" moved to Directory Defaults->Clone Settings o New File Library options: Upload Requirements, Download Requirements, Operator Requirements, Exemption Requirements, Auto-Add Sub-directories, Virtual Sub-directories o Add "Native" option for QWKnet call-out Command-lines o Add an 'enabled' property for QWKnet hubs, defaults to true o Move the QWKnet hub pack and unpack commands to an "Advanced" sub-menu since they are completely optional now (SBBS uses libarchive otherwise) o Trim leading and trailing white-space in input strings, automatically o Don't allow spaces when entering/editing internal codes o Configuration file back-ups are now configured (in SCFG->System->Advanced) rather than controlled via command-line option (-b) to SCFG o New command-line option (-A) to enable alternate/ASCII arrow symbols o SCFG Command-line options are now case-sensitive ('-a' and '-A' are unique) o SCFG Command-line options must be preceded with '-', not '/', on all OSes o '?' key can be used to pull-up online help (e.g. when F1 key is not working) JavaScript ~~~~~~~~~~ o Command-shells can be written entirely in JavaScript, no .bin stub needed o New property: file_area.web_vpath_prefix o file_area.min_diskspace is in bytes now, not kilobytes o file-metadata-object (return value of FileBase.get()) - New property: vpath - Renamed property: metadata -> auxdata o New system methods for vetting filenames (e.g. for upload by users) - illegal_filename() - check if contains illegal chars/sequences - safest_filename() - check if contains only safest chars - allowed_filename() - check if meets criteria from SCFG->File Options - check_filename() - check if legal and meets configured criteria and is not in file.can o bbs.check_filename() - ditto, except will display badfile.msg as appropriate o MsgBase.save_msg() throws an exception when empty recipient list is provided o Restore ability for MsgBase.open() to open an arbitrary SMB msgbase o Remember the last 'first_msg' property value after MsgBase is closed o Fixed user.dat open file descriptor leak in User object constructor o New User.stats.download_cps property o New User.mail_settings property o New User.close() method o Add 'fidonet_addr' property to msg_area.sub[] o New global function: rmfiles() - remove files and sub-dirs, recursively o New global function: strip_ctrl_a() - remove Ctrl-A sequences from string o New bbs.expand_atcodes() method o New bbs.batch_clear() method o New bbs.chat_sec() method o New bbs.sync() method o New bbs methods: save_msg_scan() and reload_msgs_scan() o bbs.telnet_gate() and rlogin_gate() now return boolean (success/failure) o bbs.logoff() now returns boolean (false if log-off was denied) o New bbs.load_user_text() method o bbs.text() now accepts an additional argument: default_text=false o bbs.text now has numeric properties named after each text ID (so load() or require() of text.js is no longer strictly required) o bbs.editfile() now supports overriding the maximum lines (default: 10000) and message metadata fields place in external editor drop files o New bbs properties: first_node and last_node o New system.find_login_id() method o Allow system.matchuserdata() to search deleted user records o New property: system.login_settings o New property: system.mail_settings o New property: system.guru o New console properties: - line_delay - max_getkey_inactivity - last_getkey_activity - max_socket_inactivity o New console properties that present common (possibly localized) command keys: yes_key, no_key, quit_key, all_key, list_key, next_key, prev_key o console.mouse_mode now supports being set to just true or false (no flags) o console.pause() now accepts an optional bool argument (set_abort) o New console.flush() method o New console.progress() method Display a progress indication bar, e.g. "[ Scanning 10.0% ]" with the bar backfill effect (when supported by the terminal) o New console methods: - cond_newline() - cond_blankline() - cond_contline() - linefeed() o Renamed console.crlf() to console.newline() (leaving crlf() as an alias) o File.readBin() and writeBin() methods now support 64-bit integer binary data o New method: File.iniRemoveSections() o Stock modules previously provided as Baja-compiled .bin files, now as .js: default.js (uses new load/shell_lib.js library for writing command shells) yesnobar.js noyesbar.js Servers ~~~~~~~ o MQTT (IoT protocol) statistics/status/output publishing and control See https://wiki.synchro.net/ref:mqtt for details o Allow a configured maximum-severity (minimum value) for TLS-related log messages, default is 0 (LOG_EMERG, maximum severity) o All servers support login-by-realname/user-number (if enabled by the sysop) o All servers have configurable login requirements (AR String) o All servers can be "paused" (and subsequently unpaused) via semaphore file or MQTT - paused servers don't accept new connections, but otherwise keep running o All servers (terminal, mail, web, ftp, services) recycle or shutdown faster and do not accept new connections when pending a recycle or shutdown o systemd integration (status reporting) on *nix builds o Fix uploader-notification, credit awards, download-counters in FTP downloads Services ~~~~~~~~ o Allow a per-service "LowestLogLevel" setting Have a service (e.g. imapservice.js) that logs errors that you'd rather not fill your error.log file with? Set that service's "LowestLogLevel" to "Warning" in your services.ini file ("lower" means "more severe" when it comes to log levels, so this would set the maximum severity to Warning) Terminal Server ~~~~~~~~~~~~~~~ o Multiple Language/Locale support (user selectable), see ctrl/text.*.ini and https://wiki.synchro.net/custom:localization for details o Socket inactivity watchdog now supported and configurable with separate maximum inactivity values for Login, New User Registration, and normal User Session (SCFG->Servers->Terminal Server->Max * Inactivity settings) - This inactivity detection works even when scripts don't call getkey() but is reset upon received TCP traffic of any kind (even Telnet NOP/GA) o Progress-update displays are now optimized for slow (e.g. modem) connections o Basic PETSCII output column/line counting support (for auto-pause) o All input is translated for terminal idiosyncrasies (e.g. DEL<->BackSpace and PETSCII) in the terminal server's input thread o Displayed Instant Messages (notifications and telegrams) are now stored and historic messages viewable via ;msgs command and 'V' from the ^P prompt o New display file naming (*.cXX.[asc|msg|ans|rip|seq]) where XX is the /minimum/ column width (not necessarily the exact column width) The old *.XXcol.* naming is still supported for exact-column-width files o ASC/MSG versions of display files are no longer required If all a sysop wants to create/use are .ans files, they can do that, non-ANSI users be damned o Use nearest-zone matching for source FTN address when replying to netmail o Fix lost 'unexpected characters' received in ANSI get cursor position response (issue #304) o newuser.js has a new 'notify_sysop' option (enabled via modopts.ini) Will notify the sysop via email and telegram whenever a new user account has been created o When editing existing messages (in message bases), the metadata details of the message (e.g. to, from, subject) are now passed to external editors in the editor drop file o Configurable system password timeout (default: 15 minutes) o Fix PETSCII 40/80 column port connections for IPv6 o [BBS] DefaultTermWidth=80 DefaultTermHeight=24 o Online user editor display more of the user's (long) password o Increase user's hostname field (aka user_t.comp) from 30 to 60 chars o Most sysops didn't know it, but if exec/feedback.* existed, it would be executed just before any user sent an email to the sysop (user #1), excluding new user validation requests: - make this module name configurable and loadable from mods - support JS module here (exit(1) to abort the feedback) - invoke for email being sent to *any* sysop (not just user #1) - don't invoke the module when sending *from* a sysop account o Make new user QWK-related and Chat-related settings configurable - Added chat-settings to SCFG->System->New User Values->Default Toggles - Added new menu: SCFG->System->New User Values->QWK Packet Settings o New line-based method of output throttling (e.g for animated ANSIs) Enabled/set via LINEDELAY @-code and JS console.line_delay property o New twitlist-author operator menu option when reading messages o SSH FTP (SFTP) support o SSH "any auth" support (e.g. for use with login matrixes) o SSH terminal type/size advertisement o SSH max-severity of log messages is now configurable o SSH negotiation is now asynchronous/non-blocking, so other nodes can accept connections while any SSH negotiations are in progress Doors ~~~~~ o New option to Alert/Disconnect an Inactive User while running program o Fix EXITINFO.BBS drop file generation (a few problems with size and garbage) o Total rewrite of the PCBOARD.SYS and USERS.SYS drop file generation logic o Limit DOOR.SYS drop file numeric values to 32767 - Fixes issues with doors using The DoorFrame door library (e.g Lemonade!) o Populate line 36 (user alias) of DOOR.SYS with the current user's handle Previously, we just always made this a blank line o Add line 8 (user's real name) to DOORFILE.SR (Solar Realms' drop file) o Dates in drop files are now always in MM/DD/YY format, never DD/MM/YY o Work-around BRE/FE/TAL ANSI-detection bug in second line of DOORFILE.SR o Line 37 ("Event Time") of DOOR.SYS was always 00:00, now it'll be the next event time (in HH:MM format) o Add "Disable Local Display" option for doors New option to disable local screen display for door programs: sets the 'Screen' value appropriately in door.sys or pcboard.sys drop files and on Windows, doesn't create a new console window. o Parse DOSXTRN.ERR Also, now parsing the DOSXTRN.ERR file created by the latest/greatest dosxtrn.exe when failing to execute the child/DOS program and log the parsed error details (errno value and description), helpful in debugging the reason why a DOS program may not have been successfully executed by DOSXTRN o When user hangs-up on external programs on *nix, try to terminate w/SIGTERM Previously, when a user disconnected or ran out of time while running a stdio-based external program on *nix, if the program was still running, we'd send it a SIGHUP, wait up to 10 seconds for the process to terminate and if it did not, terminate it (ungracefully) with SIGKILL. Since some programs catch SIGTERM (and not SIGHUP) to indicate a termination request, we now will first attempt a SIGHUP, wait up to 5 seconds for the process to terminate and if it does not, then send a SIGTERM and wait up to another 5 seconds for it to terminate and if it doesn't, then finally send it a SIGKILL (which cannot be caught and always results in an ungraceful termination of the child process) File Transfers ~~~~~~~~~~~~~~ o Added support for SSH FTP (SFTP) o Extracted DIZ from sub-directory of archive does not preempt root DIZ File o Increased maximum extended file description length from 4000 to 5000 chars o Filenames > 12 chars use liberal filename matching (fixes issue seen at https://youtu.be/_IWzIV0_sZ4?t=310) o Allow maximum uploaded filename length to be configured (default: 64) o Searching for file descriptions in the Terminal Server, now searching file's tags, author, and author group, if available o Fixed issue (#328) removing files from batch queues o Fix "Testable Files" file extension comparison (issue #331) Issue introduced in v3.19: Testable Files (a.k.a. upload processors) with a specified file extension/type (e.g. "ZIP" and not "*") would never run because the file extension comparison logic was "off by one". Testable Files with an extension of "*" (all files/types) would still run however. o Implement the sbbsfile.nam and sbbsfile.des post-processing removed in v3.19 o Auto-detect/display UTF-8 encoded extended file descriptions o Improved support for files > 4GB in size o Add options to sort directories by file size (ascending or descending) o Allow sysop-choice of source of virtual sub-directory source name: By default, each file transfer directory's internal code suffix is used as the source of the sub-directory of the virtual path used to represent a file area in the FTP and Web servers. Now, a sysop can change that source to either each directory's short name or long name, if they prefer. o Removed FTP server DIR_FILES option (no longer applicable) o Fix: Use the user's default download protocol for batch downloads o Increase maximum file extension length from 3 to 15 characters (e.g. tar.gz) o Fix bug (issue #515) with moving files between directories o Minimum disk space more universally enforced and no longer limited to 65535K o Better access enforcement to files in batch download queues o Remove defunct files from batch download queue during logon o Store/reuse file download throughput (in CPS), for transfer estimates/ETA o User-to-user file transfer recipients were required to have read/view access to the "User" directory - this was in error since v3.19, users don't normally have access to list the files in the user-to-user ("User") directory, for good reason (these files are supposed to be private) Message Bases/Areas ~~~~~~~~~~~~~~~~~~~ o Sanity check sub-board's assigned FTN addresses when loading configuration (use nearest matching configured AKA) o Fixed QWKnet netmail dest user look-up (using qnet/users.dat) broken back in 2021 o Fix heap corruption in decoding of quoted-printable text in smblib o Fix erroneous filtering of vote messages by age e.g. !Filtering QWK message from (null) due to age: 19321 days o Fix: Include blank lines in messages edited with internal editor o Don't save draft messages upon disconnect for Guest or "no one" (user #0) o Fixes for moderated message bases (e.g. ignore deleted messages) o Ignore trailing white-space from message subjects before hashing Use new chksmb -S option to ignore resulting Subject CRC mismatch errors o Improved MIME/QP-encoded message text parsing (including MIME non-quoted "boundary" values) o Saving changes to an existing message header can increase the required header block allocation (i.e. if new header fields are added or extended) by moving the header if necessary in the .shd file, thus eliminating the old "illegal header length increase" error that could occur when modifying an existing message header SBBSecho ~~~~~~~~ o Don't use libarchive for creating/appending bundles - Must use (have configured) external archive program (e.g. zip) o New EchoCfg->Global->Sort Linked Node List option o Supports filtering messages based on text/subject.can file o Rework NetMail routing logic to handle point destinations better o Default packet type is now configurable o Handle LF-terminated messages more "correctly" (convert to CR or CRLF) o Options for handling of incoming bad packets - delete - rename *.bad (the previous behavior) - rename *..bad (the new default behavior) HatchIT ~~~~~~~ o New non-interactive mode (patch by @acn) o Generates MS-DOS compatible 'File' key values and 'Lfile' key when source filename is not MS-DOS compatible Web Server ~~~~~~~~~~ o Direct Filebase access for downloading files with access controls, notifications, updated statistics and credits (optional, enabled via SCFG->Servers->Web Server->Filebase VPath Prefix (e.g. "/files/")) o Dynamic file area/base indexing, for a demo, set sbbs.ini [web] FileIndexScript = webfileindex.ssjs FileVPathPrefix = /files/ o Requested-path aliasing, via ctrl/web_alias.ini o Log requests that resolve outside of the web root as hack attempts o Fix HTTP-requests for files >= 2GB in size o Perform a JS garbage collection for each new request in a reused session o Add UNIX domain FastCGI support (in *nix builds only) o Fix use of IPv6 addresses in access log filenames on Windows o Fewer temp files (e.g. 0-byte) being created and potentially left behind Mail Server ~~~~~~~~~~~ o Fix corrupted RFC822 msg headers when a header field was > 1024 chars o Reject SMTP session from any client that sends illegally-long lines o SMTP mail server wasn't RFC 4954 compliant for "AUTH PLAIN" logins Fix for errant "Missing AUTH PLAIN argument" log message o Fix false FORGED mail header 'FROM' field detection/rejection o Fix max_clients limit not being effectively-imposed on SMTPS connections FTP Server ~~~~~~~~~~ o Limit uploaded file sizes, accounting for free disk space o Fix uploader-notification, credit awards, download-counters in downloads o Fix false "SUSPECTED BOUND ATTACK ATTEMPT" for IPv6 active-data connections o Source of virtual sub-directory names are now configurable in SCFG - May be derived from internal code suffix (default), the short or long name