Microsoft releases official fix for Y2K22 Exchange Server bug that was causing mail jam

System administrators looking over Microsoft Exchange servers probably haven't had the most joyous start to their new years. That's because there has been a massive date processing failure issue as the new value chosen "2.201.010.001" exceeds what the Servers are capable of processing under the current Int32 data type. As a result, the malware checking engine is crashing, and consequently, emails and messages have been stuck in transport queues across Exchange Servers 2016 and 2019 with Application event log errors 5300 and 1106 (FIPFS).

Microsoft said that it was aware of the problem and was working on a fix.

We are aware of and working on an issue causing messages to be stuck in transport queues on Exchange Server 2016 and Exchange Server 2019.

[...] We are actively working on resolving this issue and expect to release details on how to resolve this issue later today.

The company has kept its promise and has released a resolution for the problem. It has provided both an automated scan engine reset script solution as well as a manual input solution.

First, we have the automated scan engine reset script (available at https://aka.ms/ResetScanEngineVersion). It can be run parallelly across multiple servers. The successful script completion should give the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath https://amupdatedl.microsoft.com/server/amupdate
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation
Engine                : Microsoft
LastChecked       : 01/01/2022 08:58:22 PM -08:00
LastUpdated        : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001
UpdateStatus          : UpdateAttemptSuccessful

Next up, steps for the manual fix are provided for those who prefer to do a manual restore.

Remove existing engine and metadata

1. Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine

1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 .

Verify engine update info

1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

You may find more details on the official Microsoft blog post linked here.