I was getting a production crash having just pushed an update that rolled out an upgrade from .NET 7 to .NET 8. The project uses EF core with an Azure MS SQL server. The crash never showed up during testing on an testing environment which was near identical to the production. What was going wrong and how did we miss it?
The exception that was coming from EF was:
Exception thrown: 'Microsoft.Data.SqlClient.SqlException' in System.Private.CoreLib.dll
Incorrect syntax near '$'.
Not that helpful but at least is communicating that the SQL might be the cause. I was able to hunt down the generated SQL that was the cause:
...
WHERE [b].[IsDelete] = CAST(0 AS bit) AND [t0].[IsConfirmed] = CAST(1 AS bit) AND [s].[Id] = @__session_Id_0 AND [t1].[Id] IN (
SELECT [p0].[value]
FROM OPENJSON(@__pupilIds_1) WITH ([value] int '$') AS [p0]
)
OPENJSON looks new here and it turns out that EF8 has a breaking change around the use of OPENJSON referenced here: https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-8.0/breaking-changes
But why did this not fail during testing? Why only once it has gone to production, and not only that but it failed on only one database and worked perfectly on others on the same server and environment.
What I was unaware of was that one of the databases I was using was running an older compatibility_level then the other databases. This resulted in the error only coming to light when using this specific database.
You can run the following SQL to compare compatibility levels:
SELECT compatibility_level, name
FROM sys.databases
When I did this, most of the databases were set to 150 however the problem database was set to 110. Its easy enough to change the compatibility but its worth reading the docs for breaking changes. If you need to change a database’s compatibility level you can then use:
ALTER DATABASE "database-name-goes-here" SET COMPATIBILITY_LEVEL = 150
For more information about the compatibility levels see here: https://learn.microsoft.com/en-us/sql/t-sql/statements/alter-database-transact-sql-compatibility-level?view=sql-server-ver16.
My understanding is that Azure SQL is always running the newest engine but the compatibility level of each database stays the at the same value it was initially set to. Its therefore your job to keep the compatibility level in sync. Another mistake to learn from is to review the docs when upgrading .NET versions or EF Core versions for any breaking changes.