Expression Compatibility

Expressions that are not 100% Spark-compatible will fall back to Spark by default and can be enabled by setting spark.comet.expression.EXPRNAME.allowIncompatible=true, where EXPRNAME is the Spark expression class name. See the Comet Supported Expressions Guide for more information on this configuration setting.

Compatibility notes are grouped by expression category:

• Aggregate Expressions
• Array Expressions
• Date/Time Expressions
  • Date and Time Functions
• Map Expressions
  • MapSort (Spark 4.0+)
• Math Expressions
• Miscellaneous Expressions
• String Expressions
• Struct Expressions
• Cast
  • ANSI Mode Fallback
  • String to Decimal
  • String to Date
  • Date to Timestamp
  • Date to TimestampNTZ
  • Date to Numeric Types
  • String to Timestamp
  • String to TimestampNTZ
  • TimestampNTZ Casts
  • Date to String
  • String to TimestampNTZ
  • Decimal with Negative Scale to String
  • Legacy Mode
  • Try Mode
  • ANSI Mode