Drop support for Spark 3.4

[andygrove]

What is the problem the feature request solves?

This issue is to discuss the timing for dropping support for Spark 3.4

See also the proposed versioning policy.

Describe the potential solution

No response

Additional context

No response

[manuzhang]

Can we bump up minimum java version to 17 as well?

[andygrove]

I suggest that we make Comet 1.0.0 (the next release) the last one to support Spark 3.4.

[andygrove]

Can we bump up minimum java version to 17 as well?

I created separate tracking issue #4844