Release Notes - Kafka - Version 3.8.0

Below is a summary of the JIRA issues addressed in the 3.8.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

  • [KAFKA-10892] - Add Topology#connectSourceStoreAndTopic as a public method
  • [KAFKA-15045] - Move Streams task assignor to public configs
  • [KAFKA-15444] - KIP-974: Docker Image for GraalVM based Native Kafka Broker
  • [KAFKA-15942] - Implement ConsumerInterceptor
  • [KAFKA-16373] - Docker Official Image for Apache Kafka

    • Improvement

  • [KAFKA-7632] - Support Compression Level
  • [KAFKA-7663] - Custom Processor supplied on addGlobalStore is not used when restoring state from topic
  • [KAFKA-9401] - High lock contention for kafka.server.FetchManager.newContext
  • [KAFKA-10199] - Separate state restoration into separate threads
  • [KAFKA-12399] - Deprecate Log4J Appender KIP-719
  • [KAFKA-14041] - Avoid the keyword var for a variable declaration in ConfigTransformer
  • [KAFKA-14246] - Update threading model for Consumer
  • [KAFKA-14747] - FK join should record discarded subscription responses
  • [KAFKA-14822] - Allow restricting File and Directory ConfigProviders to specific paths
  • [KAFKA-15349] - ducker-ak should fail fast when gradlew systemTestLibs fails
  • [KAFKA-15420] - Kafka Tiered Storage V1
  • [KAFKA-15517] - Improve MirrorMaker logging in case of authorization errors
  • [KAFKA-15541] - Improved StateStore Iterator metrics for detecting leaks
  • [KAFKA-15625] - Do not flush global state store at each commit
  • [KAFKA-15878] - KIP-768: Extend support for opaque (i.e. non-JWT) tokens in SASL/OAUTHBEARER
  • [KAFKA-15905] - Restarts of MirrorCheckpointTask should not permanently interrupt offset translation
  • [KAFKA-15948] - Refactor AsyncKafkaConsumer shutdown
  • [KAFKA-15949] - Improve the KRaft metadata version related messages
  • [KAFKA-15951] - MissingSourceTopicException should include topic names
  • [KAFKA-15974] - Enforce that event processing respects user-provided timeout
  • [KAFKA-15996] - json serializer too slow
  • [KAFKA-16021] - Eagerly load Charset for StringSerializer/StringDeserializer
  • [KAFKA-16093] - Spurious warnings logged to stderr about empty path annotations and providers not implementing provider interfaces
  • [KAFKA-16100] - Add timeout to all the CompletableApplicationEvents
  • [KAFKA-16104] - Enable additional PlaintextConsumerTest tests for new consumer
  • [KAFKA-16113] - AsyncKafkaConsumer: Add missing offset commit metrics
  • [KAFKA-16115] - AsyncKafkaConsumer: Add missing heartbeat metrics
  • [KAFKA-16116] - AsyncKafkaConsumer: Add missing rebalance metrics
  • [KAFKA-16161] - Avoid creating remote log metadata snapshot file in partition data directory.
  • [KAFKA-16166] - Generify RetryWithToleranceOperator and ErrorReporter
  • [KAFKA-16200] - Enforce that RequestManager implementations respect user-provided timeout
  • [KAFKA-16202] - Extra dot in error message in producer
  • [KAFKA-16204] - Stray file core/00000000000000000001.snapshot created when running core tests
  • [KAFKA-16210] - Upgrade jose4j to 0.9.4
  • [KAFKA-16238] - ConnectRestApiTest broken after KIP-1004
  • [KAFKA-16239] - Clean up references to non-existent IntegrationTestHelper
  • [KAFKA-16246] - Cleanups in ConsoleConsumer
  • [KAFKA-16280] - Expose method to determine Metric Measurability
  • [KAFKA-16285] - Make group metadata available when a new assignment is set in async Kafka consumer
  • [KAFKA-16325] - Add missing producer metrics to documentation
  • [KAFKA-16343] - Improve tests of streams foreignkeyjoin package
  • [KAFKA-16391] - Cleanup .lock file after server is down
  • [KAFKA-16393] - SslTransportLayer doesn't implement write(ByteBuffer[], int, int) correctly
  • [KAFKA-16397] - Use ByteBufferOutputStream to avoid array copy
  • [KAFKA-16399] - Add JBOD support in tiered storage
  • [KAFKA-16415] - Fix handling of "--version" option in ConsumerGroupCommand
  • [KAFKA-16416] - Use NetworkClientTest to replace RequestResponseTest to be the example of log4j output
  • [KAFKA-16445] - PATCH method for connector configuration
  • [KAFKA-16455] - Check partition exists before send reassignments to server in ReassignPartitionsCommand
  • [KAFKA-16463] - Automatically delete metadata log directory on ZK brokers
  • [KAFKA-16467] - Add README to docs folder
  • [KAFKA-16477] - Detect thread leaked client-metrics-reaper in tests
  • [KAFKA-16490] - Upgrade gradle from 8.6 to 8.7
  • [KAFKA-16507] - Add raw record into RecordDeserialisationException
  • [KAFKA-16513] - Allow WriteTxnMarkers API with Alter Cluster Permission
  • [KAFKA-16544] - DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames should return null instead of throwing NPE
  • [KAFKA-16548] - Avoid decompressing/collecting all records when all we want to do is to find a single matched record from remote storage
  • [KAFKA-16549] - suppress the warnings from RemoteLogManager
  • [KAFKA-16552] - Create an internal config to control InitialTaskDelayMs in LogManager to speed up tests
  • [KAFKA-16560] - Refactor/cleanup BrokerNode/ControllerNode/ClusterConfig
  • [KAFKA-16572] - allow defining number of disks per broker in ClusterTest
  • [KAFKA-16573] - Streams does not specify where a Serde is needed
  • [KAFKA-16591] - RaftClusterInvocationContext ought to clear the admin queue when stopping
  • [KAFKA-16593] - Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions
  • [KAFKA-16598] - Mirgrate `ResetConsumerGroupOffsetTest` to new test infra
  • [KAFKA-16610] - Replace "Map#entrySet#forEach" by "Map#forEach"
  • [KAFKA-16614] - Disallow `@ClusterTemplate("")`
  • [KAFKA-16627] - Remove ClusterConfig parameter in BeforeEach and AfterEach
  • [KAFKA-16640] - Replace TestUtils#resource by scala.util.Using
  • [KAFKA-16647] - Remove setMetadataDirectory from BrokerNode/ControllerNode
  • [KAFKA-16654] - Refactor kafka.test.annotation.Type and ClusterTestExtensions
  • [KAFKA-16660] - reduce the check interval to speedup DelegationTokenRequestsTest
  • [KAFKA-16668] - Enable to set tags by `ClusterTest`
  • [KAFKA-16676] - Security docs missing RPCs from KIP-714 and KIP-1000
  • [KAFKA-16677] - Replace ClusterType#ALL and ClusterType#DEFAULT by Array
  • [KAFKA-16685] - RLMTask warning logs do not include parent exception trace
  • [KAFKA-16763] - Upgrade to scala 2.12.19 and scala 2.13.14
  • [KAFKA-16787] - Remove TRACE level logging from AsyncKafkaConsumer hot path
  • [KAFKA-16797] - A bit cleanup of FeatureControlManager
  • [KAFKA-16844] - ByteArrayConverter can't convert ByteBuffer
  • [KAFKA-16911] - Kafka Streams topology optimization docs incomplete
  • [KAFKA-16935] - Automatically wait for cluster startup in embedded Connect integration tests
  • [KAFKA-17165] - Revisit LeaderEpochFileCache#writeToFileForTruncation

    • Bug

  • [KAFKA-5863] - Potential null dereference in DistributedHerder#reconfigureConnector()
  • [KAFKA-6527] - Transient failure in DynamicBrokerReconfigurationTest.testDefaultTopicConfig
  • [KAFKA-7957] - Flaky Test DynamicBrokerReconfigurationTest#testMetricsReporterUpdate
  • [KAFKA-8206] - A consumer can't discover new group coordinator when the cluster was partly restarted
  • [KAFKA-9228] - Reconfigured converters and clients may not be propagated to connector tasks
  • [KAFKA-12679] - Rebalancing a restoring or running task may cause directory livelocking with newly created task
  • [KAFKA-13295] - Long restoration times for new tasks can lead to transaction timeouts
  • [KAFKA-13328] - Connect does not perform preflight validation for per-connector header converters
  • [KAFKA-13329] - Connect does not perform preflight validation for per-connector key and value converters
  • [KAFKA-13907] - Fix hanging ServerShutdownTest.testCleanShutdownWithKRaftControllerUnavailable
  • [KAFKA-13922] - Unable to generate coverage reports for the whole project
  • [KAFKA-13950] - Resource leak at multiple places in the code
  • [KAFKA-14089] - Flaky ExactlyOnceSourceIntegrationTest.testSeparateOffsetsTopic
  • [KAFKA-14404] - Fix & update docs on client configs controlled by Streams
  • [KAFKA-14957] - Default value for state.dir is confusing
  • [KAFKA-15018] - Potential tombstone offsets corruption for exactly-once source connectors
  • [KAFKA-15170] - CooperativeStickyAssignor cannot adjust assignment correctly
  • [KAFKA-15250] - ConsumerNetworkThread is running tight loop
  • [KAFKA-15305] - The background thread should try to process the remaining task until the shutdown timer is expired
  • [KAFKA-15417] - JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()
  • [KAFKA-15475] - Request might retry forever even if the user API timeout expires
  • [KAFKA-15610] - Fix `CoreUtils.swallow()` test gaps
  • [KAFKA-15760] - org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated is flaky
  • [KAFKA-15770] - org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy is flaky
  • [KAFKA-15804] - Broker leaks ServerSocketChannel when exception is thrown from ZkConfigManager during startup
  • [KAFKA-15816] - Typos in tests leak network sockets
  • [KAFKA-15823] - NodeToControllerChannelManager: authentication error prevents controller update
  • [KAFKA-15827] - KafkaBasedLog.withExistingClients leaks clients if start is not called
  • [KAFKA-15834] - Subscribing to non-existent topic blocks StreamThread from stopping
  • [KAFKA-15848] - Design solution for inconsistency between ConsumerDelegate timeout policies
  • [KAFKA-15915] - Flaky test - testUnrecoverableError - ProducerIdManagerTest
  • [KAFKA-15950] - Serialize broker heartbeat requests
  • [KAFKA-15964] - Flaky test: testHighAvailabilityTaskAssignorLargeNumConsumers – org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest
  • [KAFKA-16003] - The znode /config/topics is not updated during KRaft migration in "dual-write" mode
  • [KAFKA-16009] - Fix PlaintextConsumerTest.testMaxPollIntervalMsDelayInRevocation
  • [KAFKA-16015] - kafka-leader-election timeout values always overwritten by default values
  • [KAFKA-16025] - Streams StateDirectory has orphaned locks after rebalancing, blocking future rebalancing
  • [KAFKA-16047] - Source connector with EOS enabled have some InitProducerId requests timing out, effectively failing all the tasks & the whole connector
  • [KAFKA-16051] - Deadlock on connector initialization
  • [KAFKA-16052] - OOM in Kafka test suite
  • [KAFKA-16055] - Thread unsafe use of HashMap stored in QueryableStoreProvider#storeProviders
  • [KAFKA-16073] - Kafka Tiered Storage: Consumer Fetch Error Due to Delayed localLogStartOffset Update During Segment Deletion
  • [KAFKA-16083] - Exclude throttle time when expiring inflight requests on a connection
  • [KAFKA-16087] - Tasks dropping incorrect records when errors.tolerance=all and errors reported asynchronously due to data race
  • [KAFKA-16089] - Kafka Streams still leaking memory
  • [KAFKA-16105] - Reassignment of tiered topics is failing due to RemoteStorageException
  • [KAFKA-16152] - Fix PlaintextConsumerTest.testStaticConsumerDetectsNewPartitionCreatedAfterRestart
  • [KAFKA-16155] - Investigate testAutoCommitIntercept
  • [KAFKA-16167] - Fix PlaintextConsumerTest.testAutoCommitOnCloseAfterWakeup
  • [KAFKA-16169] - FencedException in commitAsync not propagated without callback
  • [KAFKA-16178] - AsyncKafkaConsumer doesn't retry joining the group after rediscovering group coordinator
  • [KAFKA-16191] - Clean up of consumer client internal events
  • [KAFKA-16196] - Cast transform doesn't handle invalid whole value casts gracefully
  • [KAFKA-16197] - Connect Worker poll timeout prints Consumer poll timeout specific warnings.
  • [KAFKA-16198] - Reconciliation may lose partitions when topic metadata is delayed
  • [KAFKA-16203] - AutoCommit of empty offsets blocks following requests due to inflight flag
  • [KAFKA-16206] - KRaftMigrationZkWriter tries to delete deleted topic configs twice
  • [KAFKA-16209] - fetchSnapshot might return null if topic is created before v2.8
  • [KAFKA-16217] - Transactional producer stuck in IllegalStateException during close
  • [KAFKA-16222] - KRaft Migration: desanitize entity name when migrate client quotas
  • [KAFKA-16226] - Java client: Performance regression in Trogdor benchmark with high partition counts
  • [KAFKA-16243] - Idle kafka-console-consumer with new consumer group protocol preemptively leaves group
  • [KAFKA-16252] - Maligned Metrics formatting
  • [KAFKA-16261] - MembershipManagerImpl.updateSubscription fails if already empty subscription
  • [KAFKA-16277] - CooperativeStickyAssignor does not spread topics evenly among consumer group
  • [KAFKA-16278] - Missing license for scala related dependencies
  • [KAFKA-16286] - KRaft doesn't always notify listener of latest leader
  • [KAFKA-16288] - Values.convertToDecimal throws ClassCastExceptions on String inputs
  • [KAFKA-16289] - Values.parseString on heterogeneous lists and maps sometimes corrupts data by inferring incorrect schema
  • [KAFKA-16305] - Optimisation in SslTransportLayer:handshakeUnwrap stalls TLS handshake
  • [KAFKA-16310] - ListOffsets doesn't report the offset with maxTimestamp anymore
  • [KAFKA-16312] - ConsumerRebalanceListener.onPartitionsAssigned() should be called after joining, even if empty
  • [KAFKA-16318] - Add javadoc to KafkaMetric
  • [KAFKA-16319] - Wrong broker destinations for DeleteRecords requests when more than one topic is involved and the topics/partitions are led by different brokers
  • [KAFKA-16322] - Fix CVE-2023-50572 by updating jline from 3.22.0 to 3.25.1
  • [KAFKA-16323] - Failing test: fix testRemoteFetchExpiresPerSecMetric
  • [KAFKA-16347] - Bump ZooKeeper to 3.8.4
  • [KAFKA-16349] - ShutdownableThread fails build by calling Exit with race condition
  • [KAFKA-16352] - Transaction may get get stuck in PrepareCommit or PrepareAbort state
  • [KAFKA-16358] - Update Connect Transformation documentation
  • [KAFKA-16359] - kafka-clients-3.7.0.jar published to Maven Central is defective
  • [KAFKA-16369] - Broker may not shut down when SocketServer fails to bind as Address already in use
  • [KAFKA-16371] - Unstable committed offsets after triggering commits where metadata for some partitions are over the limit
  • [KAFKA-16381] - use volatile to guarantee KafkaMetric#config visibility across threads
  • [KAFKA-16385] - Segment is rolled before segment.ms or segment.bytes breached
  • [KAFKA-16386] - NETWORK_EXCEPTIONs from transaction verification are not translated
  • [KAFKA-16389] - consumer_test.py’s test_valid_assignment fails with new consumer
  • [KAFKA-16392] - Spurious log warnings: "Ignoring offset partition key with an unexpected format for the second element in the partition key list. Expected type: java.util.Map, actual type: null"
  • [KAFKA-16405] - Mismatch assignment error when running consumer rolling upgrade system tests
  • [KAFKA-16408] - kafka-get-offsets / GetOffsetShell doesn't handle --version or --help
  • [KAFKA-16410] - kafka-leader-election / LeaderElectionCommand doesn't set exit code on error
  • [KAFKA-16411] - Correctly migrate default client quota entities in KRaft migration
  • [KAFKA-16424] - truncated logs will be left undeleted after alter dir completion
  • [KAFKA-16427] - KafkaConsumer#position() does not respect timeout when group protocol is CONSUMER
  • [KAFKA-16428] - Fix bug where config change notification znode may not get created during migration
  • [KAFKA-16433] - beginningOffsets and offsetsForTimes don't behave consistently when providing a zero timeout
  • [KAFKA-16435] - Add test for KAFKA-16428
  • [KAFKA-16447] - Fix failed ReplicaManagerTest
  • [KAFKA-16459] - New consumer times out joining group in consumer_test.py system test
  • [KAFKA-16461] - New consumer fails to consume records in security_test.py system test
  • [KAFKA-16465] - New consumer does not invoke rebalance callbacks as expected in consumer_test.py system test
  • [KAFKA-16466] - QuorumController is swallowing some exception messages
  • [KAFKA-16471] - SslTransportLayer may leak SSLEngine resources
  • [KAFKA-16473] - KafkaDockerWrapper uses wrong cluster ID when formatting log dir
  • [KAFKA-16474] - AsyncKafkaConsumer might send out heartbeat request without waiting for its response
  • [KAFKA-16511] - Leaking tiered segments
  • [KAFKA-16539] - Can't update specific broker configs in pre-migration mode
  • [KAFKA-16541] - Potential leader epoch checkpoint file corruption on OS crash
  • [KAFKA-16555] - Consumer's RequestState has incorrect logic to determine if inflight
  • [KAFKA-16556] - SubscriptionState should not prematurely reset 'pending' partitions
  • [KAFKA-16563] - migration to KRaft hanging after MigrationClientException
  • [KAFKA-16565] - IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned
  • [KAFKA-16566] - Update consumer static membership fencing system test to support new protocol
  • [KAFKA-16570] - FenceProducers API returns "unexpected error" when successful
  • [KAFKA-16577] - New consumer fails with stop within allotted timeout in consumer_test.py system test
  • [KAFKA-16583] - Update from 3.4.0 to 3.7.0 image write failed in Kraft mode
  • [KAFKA-16588] - broker shutdown hangs when `log.segment.delete.delay.ms` is zero
  • [KAFKA-16592] - ConfigKey constructor update can break clients using it
  • [KAFKA-16606] - JBOD support in KRaft does not seem to be gated by the metadata version
  • [KAFKA-16608] - AsyncKafkaConsumer doesn't honor interrupted thread status on KafkaConsumer.poll(Duration)
  • [KAFKA-16609] - Update parse_describe_topic to support new topic describe output
  • [KAFKA-16622] - Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once
  • [KAFKA-16637] - AsyncKafkaConsumer removes offset fetch responses from cache too aggressively
  • [KAFKA-16639] - AsyncKafkaConsumer#close does not send heartbeat to leave group
  • [KAFKA-16659] - KafkaConsumer#position() does not respect wakup when group protocol is CONSUMER
  • [KAFKA-16692] - InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not enabled when upgrading from kafka 3.5 to 3.6
  • [KAFKA-16705] - the flag "started" of RaftClusterInstance is false even though the cluster is started
  • [KAFKA-16709] - alter logDir within broker might cause log cleanup hanging
  • [KAFKA-16739] - Exclude protected variables from aggregated JavaDocs
  • [KAFKA-16757] - Fix broker re-registration issues around MV 3.7-IV2
  • [KAFKA-16759] - Invalid client telemetry transition on consumer close
  • [KAFKA-16764] - New consumer should throw InvalidTopicException on poll when invalid topic in metadata
  • [KAFKA-16766] - New consumer offsetsForTimes timeout exception has wrong message
  • [KAFKA-16777] - New consumer should throw NoOffsetForPartitionException on continuous poll zero if no reset strategy
  • [KAFKA-16786] - New consumer should not require the deprecated partition.assignment.strategy
  • [KAFKA-16790] - Calls to RemoteLogManager are made before it is configured
  • [KAFKA-16794] - Can't open videos in streams documentation
  • [KAFKA-16807] - DescribeLogDirsResponseData#results#topics have unexpected topics having empty partitions
  • [KAFKA-16814] - KRaft broker cannot startup when `partition.metadata` is missing
  • [KAFKA-16824] - Utils.getHost and Utils.getPort do not catch a lot of invalid host and ports
  • [KAFKA-16833] - Cluster missing topicIds from equals and hashCode, PartitionInfo missing equals and hashCode
  • [KAFKA-16837] - Kafka Connect fails on update connector for incorrect previous Config Provider tasks
  • [KAFKA-16838] - Kafka Connect loads old tasks from removed connectors
  • [KAFKA-16858] - Flatten SMT throws NPE
  • [KAFKA-16861] - Don't convert to group to classic if the size is larger than group max size
  • [KAFKA-16886] - KRaft partition reassignment failed after upgrade to 3.7.0
  • [KAFKA-16888] - Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory
  • [KAFKA-16890] - Failing to build aux state on broker failover
  • [KAFKA-16903] - Task should consider producer error previously occurred for different task
  • [KAFKA-16933] - New consumer leave group flow may not send leave request and clear epoch
  • [KAFKA-16946] - Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port
  • [KAFKA-16948] - Reset tier lag metrics on becoming follower
  • [KAFKA-16954] - Move consumer leave operations on close to background thread
  • [KAFKA-16969] - KRaft unable to upgrade to v3.7.1 and later when multiple log dir is set
  • [KAFKA-16977] - remote log manager dynamic configs are not available after broker restart.
  • [KAFKA-16988] - InsufficientResourcesError in ConnectDistributedTest system test
  • [KAFKA-17083] - KRaft Upgrade Failures in SystemTests
  • [KAFKA-17085] - Streams Cooperative Rebalance Upgrade Test fails in System Tests
  • [KAFKA-17098] - Error Opening RocksDBStore
  • [KAFKA-17111] - ServiceConfigurationError in JsonSerializer/Deserializer during Plugin Discovery
  • [KAFKA-17148] - Kafka storage tool prints MetaPropertiesEnsemble
  • [KAFKA-17150] - Connect converter validation does not accept class aliases

    • Task

  • [KAFKA-10047] - Unnecessary widening of (int to long) scope in FloatSerializer
  • [KAFKA-13115] - Document that doSend can be blocking
  • [KAFKA-15558] - Determine if Timer should be used elsewhere in PrototypeAsyncConsumer.updateFetchPositions()
  • [KAFKA-15575] - Prevent Connectors from exceeding tasks.max configuration
  • [KAFKA-15576] - Add 3.6.0 to broker/client and streams upgrade/compatibility tests
  • [KAFKA-15711] - KRaft support in LogRecoveryTest
  • [KAFKA-15717] - KRaft support in LeaderEpochIntegrationTest
  • [KAFKA-15719] - KRaft support in OffsetsForLeaderEpochRequestTest
  • [KAFKA-15721] - KRaft support in DeleteTopicsRequestWithDeletionDisabledTest
  • [KAFKA-15723] - KRaft support in ListOffsetsRequestTest
  • [KAFKA-15726] - KRaft support in ProduceRequestTest
  • [KAFKA-15728] - KRaft support in DescribeUserScramCredentialsRequestNotAuthorizedTest
  • [KAFKA-15729] - KRaft support in GetOffsetShellTest
  • [KAFKA-15735] - KRaft support in SaslMultiMechanismConsumerTest
  • [KAFKA-15738] - KRaft support in ConsumerWithLegacyMessageFormatIntegrationTest
  • [KAFKA-15740] - KRaft support in DeleteOffsetsConsumerGroupCommandIntegrationTest
  • [KAFKA-15741] - KRaft support in DescribeConsumerGroupTest
  • [KAFKA-15742] - KRaft support in GroupCoordinatorIntegrationTest
  • [KAFKA-15743] - KRaft support in ReplicationQuotasTest
  • [KAFKA-15747] - KRaft support in DynamicConnectionQuotaTest
  • [KAFKA-15750] - KRaft support in KafkaMetricReporterExceptionHandlingTest
  • [KAFKA-15776] - Update delay timeout for DelayedRemoteFetch request
  • [KAFKA-16004] - Review new consumer inflight offset commit logic
  • [KAFKA-16112] - Review JMX metrics in Async Consumer and determine the missing ones
  • [KAFKA-16137] - ListClientMetricsResourcesResponse definition is missing field descriptions
  • [KAFKA-16146] - Checkpoint log-start-offset for remote log enabled topics
  • [KAFKA-16356] - Remove class-name dispatch in RemoteLogMetadataSerde
  • [KAFKA-16362] - Fix type-unsafety in KStreamKStreamJoin caused by isLeftSide
  • [KAFKA-16406] - Split long-running consumer integration test
  • [KAFKA-16409] - kafka-delete-records / DeleteRecordsCommand should use standard exception handling
  • [KAFKA-16452] - Bound highwatermark offset to range b/w local-log-start-offset and log-end-offset
  • [KAFKA-16457] - Useless import class
  • [KAFKA-16493] - Avoid unneeded subscription regex check if metadata version unchanged
  • [KAFKA-16528] - Reset member heartbeat interval when request sent
  • [KAFKA-16579] - Revert changes to consumer_rolling_upgrade_test.py for the new async Consumer
  • [KAFKA-16605] - Fix the flaky LogCleanerParameterizedIntegrationTest
  • [KAFKA-16645] - CVEs in 3.7.0 docker image
  • [KAFKA-16665] - Fail to get partition's position from within onPartitionsAssigned callback in new consumer
  • [KAFKA-16675] - Move rebalance callback test for positions to callbacks test file
  • [KAFKA-16695] - Improve expired poll interval logging by showing exceeded time
  • [KAFKA-16696] - Remove the in-memory implementation of RSM and RLMM
  • [KAFKA-16771] - First log directory printed twice when formatting storage
  • [KAFKA-16815] - Handle FencedInstanceId on heartbeat for new consumer
  • [KAFKA-16825] - CVE vulnerabilities in Jetty and netty
  • [KAFKA-16826] - Integrate Native Kafka Docker Image with github Actions
  • [KAFKA-16827] - Integrate Native Apache Kafka with System tests
  • [KAFKA-16865] - Admin.describeTopics behavior change after KIP-966
  • [KAFKA-16881] - InitialState type leaks into the Connect REST API OpenAPI spec
  • [KAFKA-16932] - Documentation changes for Native Docker image
  • [KAFKA-17050] - Revert group.version for 3.8 and 3.9

    • Test

  • [KAFKA-15140] - Improve TopicCommandIntegrationTest to be less flaky
  • [KAFKA-15206] - Flaky test RemoteIndexCacheTest.testClose()
  • [KAFKA-15691] - Update system tests to use AsyncKafkaConsumer
  • [KAFKA-15692] - New integration tests to ensure full coverage
  • [KAFKA-15897] - Flaky Test: testWrongIncarnationId() – kafka.server.ControllerRegistrationManagerTest
  • [KAFKA-16011] - Fix PlaintextConsumerTest.testMultiConsumerSessionTimeoutOnClose
  • [KAFKA-16019] - Some of the tests in PlaintextConsumer can't seem to deterministically invoke and verify the consumer callback
  • [KAFKA-16023] - PlaintextConsumerTest needs to wait for reconciliation to complete before proceeding
  • [KAFKA-16037] - Update VerifiableConsumer to support KIP-848’s group protocol config
  • [KAFKA-16048] - Test failure kafka.admin.ConfigCommandTest.shouldNotSupportAlterClientMetricsWithZookeeper
  • [KAFKA-16064] - Improve ControllerApisTest
  • [KAFKA-16084] - Simplify and deduplicate StandaloneHerderTest mocking
  • [KAFKA-16134] - kafka.api.PlaintextConsumerTest.testPerPartitionLagMetricsCleanUpWithSubscribe(String, String).quorum=kraft+kip848.groupProtocol=consumer is flaky
  • [KAFKA-16135] - kafka.api.PlaintextConsumerTest.testPerPartitionLeadMetricsCleanUpWithSubscribe(String, String).quorum=kraft+kip848.groupProtocol=consumer is flaky
  • [KAFKA-16230] - Update verifiable_consumer.py to support KIP-848’s group protocol config
  • [KAFKA-16231] - Update consumer_test.py to support KIP-848’s group protocol config
  • [KAFKA-16267] - Update consumer_group_command_test.py to support KIP-848’s group protocol config
  • [KAFKA-16268] - Update fetch_from_follower_test.py to support KIP-848’s group protocol config
  • [KAFKA-16269] - Update reassign_partitions_test.py to support KIP-848’s group protocol config
  • [KAFKA-16270] - Update snapshot_test.py to support KIP-848’s group protocol config
  • [KAFKA-16271] - Update consumer_rolling_upgrade_test.py to support KIP-848’s group protocol config
  • [KAFKA-16273] - Update consume_bench_test.py to support KIP-848’s group protocol config
  • [KAFKA-16274] - Update replica_scale_test.py to support KIP-848’s group protocol config
  • [KAFKA-16275] - Update kraft_upgrade_test.py to support KIP-848’s group protocol config
  • [KAFKA-16276] - Update transactions_test.py to support KIP-848’s group protocol config
  • [KAFKA-16287] - Implement example test for common rebalance callback scenarios
  • [KAFKA-16293] - Test log directory failure in Kraft
  • [KAFKA-16388] - add production-ready test of 3.3 - 3.6 release to MetadataVersionTest.testFromVersionString
  • [KAFKA-16413] - Add tests for FileLock
  • [KAFKA-16438] - Update consumer_test.py’s static tests to support KIP-848’s group protocol config
  • [KAFKA-16439] - Update replication_replica_failure_test.py to support KIP-848’s group protocol config
  • [KAFKA-16440] - Update security_test.py to support KIP-848’s group protocol config
  • [KAFKA-16472] - Integration tests in Java don't really run kraft case
  • [KAFKA-16482] - Eliminate the IDE warnings of accepting ClusterConfig in BeforeEach
  • [KAFKA-16483] - Apply `ClusterTestExtensions` to DeleteOffsetsConsumerGroupCommandIntegrationTest
  • [KAFKA-16484] - Support to define per broker/controller property by ClusterConfigProperty
  • [KAFKA-16487] - Support to define server properties by ClusterTestDefaults
  • [KAFKA-16559] - allow defining number of disks per broker in TestKitNodes
  • [KAFKA-16596] - Flaky test – org.apache.kafka.clients.ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()
  • [KAFKA-16613] - Remove TestUtils#subscribeAndWaitForRecords
  • [KAFKA-16671] - Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured
  • [KAFKA-16678] - Remove unimplementedquorum from EndToEndAuthorizationTest
  • [KAFKA-16679] - Merge unit test down to the class of integration test
  • [KAFKA-16686] - Flakey tests in TopicBasedRemoteLogMetadataManagerTest
  • [KAFKA-16702] - KafkaLog4jAppenderTest leaks producer instances
  • [KAFKA-16703] - SocketServer leaks ServerSocketChannel when port is already in-use
  • [KAFKA-16712] - Race while setting RemoteLogMetadataTopicPartitioner in TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest
  • [KAFKA-16774] - fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
  • [KAFKA-16783] - Migrate RemoteLogMetadataManagerTest to new test infra
  • [KAFKA-16784] - Migrate TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra
  • [KAFKA-16789] - Thread leak detection checks for incorrect QuorumController thread name
  • [KAFKA-16828] - RackAwareTaskAssignorTest failed
  • [KAFKA-16866] - RemoteLogManagerTest.testCopyQuotaManagerConfig failing
  • [KAFKA-16961] - TestKRaftUpgrade system tests fail in v3.7.1 RC1
  • [KAFKA-16990] - Unrecognised flag passed to kafka-storage.sh in system test

    • Sub-task

  • [KAFKA-12187] - replace assertTrue(obj instanceof X) by assertInstanceOf when we update to JUnit 5.8
  • [KAFKA-14505] - Implement TnxOffsetCommit API
  • [KAFKA-14509] - Add ConsumerGroupDescribe API
  • [KAFKA-14576] - Move ConsoleConsumer to tools
  • [KAFKA-14589] - Move ConsumerGroupCommand to tools
  • [KAFKA-14683] - Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest
  • [KAFKA-14701] - More broker side partition assignor to common
  • [KAFKA-15460] - Add group type filter to ListGroups API
  • [KAFKA-15462] - Add group type filter to the admin client
  • [KAFKA-15538] - Client support for java regex based subscription
  • [KAFKA-15807] - Add support for compression/decompression of metrics
  • [KAFKA-15811] - implement capturing client port information from Socket Server
  • [KAFKA-15813] - Improve implementation of client instance cache
  • [KAFKA-15832] - Trigger client reconciliation based on manager poll
  • [KAFKA-15882] - Scheduled nightly github actions workflow for CVE reports on published docker images
  • [KAFKA-15989] - Upgrade existing generic group to consumer group
  • [KAFKA-16016] - Migrate utility scripts to kafka codebase
  • [KAFKA-16032] - Review client errors thrown on OffsetFetch and OffsetCommit failures
  • [KAFKA-16033] - Review client retry logic of OffsetFetch and OffsetCommit responses
  • [KAFKA-16034] - AsyncKafkaConsumer will get Invalid Request error when trying to rejoin on fenced/unknown member Id
  • [KAFKA-16053] - Fix leaked Default DirectoryService
  • [KAFKA-16058] - Fix leaked in ControllerApiTest
  • [KAFKA-16059] - Fix leak of ExpirationReaper-1-AlterAcls threads in :core:test
  • [KAFKA-16062] - Upgrade mockito to 5.8.0
  • [KAFKA-16063] - Fix leaked ApplicationShutdownHooks in EndToEndAuthorizationTests
  • [KAFKA-16065] - Fix leak in DelayedOperationTest
  • [KAFKA-16082] - Broker recreates reassigned partition after logdir failure
  • [KAFKA-16095] - Update list group state type filter to include the states for the new consumer group type
  • [KAFKA-16103] - AsyncKafkaConsumer: Always await async commit callbacks in commitSync and close
  • [KAFKA-16107] - Ensure consumer does not start fetching from added partitions until onPartitionsAssigned completes
  • [KAFKA-16117] - Add Integration test for checking if the correct assignor is chosen
  • [KAFKA-16147] - Partition is assigned to two members at the same time
  • [KAFKA-16148] - Implement GroupMetadataManager#onUnloaded
  • [KAFKA-16156] - System test failing for new consumer on endOffsets with negative timestamps
  • [KAFKA-16165] - Consumer invalid transition on expired poll interval
  • [KAFKA-16168] - Implement GroupCoordinator.onPartitionsDeleted
  • [KAFKA-16185] - Fix client reconciliation of same assignment received in different epochs
  • [KAFKA-16186] - Implement broker metrics for client telemetry usage
  • [KAFKA-16189] - Extend admin to support ConsumerGroupDescribe API
  • [KAFKA-16190] - Member should send full heartbeat when rejoining
  • [KAFKA-16194] - KafkaConsumer.groupMetadata() should be correct when first records are returned
  • [KAFKA-16207] - KRaft internal listener for KRaftVersionRecord and VotersRecord
  • [KAFKA-16215] - Consumer does not rejoin after fenced on delayed revocation
  • [KAFKA-16224] - Fix handling of deleted topic when auto-committing before revocation
  • [KAFKA-16227] - Console consumer fails with `IllegalStateException`
  • [KAFKA-16234] - Log directory failure re-creates partitions in another logdir automatically
  • [KAFKA-16249] - Improve reconciliation state machine
  • [KAFKA-16251] - Fenced member should not send heartbeats while waiting for onPartitionsLost to complete
  • [KAFKA-16258] - Stale member should trigger onPartitionsLost when leaving group
  • [KAFKA-16265] - Add durationFilter to ListTransactionsRequest
  • [KAFKA-16294] - Add group protocol migration enabling config
  • [KAFKA-16297] - Race condition while promoting future replica can lead to partition unavailability.
  • [KAFKA-16298] - Ensure user callbacks exceptions are propagated to the user on consumer poll
  • [KAFKA-16306] - GroupCoordinatorService logger is not configured
  • [KAFKA-16307] - fix EventAccumulator thread idle ratio metric
  • [KAFKA-16313] - Offline group protocol migration
  • [KAFKA-16314] - Add the new ABORTABLE_ERROR
  • [KAFKA-16341] - Fix un-compressed records
  • [KAFKA-16342] - Fix compressed records
  • [KAFKA-16363] - Storage tool crashes if dir is unavailable
  • [KAFKA-16365] - AssignmentsManager mismanages completion notifications
  • [KAFKA-16367] - Full ConsumerGroupHeartbeat response must be sent when full request is received
  • [KAFKA-16374] - High watermark updates should have a higher priority
  • [KAFKA-16375] - Fix logic for discarding reconciliation if member rejoined
  • [KAFKA-16436] - Online upgrade triggering and group type conversion
  • [KAFKA-16470] - kafka-dump-log --offsets-decoder should support new records
  • [KAFKA-16485] - Fix broker metrics to follow kebab/hyphen case
  • [KAFKA-16486] - Integrate metric measurability changes in metrics collector
  • [KAFKA-16515] - Fix the ZK Metadata cache use of voter static configuration
  • [KAFKA-16516] - Fix the controller node provider for broker to control channel
  • [KAFKA-16525] - Request manager and raft channel discover node endpoints
  • [KAFKA-16526] - Change to quorum state for KIP-853
  • [KAFKA-16530] - Fix high-watermark calculation to not assume the leader is in the voter set
  • [KAFKA-16554] - Online downgrade triggering and group type conversion
  • [KAFKA-16568] - Add JMH Benchmarks for assignor performance testing
  • [KAFKA-16587] - Store subscription model for consumer group in group state
  • [KAFKA-16615] - JoinGroup API for upgrading ConsumerGroup
  • [KAFKA-16625] - Reverse Lookup Partition to Member in Assignors
  • [KAFKA-16626] - Uuid to String for subscribed topic names in assignment spec
  • [KAFKA-16646] - Consider only running the CVE scanner action on apache/kafka and not in forks
  • [KAFKA-16663] - CoordinatorRuntime write timer tasks should be cancelled once HWM advances
  • [KAFKA-16664] - Re-add EventAccumulator.take(timeout)
  • [KAFKA-16694] - Remove rack aware code in assignors temporarily due to performance
  • [KAFKA-16722] - Add ConsumerGroupPartitionAssignor interface
  • [KAFKA-16735] - Deprecate offsets.commit.required.acks in 3.8
  • [KAFKA-16762] - SyncGroup API for upgrading ConsumerGroup
  • [KAFKA-16770] - Coalesce records into bigger batches
  • [KAFKA-16793] - Heartbeat API for upgrading ConsumerGroup
  • [KAFKA-16795] - Fix broken compatibility in kafka.tools.NoOpMessageFormatter, kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter
  • [KAFKA-16796] - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder
  • [KAFKA-16802] - Move build.gradle java version information inside of a java block
  • [KAFKA-16804] - Replace gradle archivesBaseName with archivesName
  • [KAFKA-16805] - Stop using a ClosureBackedAction to configure Spotbugs reports
  • [KAFKA-16821] - Create a new interface to store member metadata
  • [KAFKA-16831] - CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit
  • [KAFKA-16832] - LeaveGroup API for upgrading ConsumerGroup
  • [KAFKA-16860] - Introduce `group.version` feature flag
  • [KAFKA-16864] - Copy on write in the Optimized Uniform Assignor
  • [KAFKA-16930] - UniformHeterogeneousAssignmentBuilder throws NPE when member has not subscriptions
  • [KAFKA-17142] - Fix deadlock caused by LogManagerTest#testLogRecoveryMetrics
  • [KAFKA-17166] - Use `NoOpScheduler` to rewrite LogManagerTest#testLogRecoveryMetrics