1. 09 5月, 2021 1 次提交
  2. 08 5月, 2021 1 次提交
  3. 07 5月, 2021 6 次提交
  4. 06 5月, 2021 10 次提交
  5. 05 5月, 2021 3 次提交
  6. 04 5月, 2021 2 次提交
    • S
      KAFKA-10847: Add internal flag to disable KAFKA-10847 fix (#10612) · 62221eda
      Sergio Peña 提交于
      Adds an internal flag that can be used to disable the fixes in KAFKA-10847. It defaults to true if the flag is not set or has an invalid boolean value.
      The flag is named __enable.kstreams.outer.join.spurious.results.fix__. This flag is considered internal only. It is a temporary flag that will be used to help users to disable the join fixes while they do a transition from the previous semantics of left/outer joins. The flag may be removed in future releases.
      Reviewers: Guozhang Wang <wangguoz@gmail.com>
    • V
      KAFKA-5876: KIP-216 Part 3, Apply StreamsNotStartedException for Interactive Queries (#10597) · 816f5c3b
      Vito Jeng 提交于
      KIP-216 Part 3: Throw StreamsNotStartedException if KafkaStreams state is CREATED
      Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
  7. 03 5月, 2021 3 次提交
  8. 02 5月, 2021 2 次提交
    • C
      KAFKA-12661 ConfigEntry#equal does not compare other fields when value is NOT null (#10446) · e73731d8
      Chia-Ping Tsai 提交于
      Reviewers: Ismael Juma <ismael@juma.me.uk>
    • J
      KAFKA-12154; Raft Snapshot Loading API (#10085) · 6203bf8b
      José Armando García Sancio 提交于
      Implement Raft Snapshot loading API.
      1. Adds a new method `handleSnapshot` to `raft.Listener` which is called whenever the `RaftClient` determines that the `Listener` needs to load a new snapshot before reading the log. This happens when the `Listener`'s next offset is less than the log start offset also known as the earliest snapshot.
      2.  Adds a new type `SnapshotReader<T>` which provides a `Iterator<Batch<T>>` interface and de-serializes records in the `RawSnapshotReader` into `T`s
      3.  Adds a new type `RecordsIterator<T>` that implements an `Iterator<Batch<T>>` by scanning a `Records` object and deserializes the batches and records into `Batch<T>`. This type is used by both `SnapshotReader<T>` and `RecordsBatchReader<T>` internally to implement the `Iterator` interface that they expose. 
      4. Changes the `MockLog` implementation to read one or two batches at a time. The previous implementation always read from the given offset to the high-watermark. This made it impossible to test interesting snapshot loading scenarios.
      5. Removed `throws IOException` from some methods. Some of types were inconsistently throwing `IOException` in some cases and throwing `RuntimeException(..., new IOException(...))` in others. This PR improves the consistent by wrapping `IOException` in `RuntimeException` in a few more places and replacing `Closeable` with `AutoCloseable`.
      6. Updated the Kafka Raft simulation test to take into account snapshot. `ReplicatedCounter` was updated to generate snapshot after 10 records get committed. This means that the `ConsistentCommittedData` validation was extended to take snapshots into account. Also added a new invariant to ensure that the log start offset is consistently set with the earliest snapshot.
      Reviewers: dengziming <swzmdeng@163.com>, David Arthur <mumrah@gmail.com>, Jason Gustafson <jason@confluent.io>
  9. 01 5月, 2021 5 次提交
  10. 30 4月, 2021 4 次提交
    • V
      KAFKA-12396: added null check for state stores key (#10548) · e454becb
      Valery Kokorev 提交于
      Reviewers: Bruno Cadonna <bruno@confluent.io>, Matthias J. Sax <matthias@confluent.io>
    • A
      MINOR: clean up some remaining locking stuff in StateDirectory (#10608) · 9dbf2226
      A. Sophie Blee-Goldman 提交于
      Minor followup to #10342 that I noticed while working on the NamedTopology stuff. Cleans up a few things:
      We no longer need locking for the global state directory either, since it's contained within the top-level state directory lock. Definitely less critical than the task directory locking, since it's less vulnerable to IOExceptions given that it's just locked and unlocked once during the application lifetime, but nice to have nonetheless
      Clears out misc. usages of the LOCK_FILE_NAME that no longer apply. This has the awesome side effect of finally being able to actually delete obsolete task directories, whereas previously we had to leave behind the empty directory due to a ridiculous Windows bug (though I'm sure they would claim "it's not a bug it's a feature" 😉 )
      Lazily delete old-and-now-unused lock files in the StateDirectory#taskDirIsEmpty method to clean up the state directory for applications that upgraded from an older version that still used task locking
      Reviewers: Walker Carlson <wcarlson@confluent.io>
    • C
      MINOR: clean up some replication code (#10564) · 8d38189e
      Colin Patrick McCabe 提交于
      Centralize leader and ISR changes in generateLeaderAndIsrUpdates.
      Consolidate handleNodeDeactivated and handleNodeActivated into this
      Rename BrokersToIsrs#noLeaderIterator to BrokersToIsrs#partitionsWithNoLeader.
      Create BrokersToIsrs#partitionsLedByBroker, BrokersToIsrs#partitionsWithBrokerInIsr
      In ReplicationControlManagerTest, createTestTopic should be a member
      function of ReplicationControlTestContext.  It should invoke
      ReplicationControlTestContext#replay so that records are applied to all
      parts of the test context.
      Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
    • R
      KAFKA-12265; Move the BatchAccumulator in KafkaRaftClient to LeaderState (#10480) · a855f6ac
      Ryan 提交于
      The KafkaRaftClient has a field for the BatchAccumulator that is only used and set when it is the leader. In other cases, leader specific information was stored in LeaderState. In a recent change EpochState, which LeaderState implements, was changed to be a Closable. QuorumState makes sure to always close the previous state before transitioning to the next state. This redesign was used to move the BatchAccumulator to the LeaderState and simplify some of the handling in KafkaRaftClient.
      Reviewers: José Armando García Sancio <jsancio@gmail.com>, Jason Gustafson <jason@confluent.io>
  11. 29 4月, 2021 3 次提交
    • R
      KAFKA-12730; Avoid duplicate logout if Kerberos login fails (#10611) · eaa14a8a
      Rajini Sivaram 提交于
      From Java 9 onwards, LoginContext#logout() throws an NPE if invoked multiple times due to https://bugs.openjdk.java.net/browse/JDK-8173069. KerberosLogin currently attempts logout followed by login in a background refresh thread. If login fails we retry the same sequence. As a result, a single login failure prevents subsequent re-login. And clients will never be able to authenticate successfully after the first failure, until the process is restarted. The commit checks if logout is necessary before invoking LoginContext#logout(). Also adds a test for this case.
      Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
    • D
      MINOR: rename wrong topic id variable name and description (#10598) · f9de25f0
      dengziming 提交于
      Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
    • S
      KAFKA-10847: Fix spurious results on left/outer stream-stream joins (#10462) · bf359f8e
      Sergio Peña 提交于
      Fixes the issue with https://issues.apache.org/jira/browse/KAFKA-10847.
      To fix the above problem, the left/outer stream-stream join processor uses a buffer to hold non-joined records for some time until the window closes, so they are not processed if a join is found during the join window time. If the window of a record closes and a join was not found, then this should be emitted and processed by the consequent topology processor.
      A new time-ordered window store is used to temporary hold records that do not have a join and keep the records keys ordered by time. The KStreamStreamJoin has a reference to this new store . For every non-joined record seen, the processor writes it to this new state store without processing it. When a joined record is seen, the processor deletes the joined record from the new state store to prevent further processing.
      Records that were never joined at the end of the window + grace period are emitted to the next topology processor. I use the stream time to check for the expiry time for determinism results . The KStreamStreamJoin checks for expired records and emit them every time a new record is processed in the join processor.
      The new state store is shared with the left and right join nodes. The new store needs to serialize the record keys using a combined key of <joinSide-recordKey>. This key combination helps to delete the records from the other join if a joined record is found. Two new serdes are created for this, KeyAndJoinSideSerde which serializes a boolean value that specifies the side where the key is found, and ValueOrOtherValueSerde that serializes either V1 or V2 based on where the key was found.
      Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>