when i use kafka-go why do i see what seems to be batching readswrites? is there a configuration i am missing?

You need to change ReaderConfig.MinBytes, otherwise segmentio/kafka-go will set it to 1e6 = 1 MB, and in that case, Kafka will wait for that much data to accumulate before answering the request.

func newReader(url string, topic string, partition int, dialer *kafka.Dialer) *kafka.Reader {

    return kafka.NewReader(kafka.ReaderConfig{
        Brokers:   []string{url},
        Topic:     topic,
        Dialer:    dialer,
        Partition: partition,
        MinBytes:  1,         // same value of Shopify/sarama 
        MaxBytes:  57671680,
    })
}

On the other hand, the default for shopify/sarama is 1 byte.

References:

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top