生产者

依赖

1
2
3
4
5
<!--    spring boot包含了此依赖的版本,所以尽量不自定义版本    -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>

配置

下面是部分配置参考

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
###########【Kafka集群】###########
#spring.kafka.bootstrap-servers=192.168.255.101:9090,192.168.255.101:9091,192.168.255.101:9092
spring.kafka.bootstrap-servers=127.0.0.1:9092

###########【初始化生产者配置】###########
# 重试次数
spring.kafka.producer.retries=3
# 应答级别:多少个分区副本备份完成时向生产者发送ack确认(可选0、1、all/-1)
spring.kafka.producer.acks=all
# 批量大小
spring.kafka.producer.batch-size=16384
# 提交延时
# 当生产端积累的消息达到batch-size或接收到消息linger.ms后,生产者就会将消息提交给kafka
# linger.ms为0表示每接收到一条消息就提交给kafka,这时候batch-size其实就没用了
spring.kafka.producer.properties.linger.ms=0

#开启事务
#spring.kafka.producer.transaction-id-prefix=trans-

# 生产端缓冲区大小
spring.kafka.producer.buffer-memory = 33554432
# Kafka提供的序列化和反序列化类
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
# 自定义分区器
#spring.kafka.producer.properties.partitioner.class=com.kafka.producer.config.CustomizePartitioner

可通过代码设置Topic

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@Configuration
public class KafkaConfig {

public final static String TEST_TOPIC = "testTopic";

// 创建一个名为testTopic的Topic并设置分区数为4,分区副本数为1
// 如果要修改分区数,只需修改配置值重启项目即可
// 修改分区数并不会导致数据的丢失,但是分区数只能增大不能减小
@Bean
public NewTopic initialTopic() {
return new NewTopic(TEST_TOPIC, 4, (short) 1);
}

}

代码

1
2
3
4
5
6
7
8
9
10
11
@Autowired
private KafkaTemplate<String, Object> kafkaTemplate;

@GetMapping("/send/{message}")
public void send(@PathVariable String message) {
ListenableFuture<SendResult<String, Object>> listenableFuture = kafkaTemplate.send(KafkaConfig.TEST_TOPIC, message);
listenableFuture.addCallback(
sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()),
throwable -> logger.info("failure : message is {}", throwable.getMessage())
);
}

spring-kafka只支持普通发送和事务发送

消费者

依赖

这里依赖和生产者一样

1
2
3
4
5
<!--    spring boot包含了此依赖的版本,所以尽量不自定义版本    -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>

配置

以下是部分配置介绍

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 应用名称
spring.application.name=consumer
# 应用服务 WEB 访问端口
server.port=8090


###########【Kafka集群】###########
#spring.kafka.bootstrap-servers=192.168.255.101:9090,192.168.255.101:9091,192.168.255.101:9092
spring.kafka.bootstrap-servers=127.0.0.1:9092

###########【初始化消费者配置】###########
# 默认的消费组ID
spring.kafka.consumer.properties.group.id=defaultConsumerGroup
# 是否自动提交offset
spring.kafka.consumer.enable-auto-commit=false
# 提交offset延时(接收到消息后多久提交offset)
#spring.kafka.consumer.auto-commit-interval=1000
# 当kafka中没有初始offset或offset超出范围时将自动重置offset
# earliest:重置为分区中最小的offset;
# latest:重置为分区中最新的offset(消费分区中新产生的数据);
# none:只要有一个分区不存在已提交的offset,就抛出异常;
spring.kafka.consumer.auto-offset-reset=latest

# 消费会话超时时间(超过这个时间consumer没有发送心跳,就会触发rebalance操作)
spring.kafka.consumer.properties.session.timeout.ms=12000
# 消费请求超时时间
spring.kafka.consumer.properties.request.timeout.ms=60000
# 将consumer隔离级别设置成提交读
spring.kafka.consumer.properties.isolation.level=read_committed

# Kafka提供的序列化和反序列化类
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
# 消费端监听的topic不存在时,项目启动会报错(关掉)
#spring.kafka.listener.missing-topics-fatal=false
## 设置批量消费
#spring.kafka.listener.type=batch
## 批量消费每次最多消费多少条消息(可以配合生产者延时批量提交消息测试)
#spring.kafka.consumer.max-poll-records=50

#日志
logging.level.org.apache.kafka.clients.consumer.ConsumerConfig= warn

代码

1
2
3
4
5
6
7
8
9
10
11
@Slf4j
@Component
public class OneTopicReceiver{

@KafkaListener(topics = {"twoTopic"}, errorHandler = "consumerAwareErrorHandler", concurrency = "4", containerFactory = "filterContainerFactory")
public void onReceive(Message message, Acknowledgment ack){
log.info("{}:{}",message.getHeaders().get("kafka_receivedPartitionId"), message.getPayload());
ack.acknowledge();
}

}

消费者通过@KafkaListener注解开启监听,主要属性包括主题(可以监听多个主题),配置工厂,消费者组,消费并发数等

事务

开启事务

修改application.properties配置文件

1
2
//这里可以随意设置
spring.kafka.producer.transaction-id-prefix=trans-
1
2
3
4
//通过带代码可以看出,只要transaction-id-prefix配置不为null,就会开启事务
public boolean transactionCapable() {
return this.transactionIdPrefix != null;
}

设置消费者隔离级别

1
2
# 将consumer隔离级别设置成提交读(默认read_uncommitted)
spring.kafka.consumer.properties.isolation.level=read_committed

如果不设置隔离级别,默认会是读未提交(read_uncommitted),即使生产者端因为报错没有调用producer.commitTransaction(),而是调用的producer.abortTransaction(),但是消费者仍然可以消费到这些没提交的信息,所以需要将隔离级别设置成读已提交(read_committed)

使用方法

使用executeInTransaction方法

1
2
3
4
5
6
7
kafkaTemplate.executeInTransaction(operations -> {
for (int i = 0; i < 100; i++) {
ListenableFuture<SendResult<String, Object>> listenableFuture = operations.send("twoTopic", String.valueOf(i));
listenableFuture.addCallback(sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()), throwable -> logger.info("failure : message is {}",throwable.getMessage()));
}
return "";
});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
public <T> T executeInTransaction(OperationsCallback<K, V, T> callback) {
Assert.notNull(callback, "'callback' cannot be null");
Assert.state(this.transactional, "Producer factory does not support transactions");
Producer<K, V> producer = (Producer)this.producers.get();
Assert.state(producer == null, "Nested calls to 'executeInTransaction' are not allowed");
String transactionIdSuffix;
if (this.producerFactory.isProducerPerConsumerPartition()) {
transactionIdSuffix = TransactionSupport.getTransactionIdSuffix();
TransactionSupport.clearTransactionIdSuffix();
} else {
transactionIdSuffix = null;
}

producer = this.producerFactory.createProducer();

try {
//开始事务
producer.beginTransaction();
} catch (Exception var15) {
this.closeProducer(producer, false);
throw var15;
}

this.producers.set(producer);

Object var5;
try {
//真正进行的操作,即上面的发送100条消息
Object result = callback.doInOperations(this);

try {
//提交事务
producer.commitTransaction();
} catch (Exception var12) {
throw new KafkaTemplate.SkipAbortException(var12);
}

var5 = result;
} catch (KafkaTemplate.SkipAbortException var13) {
throw (RuntimeException)var13.getCause();
} catch (Exception var14) {
//放弃事务,类似于回滚
producer.abortTransaction();
throw var14;
} finally {
if (transactionIdSuffix != null) {
TransactionSupport.setTransactionIdSuffix(transactionIdSuffix);
}

this.producers.remove();
this.closeProducer(producer, false);
}

return var5;
}

使用Transactional注解

1
2
3
4
5
6
7
8
@GetMapping("/sendTransaction/{message}")
@Transactional
public void sendTransaction(@PathVariable String message) {
for (int i = 0; i < 100; i++) {
ListenableFuture<SendResult<String, Object>> listenableFuture = kafkaTemplate.send("twoTopic", String.valueOf(i));
listenableFuture.addCallback(sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()), throwable -> logger.info("failure : message is {}", throwable.getMessage()));
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
protected Object invokeWithinTransaction(Method method, @Nullable Class<?> targetClass, TransactionAspectSupport.InvocationCallback invocation) throws Throwable {
TransactionAttributeSource tas = this.getTransactionAttributeSource();
TransactionAttribute txAttr = tas != null ? tas.getTransactionAttribute(method, targetClass) : null;
PlatformTransactionManager tm = this.determineTransactionManager(txAttr);
String joinpointIdentification = this.methodIdentification(method, targetClass, txAttr);
Object result;
if (txAttr != null && tm instanceof CallbackPreferringPlatformTransactionManager) {
//该部分代码比较复杂,没有贴出,可以自行查看,其实就是我没看懂,所以没贴
} else {
TransactionAspectSupport.TransactionInfo txInfo = this.createTransactionIfNecessary(tm, txAttr, joinpointIdentification);

try {
//这里代表我们实际的方法,即加上@Transactional注解的方法,spring在注册bean时,会扫描包含@Transactional注解的方法,然后生成代理方法,即aop拦截
result = invocation.proceedWithInvocation();
} catch (Throwable var17) {
//如果发生了报错,则回滚,调用PlatformTransactionManager的rollback(),最终调用producer.abortTransaction();
this.completeTransactionAfterThrowing(txInfo, var17);
throw var17;
} finally {
this.cleanupTransactionInfo(txInfo);
}
//如果执行成功,则提交,调用PlatformTransactionManager的commit(),最终调用producer.commitTransaction();
this.commitTransactionAfterReturning(txInfo);
return result;
}
}

总结

  1. 如果采用@Transactional注解的方式,相当于采用的spring申明式事务
  2. 如果采用executeInTransaction方法,相当于是单独实现了一套事务

本质还是kafka的五个方法,类似于数据库以及RabbitMQ的事务

  • **initTransactions()**方法用来初始化事务,这个方法能够执行的前提是配置了transactionalId,如果没有则会报出IllegalStateException:
  • **beginTransaction()**方法用来开启事务;
  • **sendOffsetsToTransaction()**方法为消费者提供在事务内的位移提交的操作;
  • **commitTransaction()**方法用来提交事务;
  • **abortTransaction()**方法用来中止事务,类似于事务回滚。

本站总访问量:次本站总访问数:人
本站由 Peng Tong 使用 Stellar 1.29.1 主题创建。
本博客所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议,转载请注明出处。