生产者
依赖
1 2 3 4 5
| <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>
|
配置
下面是部分配置参考
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
|
spring.kafka.bootstrap-servers=127.0.0.1:9092
spring.kafka.producer.retries=3
spring.kafka.producer.acks=all
spring.kafka.producer.batch-size=16384
spring.kafka.producer.properties.linger.ms=0
spring.kafka.producer.buffer-memory = 33554432
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
|
可通过代码设置Topic
1 2 3 4 5 6 7 8 9 10 11 12 13 14
| @Configuration public class KafkaConfig {
public final static String TEST_TOPIC = "testTopic";
@Bean public NewTopic initialTopic() { return new NewTopic(TEST_TOPIC, 4, (short) 1); }
}
|
代码
1 2 3 4 5 6 7 8 9 10 11
| @Autowired private KafkaTemplate<String, Object> kafkaTemplate;
@GetMapping("/send/{message}") public void send(@PathVariable String message) { ListenableFuture<SendResult<String, Object>> listenableFuture = kafkaTemplate.send(KafkaConfig.TEST_TOPIC, message); listenableFuture.addCallback( sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()), throwable -> logger.info("failure : message is {}", throwable.getMessage()) ); }
|
spring-kafka只支持普通发送和事务发送
消费者
依赖
这里依赖和生产者一样
1 2 3 4 5
| <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>
|
配置
以下是部分配置介绍
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
| spring.application.name=consumer
server.port=8090
spring.kafka.bootstrap-servers=127.0.0.1:9092
spring.kafka.consumer.properties.group.id=defaultConsumerGroup
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-offset-reset=latest
spring.kafka.consumer.properties.session.timeout.ms=12000
spring.kafka.consumer.properties.request.timeout.ms=60000
spring.kafka.consumer.properties.isolation.level=read_committed
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
logging.level.org.apache.kafka.clients.consumer.ConsumerConfig= warn
|
代码
1 2 3 4 5 6 7 8 9 10 11
| @Slf4j @Component public class OneTopicReceiver{
@KafkaListener(topics = {"twoTopic"}, errorHandler = "consumerAwareErrorHandler", concurrency = "4", containerFactory = "filterContainerFactory") public void onReceive(Message message, Acknowledgment ack){ log.info("{}:{}",message.getHeaders().get("kafka_receivedPartitionId"), message.getPayload()); ack.acknowledge(); }
}
|
消费者通过@KafkaListener注解开启监听,主要属性包括主题(可以监听多个主题),配置工厂,消费者组,消费并发数等
事务
开启事务
修改application.properties配置文件
1 2
| //这里可以随意设置 spring.kafka.producer.transaction-id-prefix=trans-
|
1 2 3 4
| public boolean transactionCapable() { return this.transactionIdPrefix != null; }
|
设置消费者隔离级别
1 2
| spring.kafka.consumer.properties.isolation.level=read_committed
|
如果不设置隔离级别,默认会是读未提交(read_uncommitted),即使生产者端因为报错没有调用producer.commitTransaction(),而是调用的producer.abortTransaction(),但是消费者仍然可以消费到这些没提交的信息,所以需要将隔离级别设置成读已提交(read_committed)
使用方法
使用executeInTransaction方法
1 2 3 4 5 6 7
| kafkaTemplate.executeInTransaction(operations -> { for (int i = 0; i < 100; i++) { ListenableFuture<SendResult<String, Object>> listenableFuture = operations.send("twoTopic", String.valueOf(i)); listenableFuture.addCallback(sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()), throwable -> logger.info("failure : message is {}",throwable.getMessage())); } return ""; });
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
| public <T> T executeInTransaction(OperationsCallback<K, V, T> callback) { Assert.notNull(callback, "'callback' cannot be null"); Assert.state(this.transactional, "Producer factory does not support transactions"); Producer<K, V> producer = (Producer)this.producers.get(); Assert.state(producer == null, "Nested calls to 'executeInTransaction' are not allowed"); String transactionIdSuffix; if (this.producerFactory.isProducerPerConsumerPartition()) { transactionIdSuffix = TransactionSupport.getTransactionIdSuffix(); TransactionSupport.clearTransactionIdSuffix(); } else { transactionIdSuffix = null; }
producer = this.producerFactory.createProducer();
try { producer.beginTransaction(); } catch (Exception var15) { this.closeProducer(producer, false); throw var15; }
this.producers.set(producer);
Object var5; try { Object result = callback.doInOperations(this);
try { producer.commitTransaction(); } catch (Exception var12) { throw new KafkaTemplate.SkipAbortException(var12); }
var5 = result; } catch (KafkaTemplate.SkipAbortException var13) { throw (RuntimeException)var13.getCause(); } catch (Exception var14) { producer.abortTransaction(); throw var14; } finally { if (transactionIdSuffix != null) { TransactionSupport.setTransactionIdSuffix(transactionIdSuffix); }
this.producers.remove(); this.closeProducer(producer, false); }
return var5; }
|
使用Transactional注解
1 2 3 4 5 6 7 8
| @GetMapping("/sendTransaction/{message}") @Transactional public void sendTransaction(@PathVariable String message) { for (int i = 0; i < 100; i++) { ListenableFuture<SendResult<String, Object>> listenableFuture = kafkaTemplate.send("twoTopic", String.valueOf(i)); listenableFuture.addCallback(sendResult -> logger.info("success: partition is {}", sendResult.getRecordMetadata().partition()), throwable -> logger.info("failure : message is {}", throwable.getMessage())); } }
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
| protected Object invokeWithinTransaction(Method method, @Nullable Class<?> targetClass, TransactionAspectSupport.InvocationCallback invocation) throws Throwable { TransactionAttributeSource tas = this.getTransactionAttributeSource(); TransactionAttribute txAttr = tas != null ? tas.getTransactionAttribute(method, targetClass) : null; PlatformTransactionManager tm = this.determineTransactionManager(txAttr); String joinpointIdentification = this.methodIdentification(method, targetClass, txAttr); Object result; if (txAttr != null && tm instanceof CallbackPreferringPlatformTransactionManager) { } else { TransactionAspectSupport.TransactionInfo txInfo = this.createTransactionIfNecessary(tm, txAttr, joinpointIdentification);
try { result = invocation.proceedWithInvocation(); } catch (Throwable var17) { this.completeTransactionAfterThrowing(txInfo, var17); throw var17; } finally { this.cleanupTransactionInfo(txInfo); } this.commitTransactionAfterReturning(txInfo); return result; } }
|
总结
- 如果采用@Transactional注解的方式,相当于采用的spring申明式事务
- 如果采用executeInTransaction方法,相当于是单独实现了一套事务
本质还是kafka的五个方法,类似于数据库以及RabbitMQ的事务
- **initTransactions()**方法用来初始化事务,这个方法能够执行的前提是配置了transactionalId,如果没有则会报出IllegalStateException:
- **beginTransaction()**方法用来开启事务;
- **sendOffsetsToTransaction()**方法为消费者提供在事务内的位移提交的操作;
- **commitTransaction()**方法用来提交事务;
- **abortTransaction()**方法用来中止事务,类似于事务回滚。