SASL/SCRAM (Salted Challenge Response Authentication Mechanism) 1. 주키퍼를 실행시킨 후, 주키퍼에 Broker 간 통신에 사용할 Credential(인증정보) 생성
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
'SCRAM-SHA-256=[iterations=8192,password=admin-password]' --entity-type users --entity-name admin
2. 주키퍼에 Producer/Consumer 에서 사용할 Credential(인증정보) 생성 bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
'SCRAM-SHA-256=[iterations=8192,password=password]' --entity-type users --entity-name username 3. JAAS(Java Authentication and Authorization Service) config(kafka_server_jaas.conf) 에 Broker 용 인증정보 설정 KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin-password";
}; 4. Kafka Broker config(server.properties) 에 인증정보 설정 listeners=SASL_PLAINTEXT://localhost:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.enabled.mechanisms=SCRAM-SHA-256 SASL/SCRAM (Salted Challenge Response Authentication Mechanism) 5. Kafka Broker 실행시 JAAS config 를 사용하도록 kafka_server_jaas.conf 파일 경로를 KAFKA_OPTS 옵션에 추가한 후 Kafka Broker 를 실행
export KAFKA_OPTS="-Djava.security.auth.login.config=/Users/ocg/Downloads/kafka_2.13-2.8.2/config/kafka_server_jaas.conf"
6 -1. Java Producer 의 Properties 에 SASL/SCRAM 인증정보를 추가하여 실행하여 확인 ...
Properties configs = new Properties();
...
configs.put("security.protocol", "SASL_PLAINTEXT");
configs.put("sasl.mechanism", "SCRAM-SHA-256");
configs.put("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required
username='alice' password='alice-password';");
KafkaProducer<String, String> producer = new KafkaProducer<>(configs);
...
6-2. 또는, Producer 쪽에 SASL/SCRAM 인증정보를 별도의 파일로 만들어 놓고 실행할 수도 있음(producer.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='alice'
password='alice-password';
6-2. 현재 Kafka Broker 에서는 인증을 요구하고 있으므로 아래와 같이 CLI 호출시 인증정보를 포함하여 호출한다 bin/kafka-console-producer.sh --topic topic5 --bootstrap-server localhost:9092 --producer.config ./producer.propertieS
Kafka Monitoring Tool
1. CMAK(Kafka Manager) - by yahoo. Managing Cluster, Topic, Offset
a. https://github.com/yahoo/CMAK
2. Burrow - by linkedin. focusing lag of offset
a. https://github.com/linkedin/Burrow
3. Xinfra Monitor(Kafka Monitor) - by linkedin
a. https://github.com/linkedin/kafka-monitor
4. Cruise Control - by linkedin
a. https://github.com/linkedin/cruise-control
5. Exporter + Prometheus + Grafana
a. https://github.com/prometheus/jmx_exporter
b. https://github.com/danielqsj/kafka_exporter
c. https://github.com/prometheus/node_exporter
d. https://prometheus.io/
e. https://grafana.com/
