1. Project Overview

One-stop Apache Kafka management platform

LogiKM has been open-sourced and attracted much attention, considering that open-source projects should be more in line with the future development direction of Apache Kafka, after careful consideration by the project team, we will upgrade the brand to Know Streaming, and the new version is ready to be updated soon, thank you for your continued support! We also welcome Kafka enthusiasts to join the community together

By reading this README document, you can learn about the user groups and product positioning of Didi Know Streaming, and quickly experience the full process of Kafka cluster monitoring and operation and maintenance through the experience address

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

Didi Know Streaming is born out of Didi's internal years of Kafka operation practice experience, and is a shared multi-tenant Kafka cloud platform designed for Kafka users and Kafka operation personnel. It focuses on Kafka operation and maintenance, monitoring, alarm, and resource governance, and has undergone rigorous testing with large-scale clusters and massive big data. At the same time, it has achieved high satisfaction of 90% internally, and has also reached commercial cooperation with multiple well-known enterprises

1.1 Quick Experience Address

• Quick Experience Address (new quick experience address coming soon) view README.md document

1.2 Experience Map

Compared with similar products, Didi Logi-KafkaManager has established a role-based, multi-scene experience map, which includesUser Experience Map, Operation and Maintenance Experience Map, and Operation Experience Map

1.2.1 User Experience Map

• Platform Tenant Application: Apply for an application (App) as a Kafka username, and use AppID + password for identity authentication
• Cluster Resource Application: Apply for resources on demand, and use resources on demand. You can use the shared clusters provided by the platform, or apply for independent clusters for applications
• Topic Application: Create topics according to applications (Apps), or apply for read and write permissions for other topics
• Topic Operation and Maintenance: Sample data, adjust quotas, and apply for partitions, etc.
• Indicator Monitoring: Monitor performance indicators based on topic production and consumption time statistics, and monitor different quantile performance indicators
• Consumption Group Operation and Maintenance: Support resetting consumption offsets to a specified time or position

1.2.2 Operation and Maintenance Experience Map

• Multi-version Cluster Management: Supports from 0.10.2 to 2.x versions
• Cluster Monitoring: View historical and real-time key indicators of clusters, topics, and brokers in multiple dimensions, and establish a health score system
• Cluster Management: Divide part of the Broker as Region, use Region to define resource division unit, and divide logical cluster according to business and assurance capability
• Broker Management: including priority replica election and other operations
• Topic Management: including creation, query, expansion, attribute modification, migration, and offline

1.2.3 Operation Experience Map

• Resource Governance: sediment resource governance methods. Against Topic partition hotspots, partition shortages, etc. high-frequency common problems, sediment resource governance methods, achieve resource governance expertization
• Resource Review: Work order system. Topic creation, adjustment quotas, partition application, etc. operations, by professional operation and maintenance personnel review, standardize resource usage, ensure platform stable operation
• Accounting System: Cost Control. Topic resources, cluster resources, apply on demand, use on demand. According to traffic, calculate fees, help enterprises build big data cost accounting system

1.3 Core Advantage

• High-Efficient Problem Positioning: Monitor multiple core indicators, statistics different quantile data, provide rich types of indicator monitoring reports, help users, operation and maintenance personnel quickly and efficiently position problems
• Convenient Cluster Management: According to Region definition, divide cluster resources, define resource division unit, divide logical cluster according to assurance level. In the meantime, achieve resource isolation, improve expansion ability, and achieve strong control over service end
• Professional Resource Governance: Based on Didi internal multi-year operation practice, sediment resource governance methods, establish healthy division system. Against Topic partition hotspots, partition shortages, etc. high-frequency common problems, achieve resource governance expertization
• Friendly Operation and Maintenance Environment: Connect with Prometheus, Grafana, Didi Nightingale monitoring alarm system, integrate indicator analysis, monitoring alarm, cluster deployment, cluster upgrade, etc. capabilities. Form operation and maintenance environment, condense expert service, make operation and maintenance more efficient

5. Tech Stack

DiDi Logi-KafkaManager Architecture Diagram

Environment Dependencies

If it is installed with Release package, only install Java and MySQL. If it is to package with source package first, then use, need to install Maven and Node environment

• Java 8+(running environment needs)
• MySQL 5.7(data storage)
• Maven 3.5+(backend packaging dependency)
• Node 10+(frontend packaging dependency)