1. Project Overview

KnowAgent  has experienced multiple tests and has a perfect solution for the governance and control of large-scale Agent log collection, including the observability, health inspection, and fault diagnosis of collection tasks and Agents.KnowAgent  is based on application granularity, meeting the demand for creating collection tasks from a business perspective, and is easy to integrate file collection as a reliable data source for stream computing.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

1.1 Pre-declaration

1.1.1 Agent and collection task control scale

​ KnowAgent Based on minimal dependencies, easy to experience, MySQL storage Agent's Metrics and Error Logs data. Limited by MySQL performance bottlenecks, in a single collection task corresponding to a collection path, Metrics and Error Logs data retention period is one week, supports 50 Agent and 50 collection task control. If you need to control more Agent and collection tasks, see 《How to replace Agent's Metrics and Error Logs data storage engine to manage larger Agent clusters》.

1.1.2 Container Collection

​ KnowAgentCurrently, container log collection has not been implemented for platform-level support. In the future, a comprehensive container log collection platform-level support will be provided, see 《KnowAgent One-stop Log Collection Platform Introduction》ProspectPartially.

1.2 Quick Experience Address

• View README.md document

1.3 User Experience Map

• My Workbench Operation Dashboard: Displays operation-related core indicator data. Operation and Maintenance Dashboard: Displays operation and maintenance-related core indicator data.
• Metadata Center Application Management: Maintain application-related information, andApplication-HostAssociation Relationship. Receive End Management: Maintain Agent data flow, indicator flow, and error log flow data that need to be written to the downstream receive end related information. Metadata Management: Maintain user-uploaded metadata Excel files (metadata Excel file content includes host and application information), and provide metadata file content preview and metadata import function.
• Agent Center Agent Version Management: Maintain information of Agent installation packages for each version. Agent Management: Maintain host-related information and Agent configuration, view Agent health status and fault reason (health status is yellow or red, display fault reason). After Agent startup, it will automatically register to the management platform with host information, and after successful registration, it can be managed.
• Collection Task Management: Maintain collection task-related information, view collection task health status and fault reason (health status is yellow or red, display fault reason).
• Monitoring Center: Agent Indicator Board: View Agent running-time panoramic indicators (including system, process, and Agent business indicators). Collection Task Indicator Board: Used to view collection task running-time panoramic indicators on each host.
• Operation and Maintenance Center Indicator Exploration: View Agent and collection task running-time panoramic indicators, support multiple Agent and multiple collection task comparison analysis and unified observation.

5. Tech Stack

1.4 Core Advantage

One-stop Log Collection Scheme

Highly reliable, high-performance, and all-round observable collection engine, with a management platform facing applications, easy to control, easy to observe, and easy to govern, greatly reduces log data collection access costs, and greatly improves log data collection access efficiency.

Easy to integrate with stream computing

Application-oriented collection tasks, highly reliable collection engine can ensure data integrity in any situation, perfect monitoring indicators, support reliable collection task data integrity verification, making collection tasks a reliable data source for stream computing, forming a data integrity closed loop for the entire stream computing link.

Production-level features

Through long-term large-scale collection engine cluster operation and maintenance experience, a full range of observability and perfect fault and risk perception and diagnosis capabilities are formed, making it easy for ordinary operation and maintenance personnel to easily operate large-scale collection engine clusters without prior knowledge.

High Performance

Excellent continuous collection performance, based on JDK11+, processing single row, single file, no parsing scenario, sending log data to Kafka, continuous throughput can reach 100MB/s above, extreme throughput can reach 150MB/s above, CPU single-core situation, continuous throughput can reach 40MB/s above.

Comparison with similar products

Dimension

Flume

FileBeat

Logi-Agent

Reliability

General

General

High Reliability

Performance Collection

Mb level

Ten Mb level

Hundred Mb level

Data Integrity

Cannot Ensure Data Integrity

Cannot Ensure Data Integrity

Can Ensure Data Integrity

Tenant Isolation at Collection Task Level

Cannot Achieve Tenant Isolation at Collection Task Level

Cannot Achieve Tenant Isolation at Collection Task Level

Can Achieve Tenant Isolation at Collection Task Level

Scalability

Good

General

General

Resource Consumption

The cost of maintaining the test environment is high.

Low

CPU Consumption and FileBeat are close (in large collection scenarios, CPU consumption is less than FileBeat, and memory consumption is slightly larger (but at the same order of magnitude))

Completeness of Monitoring Indicators

Few Indicators

Few Indicators

Perfect Indicator System

Observability

No

No

Full-Range Observability

Is it easy to configure

No

No

Yes

Complexity of Large-Scale Operation and Maintenance

Extremely High

Extremely High

Low

1.5 KnowAgent Architecture

​ KnowAgent It is a one-stop log collection platform, including two components:

• Agent：is based on Java, high-performance, multi-tenant isolation support, data collection engine, it can ensure the integrity of the collected data in any situation (except: the data to be collected is removed before collection, for example: log files are archived or log file rotation speed is too fast), the continuous collection speed can reach 150MB+/second.
• Agent Manager：is a management platform for Agent. Used to manage related metadata, Agent and collection tasks, automatic patrol Agent and collection tasks health, fault diagnosis, indicator display.

1.5.1 Agent architecture

1.5.2 Agent Manager architecture