1. Project Overview

Lucene Server

Lucene Server is a simple, flexible, and high-performance search engine built on Spring Boot and the Lucene core library. Through configuration, Lucene Server enables real-time synchronization of massive datasets with zero code. It provides enterprise applications with efficient, real-time search engine infrastructure services.

2. Open Source License

Licensed under MIT

3. Screenshots

System Screenshots:

[1] Home Page

  For task descriptions, please refer to the Wiki.

[2] Data Source Management

[3] Index Management

[4] Index Fields

For field configuration instructions, please refer to the wiki 

[5] Task Management

[6] CDC Management

[7] Data Browsing

4. Feature Overview

Key features

• Zero-Code Implementation: Lucene Server follows a configuration-first design philosophy, allowing users to define index structures and search rules through simple configuration, eliminating the need for complex code development.

• Support for Massive Datasets: Leveraging the powerful capabilities of Lucene, Lucene Server can process and search large-scale datasets. Whether dealing with documents, product catalogs, or user reviews, it responds to search queries quickly.

• Highly Customizable: Although no coding is required, Lucene Server provides a rich set of configuration options, allowing users to adjust indexing strategies, search algorithms, and result sorting according to specific requirements and personalized business scenarios.

• Easy Integration and Extensibility: Built on Spring Boot, Lucene Server can easily integrate with existing microservice architectures. It also supports adding new functional modules through a plugin mechanism, ensuring flexibility and extensibility.

• Powerful Search Syntax: Supports complex query syntax, including Boolean operations, phrase matching, fuzzy search, and range queries, making searches more accurate and intelligent.

• Real-Time Updates and High Availability: Through CDC (Change Data Capture), Lucene Server supports near-real-time data updates, ensuring search results remain up to date. Meanwhile, cluster deployment and load balancing provide high availability and fault recovery, ensuring stable service operation.

Main Functions

• Full-Text Search Service:

• Built on Spring Boot and Lucene, providing efficient full-text search capabilities similar to Elasticsearch.

• Web Management Interface:

• Includes an intuitive built-in Web interface supporting configurations such as data source integration, task scheduling, and permission management, simplifying system administration.

• Flexible Data Integration:

• Easily integrate various data sources through the Web interface, with excellent extensibility.

• Task Management and Monitoring:

• Provides task scheduling and monitoring to ensure the smooth execution of data processing and search tasks. For descriptions of various system tasks, please refer to the Wiki.

• Lightweight Client API:

• Provides an extremely lightweight Java client consisting of only two files. It adopts a query wrapper pattern similar to MyBatis-Plus for easy integration. Supports CRUD operations on index libraries, Count statistics, and Group aggregation queries to meet diverse search requirements. Supports client-side permission verification to ensure data security and access control. The client introduces no additional third-party libraries, keeping the system clean and stable.

• Easy Integration and Maintenance:

• Clear architecture and interface design reduce integration complexity and simplify routine maintenance.

Lucene Server is committed to providing a high-performance, easy-to-use, and secure full-text search solution suitable for a wide range of application scenarios.

5. Tech Stack

Data Flow

Index data changes are synchronized through three modes: (1) CDC tasks, (2) SQL tasks, and (3) the client.

The server and client communicate through the HTTP protocol to perform CRUD operations on data, as well as Group and Count queries.

Usage

1. Download the code and compile it successfully (only the backend needs to be compiled; the frontend does not require compilation. It is entirely hand-written, with no external dependencies or references to webpack, nodejs, etc.).
2. Make sure the following three directories exist in the root directory: database, queue, storage, and webapp.
3. Modify the system's database connection settings. In the web project's resources/config directory, open db.setting and update the database path accordingly.
4. Start the system and visit [http://localhost:9999/manager/index.html](http://localhost:9999/manager/index.html) to access the management interface. Username and password: admin.
5. Create a data source first.
6. Create indexes and configure fields. (Note: When creating an index for the first time, you must open the field settings and save them once.)
7. Create tasks or CDC sources as needed (CDC currently supports MySQL only. Support for other database types is still under development. If you have specific requirements, feel free to submit them in Issues.)

Client Examples

public static void testQuery(String[] args) {
        QueryClient queryClient = new QueryClient("http://localhost:9999","client","123456");
        ClientWrapper clientWrapper = ClientWrapper.create()
                .SELECT("*")
                .Page(1,10)
                .From("supervision")
                .eq("main_kind","1");
        System.out.println(clientWrapper.build());
        String str = queryClient.request(clientWrapper);
        System.out.println(JsonUtil.formatJson(str) );
}

For more examples, refer to the test directory in the client project and the Wiki.

Performance:

Test machine CPU: i7-10100 2.9 GHz. Memory usage during stress testing did not exceed 600 MB.

[Search Performance]

1.8 million records, 80 fields, 100,000 queries with 3 query conditions and 1 sorting condition. 50 threads, 100,000 searches completed in 24 seconds, achieving 4,152 requests per second.

[SQLTask Indexing Performance]

1.8 million records, 80 fields: approximately 2 minutes and 30 seconds in Idea testing and approximately 1 minute and 30 seconds in deployment testing.
23 million records, 20 fields: approximately 10 minutes.

[CDC Indexing Performance]

1.8 million records, 80 fields: approximately 10 minutes. Since CDC uses the UPDATE mode, it needs to build a Query to delete existing data before indexing. However, performance is primarily limited by the Canal Server.

Technology Stack:

1. Springboot 2.x
2. Lucene 8.x
3. MySql Binlog
4. Connection Pool: Custom-built dynamic database connection pool
5. Database: Sqlite3
6. Frontend Framework: Vue3 (no nodejs, no webpack, no vue-cli), Html + Vue + Vue Sfc Loader