1. Project Overview

Kooder is a code search system developed by the Gitee team, providing code search services for Gitee/GitLab/Gitea.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

Search Interface Preview

4. Feature Overview

Background

An enterprise often has a large number of projects, each containing a substantial amount of code. This code represents one of the enterprise's core assets. Over time, different developers continuously modify and improve the code, making it difficult for anyone within the enterprise to understand all of it. Therefore, searching across the entire enterprise codebase becomes extremely important.

For example, we can search company code to determine whether it contains certain types of sensitive information or uses potentially unsafe methods.

What Is Kooder?

Kooder is an open-source code search tool designed to provide automatic source code, repository, and issue search services for code hosting platforms including Gitee/GitLab/Gitea.

5. Tech Stack

Kooder Architecture

Kooder consists of two modules: gateway and indexer (under the default configuration, indexer is integrated into gateway). The gateway receives indexing tasks via HTTP, validates them, and stores them in a queue. It also handles search requests and returns search results to clients. The indexer process monitors indexing tasks in the queue and updates the index database by executing tasks to add, delete, and modify indexes.

Module explanation

• Core Objects and Common Classes
• gateway handles HTTP requests for indexing and searching
• indexer service for building, updating, and deleting indexes

Data Flow Diagram

Source Code Installation

1. Dependencies

• openjdk >= 8
• maven > 3

2. Download the Source Code

Visit Exmay Open Source: [https://code.exmay.com/](https://code.exmay.com/) to download
$ cd kooder

Pre-Launch Preparation

Configuration File: kooder.properties

1. Configure the HTTP Service

http.url The URL of Kooder. This address is used as the webhook URL injected into the Git service. It must be accessible by the Git service, for example: http.url = http\://<kooder-host>:8080

http.port The HTTP port on which Kooder runs

2. Configure the GitLab Service URL

Kooder currently supports Gitee, GitLab, and Gitea. Support for other services is under development.

gitlab.url The URL of the GitLab homepage

gitlab.personal_access_token The Personal Access Token of the GitLab administrator account root

For more configuration options, see configuration.md

3. Build and Run

$ cd Kooder
$ mvn install
### Grant Execution Permissions to Shell Scripts
$ chmod +x bin/\*.sh
### Start the gateway
$ bin/gateway.sh
### Open in a Browser: [http://localhost:8080](http://localhost:8080)

Kooder Search API

@see API.md

Docker Installation

docker-compose

Dependency

• Docker CE Environment
• docker-compose

After optimizing the development code, deployment only requires cloning the code, setting up a container platform on the server, and executing the following commands on the platform:

### Ready to Use
docker-compose up -d 

### Stop the Containers
docker-compose down

docker-compose HA Version

Dependency

• Docker CE Environment
• docker-compose

After optimizing the development code, deployment only requires cloning the code, setting up a container platform on the server, and executing the following commands on the platform:

### Ready to Use
docker-compose -f docker-compose-ha.yaml up -d

### Stop the Containers
docker-compose -f docker-compose-ha.yaml down

The resulting setup is as follows:

Configuration file: core/src/main/resources/kooder.properties. After modifying the configuration file, execute the following command:

docker-compose down
docker-compose up -d

Check the Service Startup Status

docker logs -f CONTAINER_ID
......
2021-04-07 13:28:49 INFO [gateway] - Tasks [indexer,gitee] started.
2021-04-07 13:28:49 INFO [gateway] - READY (*:8080)!
.......

The log information above indicates that the service has started successfully.

note:

mvn install is executed every time the service starts to ensure that the configuration changes take effect.

Integration with Different Platforms

Integrating with GitLab

Configure the following items:

http.startup.tasks = indexer,gitlab
gitlab.url = http://gitlab-host:gitlab-port/  
gitlab.personal_access_token = <root user personal access token>  
git.username = root  
git.password =

If no password is provided, Kooder will automatically use the access token as the password.

Integrating with Self-Hosted Gitee

Configure the following items:

http.startup.tasks = indexer,gitee
gitee.url = https://<gitee-host>/  
gitee.personal_access_token = <root user personal access token>  
git.username = root  
git.password =

Integrating with Gitea

1. Log in to the Gitea administration panel;

2. Add a Gitea Webhook;

3. Configure the Webhook.

Set the URL to http://kooder-ip:kooder-port/gitea. Use POST + application/json as the request method and content type. For trigger conditions, select either all events or custom events.

If you select custom events, you need to check the repository events for push and repository options, as well as the issue option under issue events.

2. Configure the following items

http.startup.tasks = indexer,gitea
gitea.secret_token = <webhook secret token>
gitea.url = http://gitea-ip:prot/
gitea.personal_access_token = <admin personal access token>
git.username = <admin username>
git.password = <admin password>

Index Repositories from a File

Kooder currently supports indexing repositories from a single hosting platform at a time. To index repositories from multiple hosting platforms, import them separately.

Configure the following settings in the kooder.properties configuration file:

// Enable Kooder to index repositories from a file

http.startup.tasks = indexer,file // Add the file field

file.index.path = C:/Documents/Kooder/file.txt // Configure the file.index.path file path (file.txt is a local file; you can customize the path)

file.index.vender = gitee // Specify the corresponding code hosting platform (required)

Contents of file.txt

// Add repository URLs for the hosting platform specified in the configuration
[https://gitee.com/koode/kooder.git
https://gitee.com/ld/J2Cache.git
](https://gitee.com/koode/kooder.git
https://gitee.com/ld/J2Cache.git
)...

Build and Run

$ cd Kooder
$ mvn install
### Start the gateway
$ bin/gateway.sh
### Open in a Browser: [http://localhost:8080](http://localhost:8080)