1. Project Overview

Sharingan is a traffic recording and replay tool based on golang, recording real online requests for replay testing, suitable for project reconstruction, regression testing, etc.

Sharingan, Chinese name: Sharingan, is a technique in the manga Naruto, with the ability to replicate, illusory, etc., in the illusory world, all factors including time, location, quality, etc., are controlled by the practitioner.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

Effect Display

3.1, Traffic Replay

3.1.1, Single Replay

For scenarios with many and complex downstream requests, supports filtering downstream protocols

3.1.2, Batch Replay

Batch replay concurrency is 10 by default, and can be modified by increasing the -parallel parameter.

3.2, Coverage Report

3.2.1, Overall Report

Coverage report supports cumulative coverage results, i.e., supports generating coverage results after multiple single replays and batch replays.

5.2.1, Coverage Details

4. Feature Overview

4.1, Background

With the rise of microservices architecture, the dependency relationships between services have become increasingly complex, and software testing has faced new challenges: frequent system upgrades, numerous service dependencies, etc.

• Common testing schemes (such as unit testing, system testing, etc.) are costly to construct and maintain test cases, especially in complex business scenarios.It is difficult to construct test data.」
• There are many dependencies on third-party services, and the offline test environment is not very stable, and test failures often occur due to downstream services being unavailable.The cost of maintaining the test environment is high.」

Therefore, we need to develop a tool to alleviate the above problems.

4.2, Scheme

• Record real service requests online (including downstream call requests), and replay them offline to solve the problem of difficult test data construction.
• When replaying, match the Mock downstream calls, and no longer rely on specific downstream services to solve the high cost of maintaining the test environment.

4.3, Features

• Support downstream traffic recording. Compared to tcpcopy, goreplay, etc., replaying does not rely on the downstream environment.
• Support concurrent traffic recording and replaying. Recording has a small impact on services, and replaying is faster.
• Support time reset, noise removal, batch replay, coverage report, and common protocol parsing, etc.
• Support writing traffic replay, without polluting application data.
• Not dependent on business frameworks, low application immersion.

5. Tech Stack

Quick Start

Usage Examples

# Step1: Download the sharingan project
$ git clone https://github.com/didi/sharingan.git
$ cd sharingan

# Step2: Use a customized version of golang, for example go1.13 "slow? Try using a VPN"
$ sh install.sh go1.13 # Supports go1.10 ~ go1.15, limited to mac, linux amd64 systems
$ export GOROOT=/tmp/recorder-go1.13
$ export PATH=$GOROOT/bin:$PATH

# Step3: Compile and start the replayer-agent in the background "by default it will occupy ports 3515 and 8998, can be modified"
# [Replay Access Document] has a direct link to download the bin file, no need to build
$ cd replayer-agent
$ go build
$ nohup ./replayer-agent >> run.log 2>&1 &

# Step4: Compile and start the example in the background "by default it will occupy port 9999, can be modified"
$ cd ../example
$ go build -tags="replayer" -gcflags="all=-N -l"
$ nohup ./example >> run.log 2>&1 &

# Step5: Open the replay page
$ Open the browser, http://127.0.0.1:8998 # Replace IP if not local
$ Select the traffic point to be replayed and execute          # Built-in three example traffic records are provided

Access Documents

• Recording Access Document "Record First"
• Playback Access Document "Playback Later"

Technical Scheme

Module Division

• recorder: Traffic Recording Module, Record Traffic and Store Locally, Send Traffic to Recorder Agent, etc.
• recorder-agent: Traffic Recording Agent, Start as a Separate Process, Control Recording Ratio, Traffic Storage, etc.
• replayer: Traffic replay module, redirecting connections to Mock Server, Mock time, and adding traffic marks.
• replayer-agent: Traffic replay agent, running as a separate process, querying traffic, querying/reporting noise, traffic diff, batch replay, and generating coverage reports, etc.

Overall architecture diagram

Recording scheme

• Modify golang source code, expose Hook interface to the outside. "All changes are tested through official test cases"
• Provide API to chain network calls between different goroutines. "Common http, mysql, redis traffic do not need special settings"
• Provide a separate agent to filter traffic and control the proportion.
• More references: Traffic recording implementation principle.

Playback Scheme

• Connection Redirect: Redirect all Connect network calls to Mock Server. "Installed with replayer-agent during installation"
• Traffic Matching: Mock Server will match a return based on the actual downstream requests of the service. "Mock downstream calls"
• Time Reset: Roll back the program execution time to the recording execution time, and try to avoid the interference caused by time factors.
• Noise Removal: Provide APIs to remove known noise traffic, such as: traceID, each request is different every time.
• Common Protocol Parsing: Parse http, mysql, redis, thrift, etc. protocols, making it easier to compare and contrast.
• More References: Traffic Playback Implementation Principle.

Evolutionary Road

About Traffic Recording and Playback, we have explored it internally multiple times, mainly experiencing the following three stages:

Moonlight Treasure Box (Serial Recording, Serial Playback)

Recording: Use tcpdump to record traffic, modify the router layer to serialize requests, and use time to separate requests.

Playback: Use iptables to forward traffic to mock services, match requests, and mock returns. Supports time reset, traffic comparison, etc.

Insufficient: Low traffic coverage, can only record one request at a time. iptables forwarding, severe noise interference.

Fastdev (Parallel Recording, Serial Playback)

Recording: Modify the golang source code, use goroutine + task delegation technology to serialize and separate requests. Link tracking principle

Replay: Redirects connect system calls using mock libraries, forwards traffic to mock services. Supports Dashboard, noise removal, etc. monkey mock principle

Insufficient: Recording interfaces and implementation mixing, golang multi-version support difficult. Does not support concurrent replay, startup phase traffic cannot be proxied and will fail, scheduled task traffic interference is serious.

Sharingan (Parallel Recording, Parallel Replay)

Recording: Interface and implementation separation, golang source code modification part only exposes interface, specific recording implementation is provided separately in a package to support. Ensures golang source code changes pass official testing, supports all versions from 1.10 to 1.14; Optimizes recording service performance.

Replay: Adds traffic identification, supports concurrent replay; Supports startup phase traffic proxying; Uses customized golang to eliminate scheduled task traffic interference; Time reset no longer depends on local files, supports replayer-agent independent deployment; Supports common protocol parsing.

Thanks

Special thanks to TaoWen, the initial designer of the traffic recording and replay version, for laying a good foundation for subsequent open-source.