1. Project Overview

Distributed Retry Service Platform Easy-Retry

In today's world, where distributed systems are widely adopted, ensuring the accuracy and correctness of system data has become a major challenge. Based on the CAP theorem, adopting flexible transactions to ensure system availability and eventual data consistency has become a widely accepted technical approach. To ensure the availability and fault tolerance of distributed services, maintain service data consistency, and handle network issues between services, retry mechanisms are essential. According to Murphy's Law, adding retries to critical processes and performing data reconciliation and verification have become common technical solutions for improving system robustness.

2. Open Source License

Licensed under Apache-2.0

3. Screenshots

4. Feature Overview

Based on the principles of service governance, we developed retry governance capabilities in easy-retry. It supports dynamic configuration, requires virtually no modifications to business code for integration, and combines multiple strategies to control retry amplification at the link level. It is a distributed exception retry service platform that balances ease of use, flexibility, and security.

Common business scenarios include:

• Improving system stability by reducing exceptions caused by network fluctuations and adding retry capabilities.
• Improving service fault tolerance by splitting critical processes and performing data reconciliation during off-peak business hours.
• Ensuring message delivery by adding retries to notifications between services.

However, since retry processes are difficult to trigger under normal business conditions, development and testing teams often pay insufficient attention to retry scenarios and traffic. As a result, retry management remains in an important but unregulated management vacuum.

Easy-RETRY is a governance platform for retry traffic in business systems. It features high availability, high performance, and high load capacity. Its service capabilities include:

• Supports retry traffic distribution at the tens-of-millions level.
• Supports traffic capacity expansion, with automatic detection and processing.
• Supports horizontal scaling of traffic processing nodes.
• Efficiently utilizes system resources to support high concurrency.
• Supports multiple scheduling algorithms for client execution.
• Supports batched reporting for high-concurrency business scenarios.
• Encrypted communication to ensure information security.

Features

• Ease of Use Low business integration costs. Eliminates dependence on developers' technical expertise and ensures retry stability.
• Flexibility Supports dynamic configuration adjustments, task startup and shutdown, and termination of retry data currently being processed.
• Simple Operations Get started in minutes. Supports CRUD operations on retry data through a WEB interface.
• Data Dashboard Real-time monitoring and management of system retry data.
• Multiple Backoff Strategies Cron, fixed intervals, tier-based triggers, and random-time triggers.
• Containerized Deployment The server supports deployment in Docker containers.
• High-Performance Scheduling Platform Supports dynamic scaling of server nodes, including scaling up and down.
• Multiple Retry Types Supports multiple retry types, including ONLY_LOCAL, ONLY_REMOTE, and LOCAL_REMOTE.
• Retry Data Management Ensures retry data is not lost and supports one-click replay of retry data.
• Multiple Alerting Methods Email, WeCom, DingTalk, and Feishu.

5. Tech Stack

Client-Server Data Interaction Diagram

System architecture diagram

The Importance of Retries in Distributed Systems

In distributed systems, operations may fail due to network latency, node failures, message loss, and other factors. Retry mechanisms are particularly important in distributed systems for the following reasons:

• Network Unreliability: In distributed systems, network unreliability may cause messages to be lost or delayed. Retry mechanisms ensure that messages are sent and received, thereby improving system reliability and message delivery reliability.
• Transaction Processing: In distributed systems, transactions may fail due to network latency and other issues. Retry mechanisms help ensure that transactions are committed correctly, thereby guaranteeing eventual data consistency.
• Node Failures: Distributed systems may experience partial or complete service outages due to node failures. Retry mechanisms ensure that services automatically recover and continue running after failed nodes restart.
• Improving system availability: The retry mechanism ensures that the system can attempt to recover automatically when exceptional situations occur, preventing system crashes or unresponsiveness and thereby improving system availability and stability. Retry mechanisms are particularly important in distributed systems, as they can improve system reliability, performance, and message delivery reliability while ensuring data consistency and service availability.

Risks of Retries

• Retry Amplification Risk: This exponential amplification effect can be extremely dangerous. It increases the overall system load, and in the worst-case scenario, the traffic to the called service may be amplified by up to r times. This not only prevents requests from succeeding but also causes the overall load to continue increasing, potentially bringing the entire service down.
• Infinite Retry Problem: If no retry limit is configured, business threads may remain occupied by repeated retries indefinitely, causing the number of busy service threads to increase dramatically until the service crashes.
• Data Security: Memory-based retries may result in data loss.
• Network Congestion: Too many retries or retry intervals that are too short may cause a large number of requests to be sent simultaneously, resulting in network congestion and increased system load.

Comparison of Mature Retry Components in the Industry

Differences

SpringRetry

GuavaRetry

EasyRetry

Programming Language

Java

Java

Java

Backoff Strategy

Supports Multiple Strategies

Supports Multiple Strategies

Supports Multiple Strategies

Dependency Ecosystem

Spring Framework

No Framework Dependencies

Spring Framework, GuavaRetry

Retry Type

In-Memory Retries

In-Memory Retries

Multiple Strategies: In-Memory Retries + Server-Side Retries

Storage Medium

Memory

Memory

Memory + Database

Retry Traffic Control

No

No

Supports Multi-Dimensional Control (Single-Instance Retry Control, Link-Level Retry Control, Retry Rate Control, etc.)

Data Security

Retry data may be lost

Retry data may be lost

Persists data using LOCAL_REMOTE or ONLY_REMOTE

Retry Data Management

Not supported

Not supported

Supports pausing, stopping, adding, and modifying retry data

Source: Easy-Retry Official Documentation