# Securely Server Support Components Monorepo
This monorepo contains support crates for the Securely server, with most
components being inspired by Laravel's elegant interfaces and architecture. Each
crate aims to provide a Rust-native implementation of Laravel's well-established
patterns while embracing Rust's strengths in concurrency and type safety.
## Design Principles
1. **Laravel-Inspired Interfaces**: Each crate should closely mirror its Laravel
counterpart's interface where applicable
2. **Single Responsibility**: Crates should be focused and cohesive, handling
one core concern well
3. **Strong Interfaces**: Public APIs should be well-documented, intuitive, and
thorough
4. **Rust-Native**: While following Laravel patterns, implementations should
leverage Rust's unique features:
- Ownership and borrowing for memory safety
- Trait system for flexible interfaces
- Async/await for concurrent operations
- Type system for compile-time guarantees
5. **Comprehensive Testing**: Each crate must include:
- Unit tests for all public interfaces
- Integration tests for Laravel-equivalent functionality
- Documentation tests for usage examples
- Benchmarks for performance-critical paths
## Project Structure
/ ├── crates/ # All crates live here │ ├── config/ # Configuration management │
│ ├── src/ # Source code │ │ ├── tests/ # Integration tests │ │ └── Cargo.toml #
Config crate manifest │ │ │ ├── encrypt/ # Encryption utilities │ │ ├── src/ #
Source code │ │ ├── tests/ # Integration tests │ │ └── Cargo.toml # Encrypt
crate manifest │ │ │ ├── queue/ # Job queuing and processing │ │ ├── src/ #
Source code │ │ ├── tests/ # Integration tests │ │ └── Cargo.toml # Queue crate
manifest │ │ │ ├── settings/ # Settings management CLI │ │ ├── src/ # Source
code │ │ ├── tests/ # Integration tests │ │ └── Cargo.toml # Settings crate
manifest │ │ │ ├── store/ # Storage abstraction layer │ │ ├── src/ # Source code
│ │ ├── tests/ # Integration tests │ │ └── Cargo.toml # Store crate manifest │ │
│ ├── testing/ # Testing utilities │ │ ├── src/ # Source code │ │ └── Cargo.toml
# Testing crate manifest │ │ │ ├── workflows/ # Workflow engine and CLI │ │ ├──
src/ # Source code │ │ ├── tests/ # Integration tests │ │ └── Cargo.toml #
Workflows crate manifest │ │ └── Cargo.toml # Workspace manifest
## Current Crates
- config: Configuration management with dot notation and environment overrides
- encrypt: Encryption utilities for secure data storage and transmission
- queue: Async job queuing and processing with sled backend
- settings: Settings management CLI with encryption support
- store: Storage abstraction layer with multiple backends
- testing: Testing utilities for all components
- workflows: Workflow engine with activity handlers and async execution
## Important Notes
1. Each crate is independent but may have dependencies on other workspace crates
2. When running commands:
- Use `cargo test -p <crate-name>` to test specific crates
- Use `cargo test --all` to test all crates
- Always run commands from the workspace root unless specifically working in
a crate
3. Dependencies:
- Add crate-specific dependencies to the crate's own Cargo.toml
- Add workspace-wide settings to the root Cargo.toml
4. Current Crates:
- settings: Secure settings management CLI
- store: Storage abstraction layer
- workflows: Workflow engine and CLI
- testing: Testing utilities
## Settings: ./crates/settings/
The settings crate provides a secure CLI tool for managing application settings
with:
- Environment variable management
- Encrypted secrets using AES-256-CBC
- Server update hooks
- Hierarchical visualization
## Laravel Component Equivalents
Current and planned crates with their Laravel counterparts:
| Crate Name | Laravel Component | Status | Description |
| ---------------------- | ----------------- | ------- | ------------------------------------------------------------------------------ |
| securely settings:get | set | has | all |
| securely config:{} | Config | Active | Configuration management with dot notation, env overrides, and encryption |
| securely store:{} | Cache | Active | Storage abstraction with multiple backends |
| securely workflows:{} | Workflow | Active | Comprehensive workflow engine with CLI, activity handlers, and async execution |
| securely queue:{} | Queue | Planned | Async job queuing and processing |
| securely events:{} | Events | Planned | Event dispatching and handling |
| securely log:{} | Log | Planned | Structured logging with multiple channels |
| securely mail:{} | Mail | Planned | Email composition and delivery |
| securely validation:{} | Validation | Planned | Data validation and sanitization |
| securely schedule:{} | Schedule | Planned | Task scheduling and cron management |
Each crate maintains Laravel's interface patterns while implementing
Rust-specific optimizations and safety features.
## Implementation Guidelines
1. **Interface First**: Design the public API before implementation, ensuring it
matches Laravel's interface patterns while being idiomatic Rust
2. **Documentation**: Every public item must have comprehensive documentation
with examples
3. **Error Handling**: Use custom error types with thiserror, providing clear
context and recovery options
4. **Async Support**: Design for async from the start, using tokio as the
runtime
5. **Testing**:
- Unit tests for all public interfaces
- Integration tests comparing behavior with Laravel
- Benchmarks for performance-critical paths
- Documentation tests to verify examples
6. **Dependencies**:
- Keep dependencies minimal and justified
- Prefer standard library solutions where possible
- Use workspace dependencies for cross-crate functionality
## Development Workflow
1. **Starting a New Feature**:
- Create feature branch from main
- Update relevant crate's Cargo.toml if needed
- Implement tests first
- Document as you go
2. **Code Organization**:
- Keep modules focused and cohesive
- Use internal modules for implementation details
- Expose a clean, well-documented public API
- Follow Rust naming conventions
3. **Testing Strategy**:
- Unit tests alongside code
- Integration tests in tests/
- Benchmarks in benches/
- Use testing utilities from testing crate
4. **Documentation**:
- README.md in each crate root
- Rustdoc for all public items
- Examples in docs/
- Update .cursorrules when adding features
Note: We are building out a cross platform native virtual server/virtual
environment with a custom domain name server via hickory dns.
For our reverse proxy and background daemons we'll be using cloudflare's
pingora server.
Make our settings comprehensive based on
# Configuration
A Pingora configuration file is a list of Pingora settings in yaml format.
## Example yaml
version: 1 threads: 2 pid_file: /run/pingora.pid upgrade_sock:
/tmp/pingora_upgrade.sock user: nobody group: webusers
## Settings
| Key | meaning | value type |
| ---------------------------- | -------------------------------------------------------------------------------------------- | -------------- |
| version | the version of the conf, currently it is a constant 1 | number |
| pid_file | The path to the pid file | string |
| daemon | whether to run the server in the background | bool |
| error_log | the path to error log output file. STDERR is used if not set | string |
| upgrade_sock | the path to the upgrade socket. | string |
| threads | number of threads per service | number |
| user | the user the pingora server should be run under after daemonization | string |
| group | the group the pingora server should be run under after daemonization | string |
| client_bind_to_ipv4 | source IPv4 addresses to bind to when connecting to server | list of string |
| client_bind_to_ipv6 | source IPv6 addresses to bind to when connecting to server | list of string |
| ca_file | The path to the root CA file | string |
| work_stealing | Enable work stealing runtime (default true). See Pingora runtime (WIP) section for more info | bool |
| upstream_keepalive_pool_size | The number of total connections to keep in the connection pool | number |
## Extension
Any unknown settings will be ignored. This allows extending the conf file to add
and pass user defined settings. See User defined configuration section.
# Sharing state across phases with CTX
## Using CTX
The custom filters users implement in different phases of the request don't
interact with each other directly. In order to share information and state
across the filters, users can define a CTX struct. Each request owns a single
CTX object. All the filters are able to read and update members of the CTX
object. The CTX object will be dropped at the end of the request.
### Example
In the following example, the proxy parses the request header in the
request_filter phase, it stores the boolean flag so that later in the
upstream_peer phase the flag is used to decide which server to route traffic to.
(Technically, the header can be parsed in upstream_peer phase, but we just do it
in an earlier phase just for the demonstration.)
Rust pub struct MyProxy();
pub struct MyCtx { beta_user: bool, }
fn check_beta_user(req: &pingora_http::RequestHeader) -> bool { // some simple
logic to check if user is beta req.headers.get("beta-flag").is_some() }
#[async_trait] impl ProxyHttp for MyProxy { type CTX = MyCtx; fn new_ctx(&self)
-> Self::CTX { MyCtx { beta_user: false } }
async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {
ctx.beta_user = check_beta_user(session.req_header());
Ok(false)
}
async fn upstream_peer(
&self,
_session: &mut Session,
ctx: &mut Self::CTX,
) -> Result<Box<HttpPeer>> {
let addr = if ctx.beta_user {
info!("I'm a beta user");
("1.0.0.1", 443)
} else {
("1.1.1.1", 443)
};
let peer = Box::new(HttpPeer::new(addr, true, "one.one.one.one".to_string()));
Ok(peer)
}
}
## Sharing state across requests
Sharing state such as a counter, cache and other info across requests is common.
There is nothing special needed for sharing resources and data across requests
in Pingora. Arc, static or any other mechanism can be used.
### Example
Let's modify the example above to track the number of beta visitors as well as
the number of total visitors. The counters can either be defined in the MyProxy
struct itself or defined as a global variable. Because the counters can be
concurrently accessed, Mutex is used here.
Rust // global counter static REQ_COUNTER: Mutex<usize> = Mutex::new(0);
pub struct MyProxy { // counter for the service beta_counter: Mutex<usize>, //
AtomicUsize works too }
pub struct MyCtx { beta_user: bool, }
fn check_beta_user(req: &pingora_http::RequestHeader) -> bool { // some simple
logic to check if user is beta req.headers.get("beta-flag").is_some() }
#[async_trait] impl ProxyHttp for MyProxy { type CTX = MyCtx; fn new_ctx(&self)
-> Self::CTX { MyCtx { beta_user: false } }
async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {
ctx.beta_user = check_beta_user(session.req_header());
Ok(false)
}
async fn upstream_peer(
&self,
_session: &mut Session,
ctx: &mut Self::CTX,
) -> Result<Box<HttpPeer>> {
let mut req_counter = REQ_COUNTER.lock().unwrap();
*req_counter += 1;
let addr = if ctx.beta_user {
let mut beta_count = self.beta_counter.lock().unwrap();
*beta_count += 1;
info!("I'm a beta user #{beta_count}");
("1.0.0.1", 443)
} else {
info!("I'm an user #{req_counter}");
("1.1.1.1", 443)
};
let peer = Box::new(HttpPeer::new(addr, true, "one.one.one.one".to_string()));
Ok(peer)
}
}
The complete example can be found under
[pingora-proxy/examples/ctx.rs](../../pingora-proxy/examples/ctx.rs). You can
run it using cargo: RUST_LOG=INFO cargo run --example ctx
# Daemonization
When a Pingora server is configured to run as a daemon, after its bootstrapping,
it will move itself to the background and optionally change to run under the
configured user and group. The pid_file option comes handy in this case for the
user to track the PID of the daemon in the background.
Daemonization also allows the server to perform privileged actions like loading
secrets and then switch to an unprivileged user before accepting any requests
from the network.
This process happens in the run_forever() call. Because daemonization involves
fork(), certain things like threads created before this call are likely lost.
# Error logging
Pingora libraries are built to expect issues like disconnects, timeouts and
invalid inputs from the network. A common way to record these issues are to
output them in error log (STDERR or log files).
## Log level guidelines
Pingora adopts the idea behind [log](https://docs.rs/log/latest/log/). There are
five log levels:
- error: This level should be used when the error stops the request from being
handled correctly. For example when the server we try to connect to is
offline.
- warning: This level should be used when an error occurs but the system
recovers from it. For example when the primary DNS timed out but the system is
able to query the secondary DNS.
- info: Pingora logs when the server is starting up or shutting down.
- debug: Internal details. This log level is not compiled in release builds.
- trace: Fine-grained internal details. This log level is not compiled in
release builds.
The pingora-proxy crate has a well-defined interface to log errors, so that
users don't have to manually log common proxy errors. See its guide for more
details.
# How to return errors
For easy error handling, the pingora-error crate exports a custom Result type
used throughout other Pingora crates.
The Error struct used in this Result's error variant is a wrapper around
arbitrary error types. It allows the user to tag the source of the underlying
error and attach other custom context info.
Users will often need to return errors by propagating an existing error or
creating a wholly new one. pingora-error makes this easy with its error building
functions.
## Examples
For example, one could return an error when an expected header is not present:
rust fn validate_req_header(req: &RequestHeader) -> Result<()> { // validate
that the `host` header exists req.headers() .get(http::header::HOST)
.ok_or_else(|| Error::explain(InvalidHTTPHeader, "No host header detected")) }
impl MyServer { pub async fn handle_request_filter( &self, http_session: &mut
Session, ctx: &mut CTX, ) -> Result<bool> {
validate_req_header(session.req_header()?).or_err(HTTPStatus(400), "Missing
required headers")?; Ok(true) } }
validate_req_header returns an Error if the host header is not found, using
Error::explain to create a new Error along with an associated type
(InvalidHTTPHeader) and helpful context that may be logged in an error log.
This error will eventually propagate to the request filter, where it is returned
as a new HTTPStatus error using or_err. (As part of the default pingora-proxy
fail_to_proxy() phase, not only will this error be logged, but it will result in
sending a 400 Bad Request response downstream.)
Note that the original causing error will be visible in the error logs as well.
or_err wraps the original causing error in a new one with additional context,
but Error's Display implementation also prints the chain of causing errors.
## Guidelines
An error has a _type_ (e.g. ConnectionClosed), a _source_ (e.g. Upstream,
Downstream, Internal), and optionally, a _cause_ (another wrapped error) and a
_context_ (arbitrary user-provided string details).
A minimal error can be created using functions like new_in / new_up / new_down,
each of which specifies a source and asks the user to provide a type.
Generally speaking:
- To create a new error, without a direct cause but with more context, use
Error::explain. You can also use explain_err on a Result to replace the
potential error inside it with a new one.
- To wrap a causing error in a new one with more context, use Error::because.
You can also use or_err on a Result to replace the potential error inside it
by wrapping the original one.
## Retry
Errors can be "retry-able." If the error is retry-able, pingora-proxy will be
allowed to retry the upstream request. Some errors are only retry-able on
[reused connections](pooling.md), e.g. to handle situations where the remote end
has dropped a connection we attempted to reuse.
By default a newly created Error either takes on its direct causing error's
retry status, or, if left unspecified, is considered not retry-able.
# Handling failures and failover
Pingora-proxy allows users to define how to handle failures throughout the life
of a proxied request.
When a failure happens before the response header is sent downstream, users have
a few options:
1. Send an error page downstream and then give up.
2. Retry the same upstream again.
3. Try another upstream if applicable.
Otherwise, once the response header is already sent downstream, there is nothing
the proxy can do other than logging an error and then giving up on the request.
## Retry / Failover
In order to implement retry or failover, fail_to_connect() / error_while_proxy()
needs to mark the error as "retry-able." For failover, fail_to_connect() /
error_while_proxy() also needs to update the CTX to tell upstream_peer() not to
use the same Peer again.
### Safety
In general, idempotent HTTP requests, e.g., GET, are safe to retry. Other
requests, e.g., POST, are not safe to retry if the requests have already been
sent. When fail_to_connect() is called, pingora-proxy guarantees that nothing
was sent upstream. Users are not recommended to retry a non-idempotent request
after error_while_proxy() unless they know the upstream server enough to know
whether it is safe.
### Example
In the following example we set a tries variable on the CTX to track how many
connection attempts we've made. When setting our peer in upstream_peer we check
if tries is less than one and connect to 192.0.2.1. On connect failure we
increment tries in fail_to_connect and set e.set_retry(true) which tells Pingora
this is a retryable error. On retry, we enter upstream_peer again and this time
connect to 1.1.1.1. If we're unable to connect to 1.1.1.1 we return a 502 since
we only set e.set_retry(true) in fail_to_connect when tries is zero.
Rust pub struct MyProxy();
pub struct MyCtx { tries: usize, }
#[async_trait] impl ProxyHttp for MyProxy { type CTX = MyCtx; fn new_ctx(&self)
-> Self::CTX { MyCtx { tries: 0 } }
fn fail_to_connect(
&self,
_session: &mut Session,
_peer: &HttpPeer,
ctx: &mut Self::CTX,
mut e: Box<Error>,
) -> Box<Error> {
if ctx.tries > 0 {
return e;
}
ctx.tries += 1;
e.set_retry(true);
e
}
async fn upstream_peer(
&self,
_session: &mut Session,
ctx: &mut Self::CTX,
) -> Result<Box<HttpPeer>> {
let addr = if ctx.tries < 1 {
("192.0.2.1", 443)
} else {
("1.1.1.1", 443)
};
let mut peer = Box::new(HttpPeer::new(addr, true, "one.one.one.one".to_string()));
peer.options.connection_timeout = Some(Duration::from_millis(100));
Ok(peer)
}
}
# Graceful restart and shutdown
Graceful restart, upgrade, and shutdown mechanisms are very commonly used to
avoid errors or downtime when releasing new versions of Pingora servers.
Pingora graceful upgrade mechanism guarantees the following:
- A request is guaranteed to be handled either by the old server instance or the
new one. No request will see connection refused when trying to connect to the
server endpoints.
- A request that can finish within the grace period is guaranteed not to be
terminated.
## How to graceful upgrade
### Step 0
Configure the upgrade socket. The old and new server need to agree on the same
path to this socket. See configuration manual for details.
### Step 1
Start the new instance with the --upgrade CLI option. The new instance will not
try to listen to the service endpoint right away. It will try to acquire the
listening socket from the old instance instead.
### Step 2
Send SIGQUIT signal to the old instance. The old instance will start to transfer
the listening socket to the new instance.
Once step 2 is successful, the new instance will start to handle new incoming
connections right away. Meanwhile, the old instance will enter its graceful
shutdown mode. It waits a short period of time (to give the new instance time to
initialize and prepare to handle traffic), after which it will not accept any
new connections.
# User Guide
In this guide, we will cover the most used features, operations and settings of
Pingora.
## Running Pingora servers
- [Start and stop](start_stop.md)
- [Graceful restart and graceful shutdown](graceful.md)
- [Configuration](conf.md)
- [Daemonization](daemon.md)
- [Systemd integration](systemd.md)
- [Handling panics](panic.md)
- [Error logging](error_log.md)
- [Prometheus](prom.md)
## Building HTTP proxies
- [Life of a request: pingora-proxy phases and filters](phase.md)
- [Peer: how to connect to upstream](peer.md)
- [Sharing state across phases with CTX](ctx.md)
- [How to return errors](errors.md)
- [Examples: take control of the request](modify_filter.md)
- [Connection pooling and reuse](pooling.md)
- [Handling failures and failover](failover.md)
- [RateLimiter quickstart](rate_limiter.md)
## Advanced topics (WIP)
- [Pingora internals](internals.md)
- Using BoringSSL
- User defined configuration
- Pingora async runtime and threading model
- Background Service
- Blocking code in async context
- Tracing
# Pingora Internals
(Special thanks to [James Munns](https://github.com/jamesmunns) for writing this
section)
## Starting the Server
The pingora system starts by spawning a _server_. The server is responsible for
starting _services_, and listening for termination events.
┌───────────┐ ┌─────────>│ Service │ │ └───────────┘ ┌────────┐ │ ┌───────────┐
│ Server │──Spawns──┼─────────>│ Service │ └────────┘ │ └───────────┘ │
┌───────────┐ └─────────>│ Service │ └───────────┘
After spawning the _services_, the server continues to listen to a termination
event, which it will propagate to the created services.
## Services
_Services_ are entities that handle listening to given sockets, and perform the
core functionality. A _service_ is tied to a particular protocol and set of
options.
> NOTE: there are also "background" services, which just do _stuff_, and aren't
> necessarily listening to a socket. For now we're just talking about listener
> services.
Each service has its own threadpool/tokio runtime, with a number of threads
based on the configured value. Worker threads are not shared cross-service.
Service runtime threadpools may be work-stealing (tokio-default), or
non-work-stealing (N isolated single threaded runtimes).
┌─────────────────────────┐ │ ┌─────────────────────┐ │ │
│┌─────────┬─────────┐│ │ │ ││ Conn │ Conn ││ │ │ │├─────────┼─────────┤│ │ │
││Endpoint │Endpoint ││ │ │ │├─────────┴─────────┤│ │ │ ││ Listeners ││ │ │
│├─────────┬─────────┤│ │ │ ││ Worker │ Worker ││ │ │ ││ Thread │ Thread ││ │ │
│├─────────┴─────────┤│ │ │ ││ Tokio Executor ││ │ │ │└───────────────────┘│ │ │
└─────────────────────┘ │ │ ┌───────┐ │ └─┤Service├───────────────┘ └───────┘
## Service Listeners
At startup, each Service is assigned a set of downstream endpoints that they
listen to. A single service may listen to more than one endpoint. The Server
also passes along any relevant configuration, including TLS settings if
relevant.
These endpoints are converted into listening sockets, called TransportStacks.
Each TransportStack is assigned to an async task within that service's executor.
┌───────────────────┐ │┌─────────────────┐│ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─
─ ─ ─ ─ ─ ─ ─ ┌─────────┐ ││ TransportStack ││ ┌────────────────────┐│
┌┤Listeners├────────┐ ││ ││ │ │ ││ │ │└─────────┘ │ ││ (Listener, TLS
│├──────spawn(run_endpoint())────>│ Service<ServerApp> ││ │┌─────────────────┐│
││ Acceptor, ││ │ │ ││ │ ││ Endpoint ││ ││ UpgradeFDs) ││
└────────────────────┘│ ││ addr/ports ││ │├─────────────────┤│ │ │ │ ││ + TLS
Settings ││ ││ TransportStack ││ ┌────────────────────┐│ │├─────────────────┤│
││ ││ │ │ ││ │ ││ Endpoint ││──build()─> ││ (Listener, TLS
│├──────spawn(run_endpoint())────>│ Service<ServerApp> ││ ││ addr/ports ││ ││
Acceptor, ││ │ │ ││ │ ││ + TLS Settings ││ ││ UpgradeFDs) ││
└────────────────────┘│ │├─────────────────┤│ │├─────────────────┤│ │ │ │ ││
Endpoint ││ ││ TransportStack ││ ┌────────────────────┐│ ││ addr/ports ││ ││ ││
│ │ ││ │ ││ + TLS Settings ││ ││ (Listener, TLS
│├──────spawn(run_endpoint())────>│ Service<ServerApp> ││ │└─────────────────┘│
││ Acceptor, ││ │ │ ││ │ └───────────────────┘ ││ UpgradeFDs) ││
└────────────────────┘│ │└─────────────────┘│ │ ┌───────────────┐ │ │
┌──────────────┐ └───────────────────┘ ─│start_service()│─ ─ ─│ Worker Tasks ├ ─
─ ┘ └───────────────┘ └──────────────┘
## Downstream connection lifecycle
Each service processes incoming connections by spawning a task-per-connection.
These connections are held open as long as there are new events to be handled.
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
─ ─ ┐
│ ┌───────────────┐ ┌────────────────┐ ┌─────────────────┐ ┌─────────────┐ │
┌────────────────────┐ │ UninitStream │ │ Service │ │ App │ │ Task Ends │ │ │ │
│ ::handshake() │──>│::handle_event()│──>│ ::process_new() │──┬>│ │ │ │
Service<ServerApp> │──spawn()──> └───────────────┘ └────────────────┘
└─────────────────┘ │ └─────────────┘ │ │ │ ▲ │ │ └────────────────────┘ │ while
│ └─────────reuse │ ┌───────────────────────────┐ └ ─│ Task on Service Runtime
│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
└───────────────────────────┘
## What is a proxy then?
Interestingly, the pingora Server itself has no particular notion of a Proxy.
Instead, it only thinks in terms of Services, which are expected to contain a
particular implementor of the ServiceApp trait.
For example, this is how an HttpProxy struct, from the pingora-proxy crate,
"becomes" a Service spawned by the Server:
┌─────────────┐ │ HttpProxy │ │ (struct) │ └─────────────┘ │ implements
┌─────────────┐ │ │HttpServerApp│ └───────>│ (trait) │ └─────────────┘ │
implements ┌─────────────┐ │ │ ServerApp │ └───────>│ (trait) │ └─────────────┘
│ contained ┌─────────────────────┐ within │ │ └───────>│ Service<ServiceApp> │
│ │ └─────────────────────┘
Different functionalities and helpers are provided at different layers in this
representation.
┌─────────────┐ ┌──────────────────────────────────────┐ │ HttpProxy │ │Handles
high level Proxying workflow, │ │ (struct) │─ ─ ─ ─ │ customizable via ProxyHttp
trait │ └──────┬──────┘ └──────────────────────────────────────┘ │
┌──────▼──────┐ ┌──────────────────────────────────────┐ │HttpServerApp│ │
Handles selection of H1 vs H2 stream │ │ (trait) │─ ─ ─ ─ │ handling, incl H2
handshake │ └──────┬──────┘ └──────────────────────────────────────┘ │
┌──────▼──────┐ ┌──────────────────────────────────────┐ │ ServerApp │ │ Handles
dispatching of App instances │ │ (trait) │─ ─ ─ ─ │ as individual tasks, per
Session │ └──────┬──────┘ └──────────────────────────────────────┘ │
┌──────▼──────┐ ┌──────────────────────────────────────┐ │ Service<A> │ │
Handles dispatching of App instances │ │ (struct) │─ ─ ─ ─ │ as individual
tasks, per Listener │ └─────────────┘ └──────────────────────────────────────┘
The HttpProxy struct handles the high level workflow of proxying an HTTP
connection
It uses the ProxyHttp (note the flipped wording order!) **trait** to allow
customization at each of the following steps (note: taken from
[the phase chart](./phase_chart.md) doc):
mermaid graph TD; start("new request")-->request_filter;
request_filter-->upstream_peer;
upstream_peer-->Connect{{IO: connect to upstream}};
Connect--connection success-->connected_to_upstream;
Connect--connection failure-->fail_to_connect;
connected_to_upstream-->upstream_request_filter;
upstream_request_filter --> SendReq{{IO: send request to upstream}};
SendReq-->RecvResp{{IO: read response from upstream}};
RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq("request done");
fail_to_connect --can retry-->upstream_peer;
fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;
RecvResp--failure-->IOFailure;
SendReq--failure-->IOFailure;
error_while_proxy--can retry-->upstream_peer;
error_while_proxy--can't retry-->fail_to_proxy;
request_filter --send response-->logging
Error>any response filter error]-->error_while_proxy
IOFailure>IO error]-->error_while_proxy
## Zooming out
Before we zoom in, it's probably good to zoom out and remind ourselves how a
proxy generally works:
┌────────────┐ ┌─────────────┐ ┌────────────┐ │ Downstream │ │ Proxy │ │
Upstream │ │ Client │─────────>│ │────────>│ Server │ └────────────┘
└─────────────┘ └────────────┘
The proxy will be taking connections from the **Downstream** client, and (if
everything goes right), establishing a connection with the appropriate
**Upstream** server. This selected upstream server is referred to as the
**Peer**.
Once the connection is established, the Downstream and Upstream can communicate
bidirectionally.
So far, the discussion of Server, Services, and Listeners have focused on the
LEFT half of this diagram, handling incoming Downstream connections, and getting
it TO the proxy component.
Next, we'll look at the RIGHT half of this diagram, connecting to Upstreams.
## Managing the Upstream
Connections to Upstream Peers are made through Connectors. This is not a
specific type or trait, but more of a "style".
Connectors are responsible for a few things:
- Establishing a connection with a Peer
- Maintaining a connection pool with the Peer, allowing for connection reuse
across:
- Multiple requests from a single downstream client
- Multiple requests from different downstream clients
- Measuring health of connections, for connections like H2, which perform
regular pings
- Handling protocols with multiple poolable layers, like H2
- Caching, if relevant to the protocol and enabled
- Compression, if relevant to the protocol and enabled
Now in context, we can see how each end of the Proxy is handled:
┌────────────┐ ┌─────────────┐ ┌────────────┐ │ Downstream │ ┌ ─│─ Proxy ┌ ┼ ─ │
Upstream │ │ Client │─────────>│ │ │──┼─────>│ Server │ └────────────┘ │
└───────────┼─┘ └────────────┘ ─ ─ ┘ ─ ─ ┘ ▲ ▲ ┌──┘ └──┐ │ │ ┌ ─ ─ ─ ─ ┐ ┌ ─ ─ ─
─ ─ Listeners Connectors│ └ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─
## What about multiple peers?
Connectors only handle the connection to a single peer, so selecting one of
potentially multiple Peers is actually handled one level up, in the
upstream_peer() method of the ProxyHttp trait.
# Examples: taking control of the request
In this section we will go through how to route, modify or reject requests.
## Routing
Any information from the request can be used to make routing decision. Pingora
doesn't impose any constraints on how users could implement their own routing
logic.
In the following example, the proxy sends traffic to 1.0.0.1 only when the
request path start with /family/. All the other requests are routed to 1.1.1.1.
Rust pub struct MyGateway;
#[async_trait] impl ProxyHttp for MyGateway { type CTX = (); fn new_ctx(&self)
-> Self::CTX {}
async fn upstream_peer(
&self,
session: &mut Session,
_ctx: &mut Self::CTX,
) -> Result<Box<HttpPeer>> {
let addr = if session.req_header().uri.path().starts_with("/family/") {
("1.0.0.1", 443)
} else {
("1.1.1.1", 443)
};
info!("connecting to {addr:?}");
let peer = Box::new(HttpPeer::new(addr, true, "one.one.one.one".to_string()));
Ok(peer)
}
}
## Modifying headers
Both request and response headers can be added, removed or modified in their
corresponding phases. In the following example, we add logic to the
response_filter phase to update the Server header and remove the alt-svc header.
Rust #[async_trait] impl ProxyHttp for MyGateway { ... async fn response_filter(
&self, _session: &mut Session, upstream_response: &mut ResponseHeader, _ctx:
&mut Self::CTX, ) -> Result<()> where Self::CTX: Send + Sync, { // replace
existing header if any upstream_response .insert_header("Server", "MyGateway")
.unwrap(); // because we don't support h3
upstream_response.remove_header("alt-svc");
Ok(())
}
}
## Return Error pages
Sometimes instead of proxying the traffic, under certain conditions, such as
authentication failures, you might want the proxy to just return an error page.
Rust fn check_login(req: &pingora_http::RequestHeader) -> bool { // implement
you logic check logic here req.headers.get("Authorization").map(|v|
v.as_bytes()) == Some(b"password") }
#[async_trait] impl ProxyHttp for MyGateway { ... async fn request_filter(&self,
session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool> { if
session.req_header().uri.path().starts_with("/login") &&
!check_login(session.req_header()) { let _ = session.respond_error(403).await;
// true: tell the proxy that the response is already written return Ok(true); }
Ok(false) }
## Logging
Logging logic can be added to the logging phase of Pingora. The logging phase
runs on every request right before Pingora proxy finish processing it. This
phase runs for both successful and failed requests.
In the example below, we add Prometheus metric and access logging to the proxy.
In order for the metrics to be scraped, we also start a Prometheus metric server
on a different port.
Rust pub struct MyGateway { req_metric: prometheus::IntCounter, }
#[async_trait] impl ProxyHttp for MyGateway { ... async fn logging( &self,
session: &mut Session, _e: Option<&pingora::Error>, ctx: &mut Self::CTX, ) { let
response_code = session .response_written() .map_or(0, |resp|
resp.status.as_u16()); // access log info!( "{} response code: {response_code}",
self.request_summary(session, ctx) );
self.req_metric.inc();
}
fn main() { ... let mut prometheus_service_http =
pingora::services::listening::Service::prometheus_http_service();
prometheus_service_http.add_tcp("127.0.0.1:6192");
my_server.add_service(prometheus_service_http);
my_server.run_forever();
}
# Handling panics
Any panic that happens to particular requests does not affect other ongoing
requests or the server's ability to handle other requests. Sockets acquired by
the panicking requests are dropped (closed). The panics will be captured by the
tokio runtime and then ignored.
In order to monitor the panics, Pingora server has built-in Sentry integration.
rust my_server.sentry = Some( sentry::ClientOptions{ dsn:
"SENTRY_DSN".into_dsn().unwrap(), ..Default::default() } );
Even though a panic is not fatal in Pingora, it is still not the preferred way
to handle failures like network timeouts. Panics should be reserved for
unexpected logic errors.
# Peer: how to connect to upstream
In the upstream_peer() phase the user should return a Peer object which defines
how to connect to a certain upstream.
## Peer
A HttpPeer defines which upstream to connect to.
| attribute | meaning |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| address: SocketAddr | The IP:Port to connect to |
| scheme: Scheme | Http or Https |
| sni: String | The SNI to use, Https only |
| proxy: Option<Proxy> | The setting to proxy the request through a [CONNECT proxy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT) |
| client_cert_key: Option<Arc<CertKey>> | The client certificate to use in mTLS connections to upstream |
| options: PeerOptions | See below |
## PeerOptions
A PeerOptions defines how to connect to the upstream.
| attribute | meaning |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| bind_to: Option<InetSocketAddr> | Which local address to bind to as the client IP |
| connection_timeout: Option<Duration> | How long to wait before giving up _establishing_ a TCP connection |
| total_connection_timeout: Option<Duration> | How long to wait before giving up _establishing_ a connection including TLS handshake time |
| read_timeout: Option<Duration> | How long to wait before each individual read() from upstream. The timer is reset after each read() |
| idle_timeout: Option<Duration> | How long to wait before closing a idle connection waiting for connection reuse |
| write_timeout: Option<Duration> | How long to wait before a write() to upstream finishes |
| verify_cert: bool | Whether to check if upstream' server cert is valid and validated |
| verify_hostname: bool | Whether to check if upstream server cert's CN matches the SNI |
| alternative_cn: Option<String> | Accept the cert if the CN matches this name |
| alpn: ALPN | Which HTTP protocol to advertise during ALPN, http1.1 and/or http2 |
| ca: Option<Arc<Box<[X509]>>> | Which Root CA to use to validate the server's cert |
| tcp_keepalive: Option<TcpKeepalive> | TCP keepalive settings to upstream |
## Examples
TBD
# Life of a request: pingora-proxy phases and filters
## Intro
The pingora-proxy HTTP proxy framework supports highly programmable proxy
behaviors. This is done by allowing users to inject custom logic into different
phases (stages) in the life of a request.
## Life of a proxied HTTP request
1. The life of a proxied HTTP request starts when the proxy reads the request
header from the **downstream** (i.e., the client).
2. Then, the proxy connects to the **upstream** (i.e., the remote server). This
step is skipped if there is a previously established
[connection to reuse](pooling.md).
3. The proxy then sends the request header to the upstream.
4. Once the request header is sent, the proxy enters a duplex mode, which
simultaneously proxies: a. upstream response (both header and body) to the
downstream, and b. downstream request body to upstream (if any).
5. Once the entire request/response finishes, the life of the request is ended.
All resources are released. The downstream connections and the upstream
connections are recycled to be reused if applicable.
## Pingora-proxy phases and filters
Pingora-proxy allows users to insert arbitrary logic into the life of a request.
mermaid graph TD; start("new request")-->early_request_filter;
early_request_filter-->request_filter; request_filter-->upstream_peer;
upstream_peer-->Connect{{IO: connect to upstream}};
Connect--connection success-->connected_to_upstream;
Connect--connection failure-->fail_to_connect;
connected_to_upstream-->upstream_request_filter;
upstream_request_filter --> request_body_filter;
request_body_filter --> SendReq{{IO: send request to upstream}};
SendReq-->RecvResp{{IO: read response from upstream}};
RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq("request done");
fail_to_connect --can retry-->upstream_peer;
fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;
RecvResp--failure-->IOFailure;
SendReq--failure-->IOFailure;
error_while_proxy--can retry-->upstream_peer;
error_while_proxy--can't retry-->fail_to_proxy;
request_filter --send response-->logging
Error>any response filter error]-->error_while_proxy
IOFailure>IO error]-->error_while_proxy
### General filter usage guidelines
- Most filters return a [pingora_error::Result<_>](errors.md). When the returned
value is Result::Err, fail_to_proxy() will be called and the request will be
terminated.
- Most filters are async functions, which allows other async operations such as
IO to be performed within the filters.
- A per-request CTX object can be defined to share states across the filters of
the same request. All filters have mutable access to this object.
- Most filters are optional.
- The reason both upstream_response___filter() and response___filter() exist is
for HTTP caching integration reasons (still WIP).
### early_request_filter()
This is the first phase of every request.
This function is similar to request_filter() but executes before any other
logic, including downstream module logic. The main purpose of this function is
to provide finer-grained control of the behavior of the modules.
### request_filter()
This phase is usually for validating request inputs, rate limiting, and
initializing context.
### request_body_filter()
This phase is triggered after a response body is ready to send to upstream. It
will be called every time a piece of request body is received.
### proxy_upstream_filter()
This phase determines if we should continue to the upstream to serve a response.
If we short-circuit, a 502 is returned by default, but a different response can
be implemented.
This phase returns a boolean determining if we should continue to the upstream
or error.
### upstream_peer()
This phase decides which upstream to connect to (e.g. with DNS lookup and
hashing/round-robin), and how to connect to it.
This phase returns a Peer that defines the upstream to connect to. Implementing
this phase is **required**.
### connected_to_upstream()
This phase is executed when upstream is successfully connected.
Usually this phase is for logging purposes. Connection info such as RTT and
upstream TLS ciphers are reported in this phase.
### fail_to_connect()
The counterpart of connected_to_upstream(). This phase is called if an error is
encountered when connecting to upstream.
In this phase users can report the error in Sentry/Prometheus/error log. Users
can also decide if the error is retry-able.
If the error is retry-able, upstream_peer() will be called again, in which case
the user can decide whether to retry the same upstream or failover to a
secondary one.
If the error is not retry-able, the request will end.
### upstream_request_filter()
This phase is to modify requests before sending to upstream.
### upstream_response_filter()/upstream_response_body_filter()/upstream_response_trailer_filter()
This phase is triggered after an upstream response header/body/trailer is
received.
This phase is to modify or process response headers, body, or trailers before
sending to downstream. Note that this phase is called _prior_ to HTTP caching
and therefore any changes made here will affect the response stored in the HTTP
cache.
### response_filter()/response_body_filter()/response_trailer_filter()
This phase is triggered after a response header/body/trailer is ready to send to
downstream.
This phase is to modify them before sending to downstream.
### error_while_proxy()
This phase is triggered during proxy errors to upstream, this is after the
connection is established.
This phase may decide to retry a request if the connection was re-used and the
HTTP method is idempotent.
### fail_to_proxy()
This phase is called whenever an error is encounter during any of the phases
above.
This phase is usually for error logging and error reporting to downstream.
### logging()
This is the last phase that runs after the request is finished (or errors) and
before any of its resources are released. Every request will end up in this
final phase.
This phase is usually for logging and post request cleanup.
### request_summary()
This is not a phase, but a commonly used callback.
Every error that reaches fail_to_proxy() will be automatically logged in the
error log. request_summary() will be called to dump the info regarding the
request when logging the error.
This callback returns a string which allows users to customize what info to dump
in the error log to help track and debug the failures.
### suppress_error_log()
This is also not a phase, but another callback.
fail_to_proxy() errors are automatically logged in the error log, but users may
not be interested in every error. For example, downstream errors are logged if
the client disconnects early, but these errors can become noisy if users are
mainly interested in observing upstream issues. This callback can inspect the
error and returns true or false. If true, the error will not be written to the
log.
### Cache filters
To be documented Pingora proxy phases without caching mermaid graph TD;
start("new request")-->early_request_filter;
early_request_filter-->request_filter; request_filter-->upstream_peer;
upstream_peer-->Connect{{IO: connect to upstream}};
Connect--connection success-->connected_to_upstream;
Connect--connection failure-->fail_to_connect;
connected_to_upstream-->upstream_request_filter;
upstream_request_filter --> request_body_filter;
request_body_filter --> SendReq{{IO: send request to upstream}};
SendReq-->RecvResp{{IO: read response from upstream}};
RecvResp-->upstream_response_filter-->response_filter-->upstream_response_body_filter-->response_body_filter-->logging-->endreq("request done");
fail_to_connect --can retry-->upstream_peer;
fail_to_connect --can't retry-->fail_to_proxy--send error response-->logging;
RecvResp--failure-->IOFailure;
SendReq--failure-->IOFailure;
error_while_proxy--can retry-->upstream_peer;
error_while_proxy--can't retry-->fail_to_proxy;
request_filter --send response-->logging
Error>any response filter error]-->error_while_proxy
IOFailure>IO error]-->error_while_proxy
# Connection pooling and reuse
When the request to a Peer (upstream server) is finished, the connection to that
peer is kept alive and added to a connection pool to be _reused_ by subsequent
requests. This happens automatically without any special configuration.
Requests that reuse previously established connections avoid the latency and
compute cost of setting up a new connection, improving the Pingora server's
overall performance and scalability.
## Same Peer
Only the connections to the exact same Peer can be reused by a request. For
correctness and security reasons, two Peers are the same if and only if all the
following attributes are the same
- IP:port
- scheme
- SNI
- client cert
- verify cert
- verify hostname
- alternative_cn
- proxy settings
## Disable pooling
To disable connection pooling and reuse to a certain Peer, just set the
idle_timeout to 0 seconds to all requests using that Peer.
## Failure
A connection is considered not reusable if errors happen during the request.
# Prometheus
Pingora has a built-in prometheus HTTP metric server for scraping.
rust ... let mut prometheus_service_http = Service::prometheus_http_service();
prometheus_service_http.add_tcp("0.0.0.0:1234");
my_server.add_service(prometheus_service_http); my_server.run_forever();
The simplest way to use it is to have
[static metrics](https://docs.rs/prometheus/latest/prometheus/#static-metrics).
rust static MY_COUNTER: Lazy<IntGauge> = Lazy::new(|| {
register_int_gauge!("my_counter", "my counter").unwrap() });
This static metric will automatically appear in the Prometheus metric endpoint.
# **RateLimiter quickstart**
Pingora provides a crate pingora-limits which provides a simple and easy to use
rate limiter for your application. Below is an example of how you can use
[Rate](https://docs.rs/pingora-limits/latest/pingora_limits/rate/struct.Rate.html)
to create an application that uses multiple limiters to restrict the rate at
which requests can be made on a per-app basis (determined by a request header).
## Steps
1. Add the following dependencies to your Cargo.toml:
toml async-trait="0.1" pingora = { version = "0.3", features = [ "lb" ] }
pingora-limits = "0.3.0" once_cell = "1.19.0"
2. Declare a global rate limiter map to store the rate limiter for each client.
In this example, we use appid.
3. Override the request_filter method in the ProxyHttp trait to implement rate
limiting.
1. Retrieve the client appid from header.
2. Retrieve the current window requests from the rate limiter map. If there
is no rate limiter for the client, create a new one and insert it into the
map.
3. If the current window requests exceed the limit, return 429 and set
RateLimiter associated headers.
4. If the request is not rate limited, return Ok(false) to continue the
request.
## Example
rust use async_trait::async_trait; use once_cell::sync::Lazy; use
pingora::http::ResponseHeader; use pingora::prelude::*; use
pingora_limits::rate::Rate; use std::sync::Arc; use std::time::Duration;
fn main() { let mut server = Server::new(Some(Opt::default())).unwrap();
server.bootstrap(); let mut upstreams =
LoadBalancer::try_from_iter(["1.1.1.1:443", "1.0.0.1:443"]).unwrap(); // Set
health check let hc = TcpHealthCheck::new(); upstreams.set_health_check(hc);
upstreams.health_check_frequency = Some(Duration::from_secs(1)); // Set
background service let background = background_service("health check",
upstreams); let upstreams = background.task(); // Set load balancer let mut lb =
http_proxy_service(&server.configuration, LB(upstreams));
lb.add_tcp("0.0.0.0:6188");
// let rate = Rate
server.add_service(background);
server.add_service(lb);
server.run_forever();
}
pub struct LB(Arc<LoadBalancer<RoundRobin>>);
impl LB { pub fn get_request_appid(&self, session: &mut Session) ->
Option<String> { match session .req_header() .headers .get("appid") .map(|v|
v.to_str()) { None => None, Some(v) => match v { Ok(v) => Some(v.to_string()),
Err(_) => None, }, } } }
// Rate limiter static RATE_LIMITER: Lazy<Rate> = Lazy::new(||
Rate::new(Duration::from_secs(1)));
// max request per second per client static MAX_REQ_PER_SEC: isize = 1;
#[async_trait] impl ProxyHttp for LB { type CTX = ();
fn new_ctx(&self) {}
async fn upstream_peer(
&self,
_session: &mut Session,
_ctx: &mut Self::CTX,
) -> Result<Box<HttpPeer>> {
let upstream = self.0.select(b"", 256).unwrap();
// Set SNI
let peer = Box::new(HttpPeer::new(upstream, true, "one.one.one.one".to_string()));
Ok(peer)
}
async fn upstream_request_filter(
&self,
_session: &mut Session,
upstream_request: &mut RequestHeader,
_ctx: &mut Self::CTX,
) -> Result<()>
where
Self::CTX: Send + Sync,
{
upstream_request
.insert_header("Host", "one.one.one.one")
.unwrap();
Ok(())
}
async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool>
where
Self::CTX: Send + Sync,
{
let appid = match self.get_request_appid(session) {
None => return Ok(false), // no client appid found, skip rate limiting
Some(addr) => addr,
};
// retrieve the current window requests
let curr_window_requests = RATE_LIMITER.observe(&appid, 1);
if curr_window_requests > MAX_REQ_PER_SEC {
// rate limited, return 429
let mut header = ResponseHeader::build(429, None).unwrap();
header
.insert_header("X-Rate-Limit-Limit", MAX_REQ_PER_SEC.to_string())
.unwrap();
header.insert_header("X-Rate-Limit-Remaining", "0").unwrap();
header.insert_header("X-Rate-Limit-Reset", "1").unwrap();
session.set_keepalive(None);
session
.write_response_header(Box::new(header), true)
.await?;
return Ok(true);
}
Ok(false)
}
}
## Testing
To use the example abov me,
1. Run your program with cargo run.
2. Verify the program is working with a few executions of curl localhost:6188 -H
"appid:1" -v
- The first request should work and any later requests that arrive within 1s
of a previous request should fail with:
- Trying 127.0.0.1:6188...
- Connected to localhost (127.0.0.1) port 6188 (#0)
> GET / HTTP/1.1 Host: localhost:6188 User-Agent: curl/7.88.1 Accept: _/_
> appid:1 < HTTP/1.1 429 Too Many Requests < X-Rate-Limit-Limit: 1 <
> X-Rate-Limit-Remaining: 0 < X-Rate-Limit-Reset: 1 < Date: Sun, 14 Jul 2024
> 20:29:02 GMT < Connection: close <
- Closing connection 0
## Complete Example
You can run the pre-made example code in the
[pingora-proxy examples folder](https://github.com/cloudflare/pingora/tree/main/pingora-proxy/examples/rate_limiter.rs)
with
cargo run --example rate_limiter
# Starting and stopping Pingora server
A pingora server is a regular unprivileged multithreaded process.
## Start
By default, the server will run in the foreground.
A Pingora server by default takes the following command-line arguments:
| Argument | Effect | default |
| ------------- | ------------------------------------------------------ | ------------ |
| -d, --daemon | Daemonize the server | false |
| -t, --test | Test the server conf and then exit (WIP) | false |
| -c, --conf | The path to the configuration file | empty string |
| -u, --upgrade | This server should gracefully upgrade a running server | false |
## Stop
A Pingora server will listen to the following signals.
### SIGINT: fast shutdown
Upon receiving SIGINT (ctrl + c), the server will exit immediately with no
delay. All unfinished requests will be interrupted. This behavior is usually
less preferred because it could break requests.
### SIGTERM: graceful shutdown
Upon receiving SIGTERM, the server will notify all its services to shutdown,
wait for some preconfigured time and then exit. This behavior gives requests a
grace period to finish.
### SIGQUIT: graceful upgrade
Similar to SIGTERM, but the server will also transfer all its listening sockets
to a new Pingora server so that there is no downtime during the upgrade. See the
[graceful upgrade](graceful.md) section for more details.
# Systemd integration
A Pingora server doesn't depend on systemd but it can easily be made into a
systemd service.
ini [Service] Type=forking PIDFile=/run/pingora.pid ExecStart=/bin/pingora -d -c
/etc/pingora.conf ExecReload=kill -QUIT $MAINPID ExecReload=/bin/pingora -u -d
-c /etc/pingora.conf
The example systemd setup integrates Pingora's graceful upgrade into systemd. To
upgrade the pingora service, simply install a version of the binary and then
call systemctl reload pingora.service.
For our database we'll be using sled.
For our queue we'll be using sled.
For our caching we'll be using sled.
For our reverse proxy and background daemons we'll be using cloudflare's pingora
server.
For our domain name server we'll be using cloudflare's hickory dns.
