Slap.Files.Sweeper (slap_files v0.4.0)

Copy Markdown View Source

Deletes objects that are no longer a file's body, from the intents that name them (every sweep_interval_ms, default 10 s). Each node sweeps the intent partitions whose Slap.KV shard it owns.

For each intent that is due, the sweeper takes it (a conditional rewrite, so a writer can no longer use it), deletes each object it names that the file's record does not point to, and deletes the intent. An intent is due when an upload has had upload_timeout_ms, or an old body retention_ms, so that readers that already have its key can finish.

An upload that is due may still write its object after the sweeper has deleted it: nothing bounds when a store request completes. After deleting an object, the sweeper deletes its registration, so such an object has none. Every reconcile_interval_ms (default 1 hour), for each bucket whose Slap.KV shard it owns, it lists the bucket's objects and deletes those without a registration.

A writer may give up on a write of a file's record that is applied later. So a record write covered by an intent has a Slap.KV deadline, the intent's due time, and the sweeper acts on the intent only max_clock_skew_ms after that, when no such write can be applied any more, by any node's clock. It reads the record with a linearizable read, which sees every write that was applied.

Summary

Functions

Returns a specification to start this module under a supervisor.

Reconciles now; returns failures by bucket.

Sweeps now; returns failures by bucket.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

reconcile(name \\ Slap.Files)

Reconciles now; returns failures by bucket.

start_link(opts)

@spec start_link(keyword()) :: GenServer.on_start()

sweep(name \\ Slap.Files)

Sweeps now; returns failures by bucket.