Files stored with Slap.KV and an object store. A file lives at a ref,
{partition, id}: a partition is a stable group of files (a document's,
for example), listed together, and on one Slap.KV shard. Each file has
one record in Slap.KV, which exists exactly while the file does, and a
body stored in the record (inline) or as an object.
Files can be replaced; bodies never are. Each put/3 writes a new body,
switches the file's record to it with a conditional write, and gives the
file a new version. The old body is deleted after retention_ms, so a
reader that already has it can finish.
Start it with the object store for bodies, after Slap.KV.Cluster:
{Slap.Files, store: {:url, "s3://bucket/prefix"}, path: "files"}Options:
:store- required, as forSlap.SlateDB.open/2;:path- where the objects go in it (default"files").:inline_max_bytes- bodies up to this size are stored inline withstorage: :auto(default 16 KiB).:inline_limit- the largest inline body, even withstorage: :inline(default 1 MiB).:retention_ms- how long an old body is kept (default 5 minutes).:upload_timeout_ms- how long an upload may take before its object may be deleted (default 1 hour).:sweep_interval_ms- seeSlap.Files.Sweeper(default 10 s).:max_clock_skew_ms- how far apart the nodes' clocks may be (default 30 s): seeSlap.Files.Sweeper.:reconcile_interval_ms- seeSlap.Files.Sweeper(default 1 hour).:name- instance name (defaultSlap.Files). Pass it asfiles:to file operations.:cluster- the instance's KV cluster (defaultSlap.KV.Cluster); define an application-owned one withuse Slap.KV.Cluster, otp_app: :my_app.:namespace- stable data namespace within the KV cluster and object store (default"default"). Named instances require an explicit namespace so renaming an instance does not change its stored layout. Changing the namespace leaves existing files at the old layout. Instances with the same namespace share records, intents, and object registrations.:timeout- timeout for each KV or object-store call (default no timeout for object-store calls); operations can override it.
:clock is an internal test hook.
Errors
Invalid file data (ref, body, metadata, content type, expected SHA-256,
:if_version and :range) returns an error tuple. Invalid control options (:storage,
:files, :timeout) and unknown option names raise ArgumentError.
{:error, {:conflict, version}}- anif_version:condition does not hold;versionis the file's current one, or nil.{:error, :expired}- an upload took longer thanupload_timeout_ms; its object is deleted.{:error, {:range_not_satisfiable, size}}- a read's:rangecontains no byte of the file;sizeis the file's size.{:error, :checksum_mismatch},{:error, :too_large_for_inline},{:error, {:bad_request, reason}}.{:error, :unavailable | :timeout}- fromSlap.KVor the store. A write may still have been applied;get/1shows whether it was.
Telemetry
put/3, delete/2, get/2, read/2, stream/2, and list/2 emit
Telemetry spans under [:slap, :files, operation], with :start, :stop,
and :exception events. Metadata includes :files, the instance name;
a stop event also has :outcome (:ok or :error). A read or stream
with a :range has it in its metadata. The stream/2 span
measures opening the body stream; consuming it happens after that span ends.
Summary
Types
The bytes a read returns: {first, last} (inclusive, as in HTTP),
{first, :eof}, or {:last, n} (the last n bytes).
Functions
Returns a specification to start this module under a supervisor.
Deletes a file. Returns :ok once it is gone (also if there was none,
without :if_version); its body is deleted after retention_ms.
if_version: makes it conditional, as for put/3.
The file's metadata, or {:ok, nil}.
Lists a partition's files in id order, including their versions. Takes the
:prefix, :gte, :lt, :limit and
:cursor of Slap.KV.scan/2, and returns {:ok, %{files: files, cursor: cursor}}. An input cursor: nil starts at the beginning of the selected
range; a returned cursor: nil means there are no more pages. A binary
cursor remains valid after the file it follows is deleted. Supplied
:prefix, :gte, and :lt values must be binaries.
A new random id, for a ref.
Writes a file: creates it, or replaces its body and metadata. body is a
binary or an enumerable of binaries (read once). Returns the file's new
Slap.Files.File once it is durable.
The file's body, or {:ok, nil}. See stream/2 for large bodies, and
for the :range and :if_version options.
The file and its body as a lazy stream of binaries, or {:ok, nil}. An
object body stays readable for retention_ms after the file is replaced
or deleted. A store failure while reading the returned stream raises
Slap.SlateDB.Error; failure to open the download returns
{:error, :unavailable | :timeout}.
Types
@type error() :: Slap.KV.error() | :checksum_mismatch | :expired | :too_large_for_inline | {:range_not_satisfiable, non_neg_integer()}
@type range() :: {non_neg_integer(), non_neg_integer() | :eof} | {:last, non_neg_integer()}
The bytes a read returns: {first, last} (inclusive, as in HTTP),
{first, :eof}, or {:last, n} (the last n bytes).
Functions
@spec child_spec(keyword()) :: Supervisor.child_spec()
Returns a specification to start this module under a supervisor.
See Supervisor.
Deletes a file. Returns :ok once it is gone (also if there was none,
without :if_version); its body is deleted after retention_ms.
if_version: makes it conditional, as for put/3.
@spec get(ref(), keyword()) :: {:ok, Slap.Files.File.t() | nil} | {:error, error()}
The file's metadata, or {:ok, nil}.
@spec list(binary(), keyword()) :: {:ok, %{files: [Slap.Files.File.t()], cursor: binary() | nil}} | {:error, error()}
Lists a partition's files in id order, including their versions. Takes the
:prefix, :gte, :lt, :limit and
:cursor of Slap.KV.scan/2, and returns {:ok, %{files: files, cursor: cursor}}. An input cursor: nil starts at the beginning of the selected
range; a returned cursor: nil means there are no more pages. A binary
cursor remains valid after the file it follows is deleted. Supplied
:prefix, :gte, and :lt values must be binaries.
@spec new_id() :: binary()
A new random id, for a ref.
@spec put(ref(), binary() | Enumerable.t(), keyword()) :: {:ok, Slap.Files.File.t()} | {:error, error()}
Writes a file: creates it, or replaces its body and metadata. body is a
binary or an enumerable of binaries (read once). Returns the file's new
Slap.Files.File once it is durable.
Options:
:if_version-:absentto create only, or the version the file must be at.:storage-:auto(default),:inlineor:object.:content_type- default"application/octet-stream".:metadata- a map of strings to strings (default%{}).:expected_sha256- the body's expected SHA-256 (32 bytes).:files- the instance name (defaultSlap.Files).:timeout- overrides the instance's timeout for each KV or object-store call.
Writing the body, content type, metadata and storage the file already has
without a condition returns the file as it is. With :if_version, the
condition must hold even when the content is unchanged.
The file's body, or {:ok, nil}. See stream/2 for large bodies, and
for the :range and :if_version options.
@spec start_link(keyword()) :: Supervisor.on_start()
@spec stream(ref(), keyword()) :: {:ok, {Slap.Files.File.t(), Enumerable.t()} | {Slap.Files.File.t(), {non_neg_integer(), non_neg_integer()}, Enumerable.t()} | nil} | {:error, error()}
The file and its body as a lazy stream of binaries, or {:ok, nil}. An
object body stays readable for retention_ms after the file is replaced
or deleted. A store failure while reading the returned stream raises
Slap.SlateDB.Error; failure to open the download returns
{:error, :unavailable | :timeout}.
Options:
:range- read only part of the body (seerange/0). Returns{:ok, {file, {first, last}, chunks}}, where{first, last}is the range clamped to the file's size, as HTTP clamps it: alastpast the end becomes the last byte. A range that starts past the end,{:last, 0}, and every range of an empty file return{:error, {:range_not_satisfiable, size}}. For an object body, only the requested bytes are downloaded.:if_version- the version the file must have; otherwise the read returns{:error, {:conflict, version}}. To read a file in several ranges from the same body, pass the version from the first read to the rest.:files,:timeout- as forput/3.