View Source Data modelling with Dynamo

Composite keys

Key fields are joined in declaration order with key_separator (default #). The partition key is prefixed with the entity name so several entity types can share a table.

defmodule Shop.Order do
  use Dynamo.Schema

  item do
    table "shop"
    entity_name "order"

    field :tenant, :string
    field :customer_id, :string
    field :placed_at, :utc_datetime
    field :order_id, :string
    field :total, :decimal

    partition_key [:tenant, :customer_id]
    sort_key [:placed_at, :order_id]
  end
end

{:ok, key} = Dynamo.Schema.primary_key(%Shop.Order{tenant: "acme", customer_id: "c1", placed_at: ~U[2024-01-15 10:00:00Z], order_id: "o1"})
key.pk  #=> %{"S" => "order#acme#c1"}
key.sk  #=> %{"S" => "2024-01-15T10:00:00Z#o1"}

Rules:

All partition key fields must be non-nil. {:error, %Dynamo.Error{type: :validation_error}} otherwise.
Sort key fields may be partially populated (prefix). %Shop.Order{tenant: "acme", customer_id: "c1", placed_at: dt} gives sk = "2024-01-15T10:00:00Z", which query/2 turns into begins_with.
A value containing the separator is rejected. Keep # unless your data contains it.
prefix_sort_key: true renders "placed_at#2024-…#order_id#o1" if you want self-describing keys.

Sort key strategies

Time ordering – put an ISO 8601 datetime first; descending: true gives newest first.

Repo.query(%Shop.Order{tenant: "acme", customer_id: "c1"}, sort_key: {:gte, "2024-01-01"}, descending: true)

Type prefix in a single partition – sort_key [:kind, :id] with kind as "PROFILE", "ORDER", "ADDRESS":

Repo.query(%Account.Item{account_id: "a1", kind: "ORDER"})           # begins_with "ORDER"
Repo.query(%Account.Item{account_id: "a1"}, sort_key: {:eq, "PROFILE"})

Range with BETWEEN – composite string keys compare lexicographically, so an upper bound like "2024-01-31" excludes "2024-01-31T10:00:00Z#o1". Append ~ (0x7E, above every character that can follow) to make the bound inclusive:

Repo.query(%Shop.Order{...}, sort_key: {:between, "2024-01-01", "2024-01-31~"})

Global secondary indexes

item do
  table "shop"
  field :order_id, :string, partition_key: true
  field :status, :string
  field :placed_at, :utc_datetime
  field :total, :decimal

  global_secondary_index "StatusIndex", partition_key: :status, sort_key: :placed_at
  global_secondary_index "TotalIndex", partition_key: :status, sort_key: :total, projection: :keys_only
end

GSI key attributes are the raw field values with their own types – no entity prefix, no separator. Numbers stay N, so range conditions work numerically:

Repo.query(%Shop.Order{status: "paid"}, index: "TotalIndex", sort_key: {:gt, Decimal.new("100")})

mix dynamo.create_table --schema Shop.Order creates the indexes with the right attribute types. Consistent reads are rejected on GSIs (DynamoDB does not support them).

Single-table design

Several schemas can point at the same table; the entity prefix keeps them apart:

defmodule App.User do
  use Dynamo.Schema
  item do
    table "app"
    field :id, :string, partition_key: true
    field :email, :string
  end
end

defmodule App.Session do
  use Dynamo.Schema
  item do
    table "app"
    field :user_id, :string, partition_key: true      # pk = "session#<user_id>"
    field :token, :string, sort_key: true
    field :expires_at, :utc_datetime
  end
end

To store child items under the parent's partition (adjacency list), point the child's entity_name at the parent module and put the type in the sort key:

defmodule App.Order do
  use Dynamo.Schema
  item do
    table "app"
    entity_name App.User                                # pk = "user#<id>" – inherited from App.User
    field :id, :string, partition_key: true             # the parent's id
    field :kind, :string, sort_key: true, default: "order"
    field :order_id, :string, sort_key: true
    field :total, :decimal
  end
end

Repo.query(%App.Order{id: "u1"})                       # all orders of u1: begins_with(sk, "order")
Repo.get(%App.User{id: "u1"})                          # the parent, with its own sort key

entity_name App.User is resolved at compile time, so renaming the parent moves both entities together and a typo is a compile error. The string form (entity_name "user") does the same thing without the compile-time link. Give the parent a fixed sort-key value too (e.g. field :kind, default: "profile") so parent and children never collide in the sort-key space. If two kinds share a prefix ("order", "order_note"), query with an explicit sort_key: {:begins_with, "order#"}.

Existing tables

For a table you did not design around this layout:

defmodule Legacy.Customer do
  use Dynamo.Schema, raw_keys: true

  item do
    table "Customers"
    field :CustomerId, :string, partition_key: true    # attribute names are used as-is
    field :Version, :integer, sort_key: true
    field :Name, :string
  end
end

mix dynamo.generate_schema Customers writes this for you after sampling the table.

Embedded schemas

Nested structures keep their types when you declare them as embedded schemas:

defmodule Shop.Address do
  use Dynamo.Schema, embedded: true       # no table, keys or indexes

  item do
    field :street, :string
    field :city, :string
    field :geo, {:list, :float}
  end
end

defmodule Shop.Customer do
  use Dynamo.Schema
  item do
    table "shop"
    field :id, :string, partition_key: true
    field :billing, Shop.Address
    field :shipping, {:list, Shop.Address}, default: []
    field :prices, {:list, :decimal}
  end
end

A :map field would hand you string-keyed maps with floats and strings back; an embedded schema gives you %Shop.Address{} with Decimal/Date values intact. Any table schema can be embedded as well – only its declared fields are stored, never the generated pk/sk.

Conditions without placeholders

Every condition:/filter: accepts a small DSL instead of a raw expression:

Repo.put(order, condition: [pk: :not_exists])
Repo.update(acct, [balance: {:decrement, 50}], condition: [balance: {:gte, 50}])
Repo.query(%Order{user_id: "u"}, filter: {:and, [status: {:in, ["open", "paid"]}], {:not, [flagged: true]}})
Repo.scan(Customer, filter: [{[:billing, :city], "Berlin"}, tags: {:contains, "vip"}])

Values are dumped with the field's type; placeholders (#c0, :c0) are generated so reserved words and odd attribute names are never a problem.

Nil, defaults and removal

Nil fields are omitted from written items. Reading an item that lacks an attribute yields the struct default. To delete an attribute use Repo.update(item, field: :remove); Repo.update(item, field: nil) stores an explicit NULL, which loads back as nil too but is visible to attribute_exists.

Vector search

Store embeddings next to the data they describe and search them without a second database:

defmodule Kb.Chunk do
  use Dynamo.Schema

  item do
    table "kb"
    field :doc_id, :string, partition_key: true
    field :chunk_id, :string, sort_key: true
    field :tenant, :string
    field :lang, :string
    field :text, :string
    field :embedding, {:vector, 1024}

    vector_index "Semantic",
      field: :embedding,
      distance: :cosine,
      projection: :include,
      projected_attributes: [:text, :lang],
      partition_key: :tenant,
      filters: [:lang]
  end
end

embedding = MyApp.Embeddings.embed("how do I rotate credentials?")   # your model, e.g. Bedrock Titan

{:ok, %{results: hits}} =
  Repo.search_vectors(Kb.Chunk, "Semantic", embedding, top_k: 8, filter: [tenant: "acme", lang: "en"])

Decisions:

Dimensions must equal your embedding model's output; {:vector, n} checks it on write. Fewer dimensions cost less to store and search.
Distance: :cosine for raw embeddings, :dot_product if you normalise to unit vectors (same ranking, cheaper), :euclidean when magnitude matters.
Partition key on the index (partition_key: :tenant) scopes each search to one value and raises throughput quotas per value. It becomes mandatory in filter:. Inline filters: narrow further with equality/range conditions.
Projection decides what a search returns and how much a write costs. The vector itself is never returned unless you list it in projection:.
Vector indexes require on-demand billing, are eventually consistent, and are immutable once created (add a new index instead).
The item is a normal item: get, query, update all work on it. Updating the vector attribute re-indexes it.

Custom types

defmodule App.Type.Money do
  use Dynamo.Type

  def dump(%Money{amount: a, currency: c}), do: {:ok, %{"S" => "#{c}:#{a}"}}
  def dump(_), do: :error

  def load(%{"S" => s}) do
    [c, a] = String.split(s, ":")
    {:ok, %Money{amount: Decimal.new(a), currency: c}}
  end

  def load(_), do: :error
end

field :price, App.Type.Money

For values nested inside :map/:list/:any fields, implement Dynamo.Encodable instead (one-way: they load back as plain maps).

Checklist

[ ] List access patterns before choosing keys
[ ] Partition key has high cardinality (no dates, no statuses)
[ ] Sort key supports the range queries you listed
[ ] GSIs for remaining patterns; :keys_only/:include where possible
[ ] Items < 400 KB – reference S3 for blobs
[ ] TTL attribute (:integer epoch seconds) for ephemeral data
[ ] entity_name set explicitly (string) if the module might be renamed and data already exists

← Previous Page Getting started

Next Page → Streaming, batches and transactions