SyncStore Protocol

SyncStore is an open protocol specification for client-first, offline-capable datastores. It provides a standardized way to synchronize collected key-value pairs holding JSON data between clients and servers, with optional graph-like relations between records.

Why SyncStore?

How It Works

Concepts

Collections

A collection is a named container for key-value pairs. Keys are strings and are automatically sorted in lexicographical order. Each key maps to a JSON value. Collections provide the primary way to organize and query data in SyncStore.

Indexes

Indexes allow you to filter and re-order a subset of items in a collection using arbitrary string values as ordering keys. An item can belong to multiple indexes, each with its own ordering key. When querying with an index, only items in that index are returned, sorted lexicographically by their ordering key.

Relations

Relations create graph-like connections between records across collections using a triple-based model: subject → predicate → object. The subject is the item the relation belongs to, the predicate (type) describes the relationship, and the object references another item in any collection.

Tags

Tags provide row-level access control by attaching string labels to key-value entries. Each authenticated client has a set of allowed tags, and can only read or sync entries where at least one tag matches. Tags are assigned during the approval/commit process, enabling fine-grained data partitioning—from simple user isolation to complex multi-tier architectures like CQRS event sourcing across services.

Operations

Data Operations

Endpoints

Sync

[
  // Data operations
  { "ref": "uuid-1", "type": "set", "collection": "tasks", "key": "task_1", "data": { ... }, "tags": ["user:123"], "timestamp": "..." },
  { "ref": "uuid-2", "type": "patch", "collection": "tasks", "key": "task_1", "data": [{ "op": "replace", ... }], "timestamp": "..." },
  { "ref": "uuid-3", "type": "delete", "collection": "tasks", "key": "task_1", "timestamp": "..." },

  // Index operations
  { "ref": "uuid-4", "type": "index_set", "collection": "tasks", "key": "task_1", "index": "by_due", "value": "2024-06-15", "timestamp": "..." },
  { "ref": "uuid-5", "type": "index_delete", "collection": "tasks", "key": "task_1", "index": "by_due", "timestamp": "..." },

  // Relation operations
  { "ref": "uuid-6", "type": "relation_set", "collection": "tasks", "key": "task_1", "relation": "rel_1", "relationType": "assigned_to", "object": { "collection": "users", "key": "user_1" }, "timestamp": "..." },
  { "ref": "uuid-7", "type": "relation_delete", "collection": "tasks", "key": "task_1", "relation": "rel_1", "timestamp": "..." },

  // Tag operations
  { "ref": "uuid-8", "type": "tag_add", "collection": "tasks", "key": "task_1", "tag": "project:abc", "timestamp": "..." },
  { "ref": "uuid-9", "type": "tag_remove", "collection": "tasks", "key": "task_1", "tag": "project:abc", "timestamp": "..." }
]

Data

{
  "name": "My Task",
  "completed": false,
  "dueDate": "2024-06-15"
}

[
  { "op": "replace", "path": "/completed", "value": true },
  { "op": "add", "path": "/completedAt", "value": "2024-06-10T14:30:00Z" }
]

Indexes

Indexes allow you to assign an arbitrary ordering key to items in a collection. When querying with an index, results are sorted lexicographically by this key, enabling filtering and custom ordering. Only items with an index entry are returned.

{
  "value": "2024-06-10T14:30:00Z"
}

Relations

{
  "type": "belongs_to",
  "object": {
    "collection": "users",
    "key": "user_123"
  }
}

Tags

Batch

[
  {
    "method": "PUT",
    "path": "/data/tasks/task_1",
    "body": { "name": "Task 1" }
  },
  {
    "method": "PUT",
    "path": "/data/tasks/task_2",
    "body": { "name": "Task 2" }
  },
  {
    "method": "DELETE",
    "path": "/data/tasks/task_3"
  }
]

Supported Formats

SyncStore is transport-format agnostic. Any format that can be encoded to and decoded from JSON without loss is valid. We recommend implementors support JSON and MessagePack at minimum.

Clients specify their preferred format via the Content-Type and Accept headers.

Recommended Limits

These limits are recommendations to keep the protocol lightweight and performant. Implementors may choose to adjust them based on their use case.

Example Architectures

SyncStore's protocol enables a variety of architectural patterns, from simple client-server setups to complex multi-tier systems with CQRS event sourcing. Tags provide row-level access control at each tier.

%%{init: {'theme': 'base', 'themeVariables': { 'edgeLabelBackground':'#292524' }, 'flowchart': { 'htmlLabels': true }}}%%
flowchart LR
    C1[["🖥️ Client
───
Local DB"]]
    SS1[["📦 SyncStore
───
Full DB"]]
    C1 -.->|"Reads / Writes"| SS1
    C1 -.->|"Syncs"| SS1

%% Styling
style C1 fill:#162456,stroke:#2b7fff,color:#eff6ff
style SS1 fill:#032e15,stroke:#00c951,color:#f0fdf4
linkStyle default stroke:#a6a09b
                            

%%{init: {'theme': 'base', 'themeVariables': { 'edgeLabelBackground':'#292524' }, 'flowchart': { 'htmlLabels': true }}}%%
flowchart LR
    C2[["🖥️ Client
───
Local DB"]]
    P2[["🔀 Proxy
───
Local Cache"]]
    SS2[["📦 SyncStore
───
Full DB"]]
    C2 -.->|"Reads / Writes"| P2
    P2 -.->|"Reads / Writes"| SS2
    P2 -.->|"Syncs"| C2

%% Styling
style C2 fill:#162456,stroke:#2b7fff,color:#eff6ff
style SS2 fill:#032e15,stroke:#00c951,color:#f0fdf4
style P2 fill:#1e1a4d,stroke:#615fff,color:#eef2ff
linkStyle default stroke:#a6a09b
                            

%%{init: {'theme': 'base', 'themeVariables': { 'edgeLabelBackground':'#292524' }, 'flowchart': { 'htmlLabels': true }}}%%
flowchart LR
    C3[["🖥️ Client
───
Local DB"]]
    G3[["🚪 Gateway
───
Local Cache"]]

    subgraph Services[" "]
        M3a[["⚙️ Microservice
───
Local DB"]]
        M3b[["⚙️ Microservice
───
Local DB"]]
        M3a ~~~ M3b
    end

    P3[["🔀 Proxy
───
Local Cache"]]
    SS3[["📦 SyncStore
───
Full DB"]]

    C3 -.->|"Reads / Writes"| G3
    G3 -.->|"Reads / Writes"| M3a
    G3 -.->|"Reads / Writes"| M3b
    M3a -.->|"Reads / Writes"| P3
    M3b -.->|"Reads / Writes"| P3
    P3 -.->|"Reads / Writes"| SS3

    SS3 -.->|"Syncs"| P3
    P3 -.->|"Syncs"| M3a
    P3 -.->|"Syncs"| M3b

%% Styling
style C3 fill:#162456,stroke:#2b7fff,color:#eff6ff
style SS3 fill:#032e15,stroke:#00c951,color:#f0fdf4
style P3 fill:#1e1a4d,stroke:#615fff,color:#eef2ff
style G3 fill:#1e1a4d,stroke:#615fff,color:#eef2ff
style M3a fill:#461901,stroke:#fd9a00,color:#fffbeb
style M3b fill:#461901,stroke:#fd9a00,color:#fffbeb
style Services fill:transparent,stroke:transparent
linkStyle default stroke:#a6a09b