Vault sync with ETags

This recipe syncs a folder of notes (a “vault”) between machines that belong to one user, through the document store. Each machine uploads what changed locally, downloads what changed elsewhere, and detects when both changed the same note. It uses documents, ETags and batch writes.

The idea

Every document’s ETag is sha256: plus the hex SHA-256 of its content, so a machine can compute the ETag of a local file without asking the server. One ListDocuments call returns every path and ETag. Comparing three values per note tells you what to do:

• local: the hash of the file on disk now
• remote: the ETag in the listing
• base: the ETag both sides agreed on at the last sync, which you store locally per note

local vs base	remote vs base	Action
same	same	Nothing.
changed	same	Upload with IfMatch: base.
same	changed	Download.
changed	changed	Conflict, unless local equals remote.
new file, no base	not in listing	Upload.
no file, had base	same	Deleted locally: delete remotely.
same	not in listing	Deleted remotely: delete locally.

These are the shapes ricochet-bench measures as sync-cold (first upload, all creates), sync-warm (re-upload after edits, all replaces) and sync-noop (list and compare, nothing written). The no-op pass is the one that runs most often, and it costs one paged listing instead of one request per note. See Benchmarking.

Map file names to paths

Document paths allow only letters, digits, -, _ and /, up to 256 characters. A file name like Garden plans.md has to be encoded. Encoding each path segment with unpadded base32 keeps / as the separator and round-trips exactly:

// Sketch: reversible mapping between vault-relative file paths and document paths.
var b32 = base32.StdEncoding.WithPadding(base32.NoPadding)

func docPath(vault, rel string) string {
  parts := strings.Split(filepath.ToSlash(rel), "/")
  for i, p := range parts {
    parts[i] = b32.EncodeToString([]byte(p))
  }
  return "vault/" + vault + "/" + strings.Join(parts, "/")
}

func etagOf(content []byte) string {
  sum := sha256.Sum256(content)
  return fmt.Sprintf("sha256:%x", sum)
}

Base32 makes names 60% longer, so very deep or long paths can pass the 256-character limit. Keep the vault ID short.

One sync pass

// Sketch: one sync pass. Local is your app's view of the folder and its stored base ETags.
type Local interface {
  Files() map[string][]byte    // document path -> content
  Base(path string) string     // ETag at last sync, "" if never synced
  SetBase(path, etag string)   // record agreement
  Write(path string, b []byte) // replace the local file
  Remove(path string)          // delete the local file and its base
  Conflict(path string, remote []byte)
}

func Sync(ctx context.Context, cl *client.Client, local Local) error {
  owner := cl.PeerID()

  remote := map[string]string{}
  infos, err := cl.ListDocuments(ctx, owner)
  if err != nil {
    return err
  }
  for _, d := range infos {
    remote[d.Path] = d.ETag
  }

  var uploads []client.BatchDocumentPut
  for path, content := range local.Files() {
    l, r, b := etagOf(content), remote[path], local.Base(path)
    switch {
    case l == r:
      local.SetBase(path, l) // already in agreement
    case r == "" && b != "" && l == b:
      local.Remove(path) // deleted on another machine, unchanged here
    case r == "" || r == b:
      // Changed here only (or new). If-Match catches a write that lands meanwhile.
      uploads = append(uploads, client.BatchDocumentPut{
        Path: path, Content: content, ContentType: "text/markdown", IfMatch: b,
      })
    case l == b:
      if err := download(ctx, cl, local, path); err != nil {
        return err
      }
    default:
      doc, err := cl.GetDocument(ctx, owner, path)
      if err != nil {
        return err
      }
      local.Conflict(path, doc.Content)
    }
  }

  for _, chunk := range client.BatchDocuments(uploads) {
    results, err := cl.PutDocuments(ctx, owner, chunk)
    if err != nil {
      return err // whole request failed; nothing in this chunk is recorded
    }
    for _, r := range results {
      switch {
      case r.OK():
        local.SetBase(r.Path, r.ETag)
      case r.Status == wire.StatusConflict:
        // Another machine wrote since the listing. r.ActualETag is the new
        // remote; fetch it and treat the note as changed on both sides.
        doc, err := cl.GetDocument(ctx, owner, r.Path)
        if err != nil {
          return err
        }
        local.Conflict(r.Path, doc.Content)
      default:
        return fmt.Errorf("%s: status %d: %s", r.Path, r.Status, r.Error)
      }
    }
  }
  // Remote-only paths are new notes from another machine: download them.
  // Deletions follow the table above.
  return nil
}

func download(ctx context.Context, cl *client.Client, local Local, path string) error {
  doc, err := cl.GetDocument(ctx, cl.PeerID(), path)
  if err != nil {
    return err
  }
  if !doc.IsSuccess() {
    return fmt.Errorf("%s: status %d", path, doc.Status)
  }
  local.Write(path, doc.Content)
  local.SetBase(path, doc.ETag)
  return nil
}

Notes on the details

• Batches. client.BatchDocuments splits uploads by count (100) and by about 2 MiB of content. A 500-note vault of small notes is a handful of requests instead of 500, which keeps a sync clear of the server’s per-request rate limit.
• New notes on two machines. There is no create-only precondition: an If-Match against a path that does not exist yet succeeds. Two machines creating the same new path in the same window both succeed, and the second write wins. The next sync on the first machine sees remote != base and local != base and reports a conflict, so nothing is lost silently.
• Deletes are not conditional. DeleteDocument ignores ETags. Re-list just before deleting and skip any note whose remote ETag moved off its base.
• Rate limits. On client.ErrRateLimited, sleep for client.RetryAfter(err) and continue; it is exact. Do not slow the whole sync down permanently.
• Privacy. Documents are private by default and stored as sent. If the server must not read the notes, encrypt the content before upload. The ETags are then hashes of ciphertext, so keep each note’s plaintext hash locally next to its base ETag and use it to detect local changes.
• Notifying other machines. To make other machines sync promptly instead of on a timer, send them a short message after a sync that uploaded something, and have them sync on receipt. Batch those with SendMessages.