llink/go/docs/particle-system-plan.md

Particle System & Network Capacity Implementation Plan

Summary

Implement a unified particle system where all content types (streams, folders, media, files, text, quests, papers, AI chats) are particles with a common structure but type-specific data. Add network capacity management for billing/limiting open streams.

---

Key Design Decisions

Decision	Choice
Data model	Unified particle with type field + JSONB data
Hierarchy	Arbitrary nesting (parent_id references another particle)
Access control	Split: Handler checks network membership, Particle service checks particle visibility
Stream membership	Auto-visible to all network members OR custom member list
Open streams	Only "open" streams count against capacity
Substream counting	All stream particles count (including nested)
Service coupling	Loose - no FK constraints, particle service is independent
Listing API	Unified ListParticles(networkID, parentID, ...) - file explorer style
Sorting	Streams: updated_at DESC (activity), Folders: created_at (client sorts by type)
Pagination	Bidirectional cursor for streams (chat-like); folders return all
Network capacity	Stored on networks table (open_stream_capacity), updated via admin API
Data validation	Start simple with required field validation in service code

---

Architecture: Access Control Split

┌─────────────────────────────────────────────────────────────────────┐
│                           HANDLER LAYER                              │
│  1. Extract user email from auth context                            │
│  2. Check network membership (via network service)                  │
│  3. Call particle service with (networkID, email)                   │
│  4. Transform Particle structs → Response DTOs                      │
└─────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        PARTICLE SERVICE                              │
│  - Trusts that handler verified network membership                  │
│  - Checks particle-level visibility (network_all vs custom)         │
│  - Filters results to only particles user can see                   │
│  - Returns Particle domain structs                                  │
│  - NO dependency on network service                                 │
└─────────────────────────────────────────────────────────────────────┘

Why this split?

• Particle service stays decoupled from network service
• Network membership is a cross-cutting concern (handler already knows user context)
• Particle visibility is domain-specific (belongs in particle service)

---

UI Data Flow Examples

Mental Model: File Explorer

The API follows a file explorer pattern:

• parentID = nil → root items of network
• parentID = "p_123" → children of that particle
• Same method works at every level
• Response includes parent for breadcrumbs/context

Example 1: User Opens App → Network Sidebar

┌─────────────────────────────────────────────────────┐
│  Streams in Acme Corp                               │
│  ├─ 📂 Projects (folder)                           │
│  │   ├─ 💬 Website Redesign (stream, open)         │
│  │   └─ 💬 Mobile App (stream, closed)             │
│  ├─ 💬 General Chat (stream, open)                 │
│  └─ 💬 Support Tickets (stream, open)              │
└─────────────────────────────────────────────────────┘

API Calls:

// 1. Get root particles for selected network
resp := particleService.ListParticles(ctx, "net_abc", nil, email, ListFilter{}, nil)
// Returns:
// {
//   Parent: nil,  // No parent at root
//   Particles: [
//     {ID: "p_1", Type: "folder", Data: {"name": "Projects"}, ...},
//     {ID: "p_2", Type: "stream", Data: {"title": "General Chat"}, StreamStatus: "open"},
//     {ID: "p_3", Type: "stream", Data: {"title": "Support Tickets"}, StreamStatus: "open"},
//   ]
// }

// 2. User clicks "Projects" folder → fetch children
resp := particleService.ListParticles(ctx, "net_abc", ptr("p_1"), email, ListFilter{}, nil)
// Returns:
// {
//   Parent: {ID: "p_1", Type: "folder", Data: {"name": "Projects"}},  // For breadcrumbs
//   Particles: [
//     {ID: "p_4", Type: "stream", Data: {"title": "Website Redesign"}, StreamStatus: "open"},
//     {ID: "p_5", Type: "stream", Data: {"title": "Mobile App"}, StreamStatus: "closed"},
//   ]
// }

Example 2: User Opens Stream → Chat View

┌─────────────────────────────────────────────────────┐
│  💬 Website Redesign                     [Members]  │
├─────────────────────────────────────────────────────┤
│  ↑ Load older                                      │
│  ─────────────────────────────────────────────────  │
│  [Alice] Here's the new mockup                     │
│  📎 mockup-v2.png (media particle)                 │
│  ─────────────────────────────────────────────────  │
│  [Bob] Looks great! Question about nav             │
│  ─────────────────────────────────────────────────  │
│  📋 Update navigation colors (quest)               │
│  ─────────────────────────────────────────────────  │
│  [Type a message...]                    [Send]      │
└─────────────────────────────────────────────────────┘

API Calls:

// 1. Initial load - most recent particles (sorted by updated_at DESC)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{}, nil)
// Returns:
// {
//   Parent: {ID: "stream_123", Type: "stream", Data: {"title": "Website Redesign"}},
//   Particles: [newest...oldest],  // Sorted by updated_at DESC
//   HasMore: true,
//   PrevCursor: {Position: "p_oldest_in_batch", Direction: "before"},
//   NextCursor: nil  // At newest
// }

// 2. User scrolls UP → load older messages
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
    &Cursor{Position: "p_oldest_in_batch", Direction: "before"})

// 3. User scrolls DOWN → load newer (after scrolling up)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
    &Cursor{Position: "some_id", Direction: "after"})

// 4. User sends message
newParticle := particleService.Create(ctx, CreateInput{
    Type: TypeText,
    NetworkID: networkID,
    ParentID: ptr("stream_123"),
    Data: json.RawMessage(`{"content": "My message"}`),
}, email)
// UI inserts at bottom

Handler Implementation

// Handler pseudocode
func (h *Handler) ListParticles(w http.ResponseWriter, r *http.Request) {
    email := getAuthEmail(r.Context())
    networkID := r.URL.Query().Get("network_id")
    parentID := r.URL.Query().Get("parent_id")  // Optional

    // 1. Check network membership (handler responsibility)
    network, err := h.networkService.GetByID(ctx, networkID)
    if err != nil { return NotFound }

    if !network.HasMember(email) && network.AdminEmail != email {
        return Forbidden("not a network member")
    }

    // 2. Parse cursor if provided
    var cursor *particle.Cursor
    if r.URL.Query().Has("cursor") {
        cursor = parseCursor(r.URL.Query().Get("cursor"))
    }

    // 3. Call particle service
    var parentPtr *string
    if parentID != "" {
        parentPtr = &parentID
    }

    result, err := h.particleService.ListParticles(ctx, networkID, parentPtr, email, filter, cursor)
    if err != nil { return err }

    // 4. Transform to response
    json.NewEncoder(w).Encode(toParticleListResponse(result))
}

Response DTO structure

// Handler layer DTOs (in handler.go)
type ParticleResponse struct {
    ID          string          `json:"id"`
    Type        string          `json:"type"`
    NetworkID   string          `json:"network_id"`
    ParentID    *string         `json:"parent_id,omitempty"`
    Visibility  string          `json:"visibility"`
    StreamStatus *string        `json:"stream_status,omitempty"`  // Only for streams
    Data        json.RawMessage `json:"data"`
    CreatedBy   string          `json:"created_by"`
    UpdatedAt   string          `json:"updated_at"`
    CreatedAt   string          `json:"created_at"`
}

// Transform function
func toParticleResponse(p *particle.Particle) ParticleResponse {
    return ParticleResponse{
        ID:           p.ID,
        Type:         string(p.Type),
        NetworkID:    p.NetworkID,
        ParentID:     p.ParentID,
        Visibility:   string(p.Visibility),
        StreamStatus: (*string)(p.StreamStatus),
        Data:         p.Data,
        CreatedBy:    p.CreatedByEmail,
        UpdatedAt:    p.UpdatedAt.Format(time.RFC3339),
        CreatedAt:    p.CreatedAt.Format(time.RFC3339),
    }
}

---

Particle Types

Type	Purpose	Required Data Fields
stream	Temporal container (chat-like)	title
folder	Structural container	name
media	Images, video, audio, clips	url, mime_type
file	Documents, PDFs, attachments	url, filename
text	Quick text messages	content
quest	Tasks/requests	title
paper	Rich documents	title
think	AI chat container	title

Note: Quest with assigned_to = current_user is presented as a "request" in UI.

---

Database Schema

Migration 1: Network Capacity

-- migrations/000003_network_capacity.up.sql
ALTER TABLE networks
  ADD COLUMN open_stream_capacity INTEGER NOT NULL DEFAULT 5,
  ADD COLUMN open_stream_count INTEGER NOT NULL DEFAULT 0;

-- migrations/000003_network_capacity.down.sql
ALTER TABLE networks
  DROP COLUMN open_stream_capacity,
  DROP COLUMN open_stream_count;

Migration 2: Particles

-- migrations/000004_particles.up.sql

CREATE TYPE particle_type AS ENUM (
  'stream', 'folder', 'media', 'file', 'text', 'quest', 'paper', 'think'
);

CREATE TYPE visibility_mode AS ENUM ('network_all', 'custom');

CREATE TABLE particles (
  id TEXT PRIMARY KEY,
  type particle_type NOT NULL,
  network_id TEXT NOT NULL,  -- NO FK constraint, loose coupling
  parent_id TEXT,            -- NO FK constraint, loose coupling
  created_by_email VARCHAR(255) NOT NULL,
  visibility visibility_mode NOT NULL DEFAULT 'network_all',

  -- Stream-specific (NULL for non-streams)
  stream_status VARCHAR(20) CHECK (stream_status IN ('open', 'closed')),

  -- Type-specific data
  data JSONB NOT NULL DEFAULT '{}',

  -- Timestamps
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

  CONSTRAINT stream_status_check CHECK (
    (type = 'stream' AND stream_status IS NOT NULL) OR
    (type != 'stream' AND stream_status IS NULL)
  )
);

CREATE TABLE particle_members (
  particle_id TEXT NOT NULL,  -- NO FK constraint
  email VARCHAR(255) NOT NULL,
  added_at TIMESTAMPTZ DEFAULT NOW(),
  PRIMARY KEY (particle_id, email)
);

-- Indexes for query performance
CREATE INDEX idx_particles_parent_updated ON particles(parent_id, updated_at DESC);
CREATE INDEX idx_particles_network_root ON particles(network_id, updated_at DESC) WHERE parent_id IS NULL;
CREATE INDEX idx_particles_open_streams ON particles(network_id) WHERE type = 'stream' AND stream_status = 'open';
CREATE INDEX idx_particle_members_email ON particle_members(email, particle_id);
CREATE INDEX idx_particles_network_id ON particles(network_id);

-- migrations/000004_particles.down.sql
DROP TABLE IF EXISTS particle_members;
DROP TABLE IF EXISTS particles;
DROP TYPE IF EXISTS visibility_mode;
DROP TYPE IF EXISTS particle_type;

---

Service Layer Design

Package Structure

internal/particle/
├── models.go       # Particle struct, type constants, data structs
├── errors.go       # ErrNotFound, ErrCapacityExceeded, ErrAccessDenied
├── repository.go   # Database operations (internal)
├── service.go      # Public Service interface + implementation
├── validation.go   # Type-specific validation (simple required fields)
└── service_test.go # Integration tests

Domain Models

// internal/particle/models.go

type ParticleType string

const (
    TypeStream ParticleType = "stream"
    TypeFolder ParticleType = "folder"
    TypeMedia  ParticleType = "media"
    TypeFile   ParticleType = "file"
    TypeText   ParticleType = "text"
    TypeQuest  ParticleType = "quest"
    TypePaper  ParticleType = "paper"
    TypeThink  ParticleType = "think"
)

type VisibilityMode string

const (
    VisibilityNetworkAll VisibilityMode = "network_all"
    VisibilityCustom     VisibilityMode = "custom"
)

type StreamStatus string

const (
    StreamOpen   StreamStatus = "open"
    StreamClosed StreamStatus = "closed"
)

type Particle struct {
    ID             string
    Type           ParticleType
    NetworkID      string
    ParentID       *string
    CreatedByEmail string
    Visibility     VisibilityMode
    StreamStatus   *StreamStatus  // Only for type=stream
    Data           json.RawMessage
    UpdatedAt      time.Time
    CreatedAt      time.Time
}

type CreateInput struct {
    Type           ParticleType
    NetworkID      string
    ParentID       *string
    Visibility     VisibilityMode
    Data           json.RawMessage
    MemberEmails   []string  // For custom visibility
}

type ListFilter struct {
    Types        []ParticleType
    StreamStatus *StreamStatus
}

type Cursor struct {
    Position  string  // particle ID or timestamp
    Direction string  // "before" | "after"
}

type ParticleList struct {
    Parent     *Particle   // The parent particle (nil if root level)
    Particles  []*Particle
    HasMore    bool
    NextCursor *Cursor     // For loading more in same direction
    PrevCursor *Cursor     // For bidirectional (streams)
}

Service Interface

// internal/particle/service.go

type Service interface {
    // Core CRUD
    // Note: Caller (handler) is responsible for verifying network membership
    // Service handles particle-level visibility filtering

    Create(ctx context.Context, input CreateInput, creatorEmail string) (*Particle, error)
    GetByID(ctx context.Context, id, requesterEmail string) (*Particle, error)
    Update(ctx context.Context, id string, data json.RawMessage, requesterEmail string) (*Particle, error)
    Delete(ctx context.Context, id, requesterEmail string) error

    // Unified listing - file explorer style
    // - parentID = nil → root particles of network
    // - parentID = "p_123" → children of that particle
    // - Automatically filters by visibility
    // - Streams: sorted by updated_at DESC (activity-based)
    // - Folders: sorted by created_at (client can re-sort by type)
    ListParticles(ctx context.Context, networkID string, parentID *string, requesterEmail string, filter ListFilter, cursor *Cursor) (*ParticleList, error)

    // Stream lifecycle
    OpenStream(ctx context.Context, id, requesterEmail string) error
    CloseStream(ctx context.Context, id, requesterEmail string) error

    // Returns current open stream count for a network (for capacity check)
    GetOpenStreamCount(ctx context.Context, networkID string) (int, error)

    // Membership (for custom visibility)
    SetVisibility(ctx context.Context, id string, mode VisibilityMode, requesterEmail string) error
    AddMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
    RemoveMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
    GetMembers(ctx context.Context, id string) ([]string, error)
}

ListParticles Implementation Logic

func (s *service) ListParticles(ctx context.Context, networkID string, parentID *string, email string, filter ListFilter, cursor *Cursor) (*ParticleList, error) {
    var parent *Particle
    var sortBy string

    // 1. Determine parent and sort strategy
    if parentID != nil {
        var err error
        parent, err = s.repo.getByID(ctx, *parentID)
        if err != nil {
            return nil, ErrNotFound
        }

        // Check visibility access to parent
        if !s.canAccess(ctx, *parentID, email) {
            return nil, ErrAccessDenied
        }

        // Sort based on parent type
        if parent.Type == TypeStream {
            sortBy = "updated_at DESC"  // Activity-based for streams
        } else {
            sortBy = "created_at DESC"  // Chronological for folders
        }
    } else {
        sortBy = "updated_at DESC"  // Root level: activity-based
    }

    // 2. Fetch particles with visibility filtering
    particles, hasMore, nextCursor, prevCursor := s.repo.listChildren(ctx, networkID, parentID, email, sortBy, filter, cursor)

    return &ParticleList{
        Parent:     parent,
        Particles:  particles,
        HasMore:    hasMore,
        NextCursor: nextCursor,
        PrevCursor: prevCursor,
    }, nil
}

Simple Validation (Start Simple)

// internal/particle/validation.go

func validateData(t ParticleType, data json.RawMessage) error {
    switch t {
    case TypeStream, TypeQuest, TypePaper, TypeThink:
        return requireField(data, "title")
    case TypeFolder:
        return requireField(data, "name")
    case TypeMedia, TypeFile:
        return requireFields(data, "url", "mime_type")
    case TypeText:
        return requireField(data, "content")
    default:
        return ErrInvalidParticleType
    }
}

func requireField(data json.RawMessage, field string) error {
    var m map[string]interface{}
    if err := json.Unmarshal(data, &m); err != nil {
        return fmt.Errorf("invalid data JSON: %w", err)
    }
    if _, ok := m[field]; !ok {
        return fmt.Errorf("missing required field: %s", field)
    }
    return nil
}

---

Particle Visibility Logic

The particle service handles visibility filtering. It does not check network membership (handler does that).

For visibility = 'network_all'

• All network members can see it
• Since handler already verified network membership, service returns it

For visibility = 'custom'

• Only users in particle_members table can see it
• Service checks particle_members table

Inheritance Rule

• Children can only restrict access, not expand
• If parent has custom visibility, child must also be custom (or more restrictive)
• Creating a child with network_all under a custom parent → error

Access Check Query

-- Check if user can access a specific particle
-- Walks up the ancestor chain, verifies access at each level
WITH RECURSIVE ancestors AS (
  SELECT id, parent_id, visibility
  FROM particles
  WHERE id = $1

  UNION ALL

  SELECT p.id, p.parent_id, p.visibility
  FROM particles p
  JOIN ancestors a ON p.id = a.parent_id
)
SELECT bool_and(
  CASE
    WHEN visibility = 'network_all' THEN true  -- Handler already verified network membership
    WHEN visibility = 'custom' THEN (
      EXISTS (SELECT 1 FROM particle_members pm
              WHERE pm.particle_id = ancestors.id AND pm.email = $2)
    )
  END
) AS has_access
FROM ancestors;

---

Network Service Updates

New Methods (no changes to coupling)

// In internal/network/service.go
type Service interface {
    // ... existing methods ...

    // Admin capacity management
    SetOpenStreamCapacity(ctx context.Context, networkID string, capacity int) error
    GetCapacityInfo(ctx context.Context, networkID string) (capacity int, current int, err error)

    // Stream count updates (called by handler, not particle service)
    IncrementOpenStreamCount(ctx context.Context, networkID string) error
    DecrementOpenStreamCount(ctx context.Context, networkID string) error
}

Capacity Enforcement (in Handler)

// Handler: Opening a stream
func (h *Handler) OpenStream(w http.ResponseWriter, r *http.Request) {
    // ... auth and validation ...

    // 1. Get particle to find its network
    particle, err := h.particleService.GetByID(ctx, particleID, email)

    // 2. Check capacity
    capacity, current, err := h.networkService.GetCapacityInfo(ctx, particle.NetworkID)
    if current >= capacity {
        return Error("stream capacity exceeded")
    }

    // 3. Open the stream
    err = h.particleService.OpenStream(ctx, particleID, email)

    // 4. Increment counter
    err = h.networkService.IncrementOpenStreamCount(ctx, particle.NetworkID)
}

---

Files to Modify

File	Changes
internal/network/models.go	Add OpenStreamCapacity, OpenStreamCount fields
internal/network/repository.go	Add capacity CRUD methods
internal/network/service.go	Add SetOpenStreamCapacity, GetCapacityInfo, counter methods
internal/handler/handler.go	Wire up particle endpoints, add admin capacity endpoint
migrations/	Add 000003 and 000004 migration files

Files to Create

File	Purpose
internal/particle/models.go	Particle struct, type constants
internal/particle/errors.go	Domain errors
internal/particle/repository.go	Database operations
internal/particle/service.go	Business logic + visibility filtering
internal/particle/validation.go	Simple required field validation
internal/particle/service_test.go	Integration tests

---

Implementation Phases

Phase 1: Network Capacity

1. Create migration 000003_network_capacity
2. Update network models with capacity fields
3. Update network repository with capacity methods
4. Update network service with SetOpenStreamCapacity, GetCapacityInfo
5. Add admin endpoint in handler

Phase 2: Core Particle CRUD

1. Create migration 000004_particles
2. Create particle package with models, errors
3. Implement repository with basic CRUD
4. Implement simple validation
5. Write integration tests

Phase 3: Particle Visibility

1. Implement visibility filtering in list queries
2. Implement access check for GetByID
3. Implement membership management (AddMembers, RemoveMembers)
4. Test visibility scenarios

Phase 4: Stream Lifecycle

1. Implement OpenStream/CloseStream in particle service
2. Wire up capacity checks in handler
3. Test capacity enforcement

Phase 5: Unified ListParticles

1. Implement unified ListParticles method
2. Add parent-type-aware sorting (streams: updated_at, folders: created_at)
3. Implement bidirectional cursor pagination
4. Include parent in response for breadcrumbs

Phase 6: Handler Integration

1. Wire particle service to handler
2. Implement all particle endpoints
3. Add DTO transformations
4. End-to-end testing

---

Verification Plan

1. Integration tests: Full flow with test database (following existing pattern in network/service_test.go)
2. Manual testing:
   • Create network with capacity 2
   • Open 2 streams → succeeds
   • Open 3rd stream → fails with capacity error
   • Close a stream → can open new one
   • Create nested particles, verify visibility inheritance
   • Add custom members, verify restricted access
   • Test handler data flow end-to-end