# Particle System & Network Capacity Implementation Plan

## Summary

Implement a unified particle system where all content types (streams, folders, media, files, text, quests, papers, AI chats) are particles with a common structure but type-specific data. Add network capacity management for billing/limiting open streams.

---

## Key Design Decisions

| Decision | Choice |
|----------|--------|
| Data model | Unified particle with `type` field + JSONB `data` |
| Hierarchy | Arbitrary nesting (parent_id references another particle) |
| Access control | Split: Handler checks network membership, Particle service checks particle visibility |
| Stream membership | Auto-visible to all network members OR custom member list |
| Open streams | Only "open" streams count against capacity |
| Substream counting | All stream particles count (including nested) |
| Service coupling | **Loose** - no FK constraints, particle service is independent |
| Listing API | Unified `ListParticles(networkID, parentID, ...)` - file explorer style |
| Sorting | Streams: `updated_at DESC` (activity), Folders: `created_at` (client sorts by type) |
| Pagination | Bidirectional cursor for streams (chat-like); folders return all |
| Network capacity | Stored on networks table (`open_stream_capacity`), updated via admin API |
| Data validation | Start simple with required field validation in service code |

---

## Architecture: Access Control Split

```
┌─────────────────────────────────────────────────────────────────────┐
│                           HANDLER LAYER                              │
│  1. Extract user email from auth context                            │
│  2. Check network membership (via network service)                  │
│  3. Call particle service with (networkID, email)                   │
│  4. Transform Particle structs → Response DTOs                      │
└─────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        PARTICLE SERVICE                              │
│  - Trusts that handler verified network membership                  │
│  - Checks particle-level visibility (network_all vs custom)         │
│  - Filters results to only particles user can see                   │
│  - Returns Particle domain structs                                  │
│  - NO dependency on network service                                 │
└─────────────────────────────────────────────────────────────────────┘
```

**Why this split?**
- Particle service stays decoupled from network service
- Network membership is a cross-cutting concern (handler already knows user context)
- Particle visibility is domain-specific (belongs in particle service)

---

## UI Data Flow Examples

### Mental Model: File Explorer

The API follows a file explorer pattern:
- `parentID = nil` → root items of network
- `parentID = "p_123"` → children of that particle
- Same method works at every level
- Response includes parent for breadcrumbs/context

### Example 1: User Opens App → Network Sidebar

```
┌─────────────────────────────────────────────────────┐
│  Streams in Acme Corp                               │
│  ├─ 📂 Projects (folder)                           │
│  │   ├─ 💬 Website Redesign (stream, open)         │
│  │   └─ 💬 Mobile App (stream, closed)             │
│  ├─ 💬 General Chat (stream, open)                 │
│  └─ 💬 Support Tickets (stream, open)              │
└─────────────────────────────────────────────────────┘
```

**API Calls:**

```go
// 1. Get root particles for selected network
resp := particleService.ListParticles(ctx, "net_abc", nil, email, ListFilter{}, nil)
// Returns:
// {
//   Parent: nil,  // No parent at root
//   Particles: [
//     {ID: "p_1", Type: "folder", Data: {"name": "Projects"}, ...},
//     {ID: "p_2", Type: "stream", Data: {"title": "General Chat"}, StreamStatus: "open"},
//     {ID: "p_3", Type: "stream", Data: {"title": "Support Tickets"}, StreamStatus: "open"},
//   ]
// }

// 2. User clicks "Projects" folder → fetch children
resp := particleService.ListParticles(ctx, "net_abc", ptr("p_1"), email, ListFilter{}, nil)
// Returns:
// {
//   Parent: {ID: "p_1", Type: "folder", Data: {"name": "Projects"}},  // For breadcrumbs
//   Particles: [
//     {ID: "p_4", Type: "stream", Data: {"title": "Website Redesign"}, StreamStatus: "open"},
//     {ID: "p_5", Type: "stream", Data: {"title": "Mobile App"}, StreamStatus: "closed"},
//   ]
// }
```

### Example 2: User Opens Stream → Chat View

```
┌─────────────────────────────────────────────────────┐
│  💬 Website Redesign                     [Members]  │
├─────────────────────────────────────────────────────┤
│  ↑ Load older                                      │
│  ─────────────────────────────────────────────────  │
│  [Alice] Here's the new mockup                     │
│  📎 mockup-v2.png (media particle)                 │
│  ─────────────────────────────────────────────────  │
│  [Bob] Looks great! Question about nav             │
│  ─────────────────────────────────────────────────  │
│  📋 Update navigation colors (quest)               │
│  ─────────────────────────────────────────────────  │
│  [Type a message...]                    [Send]      │
└─────────────────────────────────────────────────────┘
```

**API Calls:**

```go
// 1. Initial load - most recent particles (sorted by updated_at DESC)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{}, nil)
// Returns:
// {
//   Parent: {ID: "stream_123", Type: "stream", Data: {"title": "Website Redesign"}},
//   Particles: [newest...oldest],  // Sorted by updated_at DESC
//   HasMore: true,
//   PrevCursor: {Position: "p_oldest_in_batch", Direction: "before"},
//   NextCursor: nil  // At newest
// }

// 2. User scrolls UP → load older messages
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
    &Cursor{Position: "p_oldest_in_batch", Direction: "before"})

// 3. User scrolls DOWN → load newer (after scrolling up)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
    &Cursor{Position: "some_id", Direction: "after"})

// 4. User sends message
newParticle := particleService.Create(ctx, CreateInput{
    Type: TypeText,
    NetworkID: networkID,
    ParentID: ptr("stream_123"),
    Data: json.RawMessage(`{"content": "My message"}`),
}, email)
// UI inserts at bottom
```

### Handler Implementation

```go
// Handler pseudocode
func (h *Handler) ListParticles(w http.ResponseWriter, r *http.Request) {
    email := getAuthEmail(r.Context())
    networkID := r.URL.Query().Get("network_id")
    parentID := r.URL.Query().Get("parent_id")  // Optional

    // 1. Check network membership (handler responsibility)
    network, err := h.networkService.GetByID(ctx, networkID)
    if err != nil { return NotFound }

    if !network.HasMember(email) && network.AdminEmail != email {
        return Forbidden("not a network member")
    }

    // 2. Parse cursor if provided
    var cursor *particle.Cursor
    if r.URL.Query().Has("cursor") {
        cursor = parseCursor(r.URL.Query().Get("cursor"))
    }

    // 3. Call particle service
    var parentPtr *string
    if parentID != "" {
        parentPtr = &parentID
    }

    result, err := h.particleService.ListParticles(ctx, networkID, parentPtr, email, filter, cursor)
    if err != nil { return err }

    // 4. Transform to response
    json.NewEncoder(w).Encode(toParticleListResponse(result))
}
```

### Response DTO structure

```go
// Handler layer DTOs (in handler.go)
type ParticleResponse struct {
    ID          string          `json:"id"`
    Type        string          `json:"type"`
    NetworkID   string          `json:"network_id"`
    ParentID    *string         `json:"parent_id,omitempty"`
    Visibility  string          `json:"visibility"`
    StreamStatus *string        `json:"stream_status,omitempty"`  // Only for streams
    Data        json.RawMessage `json:"data"`
    CreatedBy   string          `json:"created_by"`
    UpdatedAt   string          `json:"updated_at"`
    CreatedAt   string          `json:"created_at"`
}

// Transform function
func toParticleResponse(p *particle.Particle) ParticleResponse {
    return ParticleResponse{
        ID:           p.ID,
        Type:         string(p.Type),
        NetworkID:    p.NetworkID,
        ParentID:     p.ParentID,
        Visibility:   string(p.Visibility),
        StreamStatus: (*string)(p.StreamStatus),
        Data:         p.Data,
        CreatedBy:    p.CreatedByEmail,
        UpdatedAt:    p.UpdatedAt.Format(time.RFC3339),
        CreatedAt:    p.CreatedAt.Format(time.RFC3339),
    }
}
```

---

## Particle Types

| Type | Purpose | Required Data Fields |
|------|---------|---------------------|
| `stream` | Temporal container (chat-like) | `title` |
| `folder` | Structural container | `name` |
| `media` | Images, video, audio, clips | `url`, `mime_type` |
| `file` | Documents, PDFs, attachments | `url`, `filename` |
| `text` | Quick text messages | `content` |
| `quest` | Tasks/requests | `title` |
| `paper` | Rich documents | `title` |
| `think` | AI chat container | `title` |

**Note:** Quest with `assigned_to = current_user` is presented as a "request" in UI.

---

## Database Schema

### Migration 1: Network Capacity

```sql
-- migrations/000003_network_capacity.up.sql
ALTER TABLE networks
  ADD COLUMN open_stream_capacity INTEGER NOT NULL DEFAULT 5,
  ADD COLUMN open_stream_count INTEGER NOT NULL DEFAULT 0;
```

```sql
-- migrations/000003_network_capacity.down.sql
ALTER TABLE networks
  DROP COLUMN open_stream_capacity,
  DROP COLUMN open_stream_count;
```

### Migration 2: Particles

```sql
-- migrations/000004_particles.up.sql

CREATE TYPE particle_type AS ENUM (
  'stream', 'folder', 'media', 'file', 'text', 'quest', 'paper', 'think'
);

CREATE TYPE visibility_mode AS ENUM ('network_all', 'custom');

CREATE TABLE particles (
  id TEXT PRIMARY KEY,
  type particle_type NOT NULL,
  network_id TEXT NOT NULL,  -- NO FK constraint, loose coupling
  parent_id TEXT,            -- NO FK constraint, loose coupling
  created_by_email VARCHAR(255) NOT NULL,
  visibility visibility_mode NOT NULL DEFAULT 'network_all',

  -- Stream-specific (NULL for non-streams)
  stream_status VARCHAR(20) CHECK (stream_status IN ('open', 'closed')),

  -- Type-specific data
  data JSONB NOT NULL DEFAULT '{}',

  -- Timestamps
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

  CONSTRAINT stream_status_check CHECK (
    (type = 'stream' AND stream_status IS NOT NULL) OR
    (type != 'stream' AND stream_status IS NULL)
  )
);

CREATE TABLE particle_members (
  particle_id TEXT NOT NULL,  -- NO FK constraint
  email VARCHAR(255) NOT NULL,
  added_at TIMESTAMPTZ DEFAULT NOW(),
  PRIMARY KEY (particle_id, email)
);

-- Indexes for query performance
CREATE INDEX idx_particles_parent_updated ON particles(parent_id, updated_at DESC);
CREATE INDEX idx_particles_network_root ON particles(network_id, updated_at DESC) WHERE parent_id IS NULL;
CREATE INDEX idx_particles_open_streams ON particles(network_id) WHERE type = 'stream' AND stream_status = 'open';
CREATE INDEX idx_particle_members_email ON particle_members(email, particle_id);
CREATE INDEX idx_particles_network_id ON particles(network_id);
```

```sql
-- migrations/000004_particles.down.sql
DROP TABLE IF EXISTS particle_members;
DROP TABLE IF EXISTS particles;
DROP TYPE IF EXISTS visibility_mode;
DROP TYPE IF EXISTS particle_type;
```

---

## Service Layer Design

### Package Structure

```
internal/particle/
├── models.go       # Particle struct, type constants, data structs
├── errors.go       # ErrNotFound, ErrCapacityExceeded, ErrAccessDenied
├── repository.go   # Database operations (internal)
├── service.go      # Public Service interface + implementation
├── validation.go   # Type-specific validation (simple required fields)
└── service_test.go # Integration tests
```

### Domain Models

```go
// internal/particle/models.go

type ParticleType string

const (
    TypeStream ParticleType = "stream"
    TypeFolder ParticleType = "folder"
    TypeMedia  ParticleType = "media"
    TypeFile   ParticleType = "file"
    TypeText   ParticleType = "text"
    TypeQuest  ParticleType = "quest"
    TypePaper  ParticleType = "paper"
    TypeThink  ParticleType = "think"
)

type VisibilityMode string

const (
    VisibilityNetworkAll VisibilityMode = "network_all"
    VisibilityCustom     VisibilityMode = "custom"
)

type StreamStatus string

const (
    StreamOpen   StreamStatus = "open"
    StreamClosed StreamStatus = "closed"
)

type Particle struct {
    ID             string
    Type           ParticleType
    NetworkID      string
    ParentID       *string
    CreatedByEmail string
    Visibility     VisibilityMode
    StreamStatus   *StreamStatus  // Only for type=stream
    Data           json.RawMessage
    UpdatedAt      time.Time
    CreatedAt      time.Time
}

type CreateInput struct {
    Type           ParticleType
    NetworkID      string
    ParentID       *string
    Visibility     VisibilityMode
    Data           json.RawMessage
    MemberEmails   []string  // For custom visibility
}

type ListFilter struct {
    Types        []ParticleType
    StreamStatus *StreamStatus
}

type Cursor struct {
    Position  string  // particle ID or timestamp
    Direction string  // "before" | "after"
}

type ParticleList struct {
    Parent     *Particle   // The parent particle (nil if root level)
    Particles  []*Particle
    HasMore    bool
    NextCursor *Cursor     // For loading more in same direction
    PrevCursor *Cursor     // For bidirectional (streams)
}
```

### Service Interface

```go
// internal/particle/service.go

type Service interface {
    // Core CRUD
    // Note: Caller (handler) is responsible for verifying network membership
    // Service handles particle-level visibility filtering

    Create(ctx context.Context, input CreateInput, creatorEmail string) (*Particle, error)
    GetByID(ctx context.Context, id, requesterEmail string) (*Particle, error)
    Update(ctx context.Context, id string, data json.RawMessage, requesterEmail string) (*Particle, error)
    Delete(ctx context.Context, id, requesterEmail string) error

    // Unified listing - file explorer style
    // - parentID = nil → root particles of network
    // - parentID = "p_123" → children of that particle
    // - Automatically filters by visibility
    // - Streams: sorted by updated_at DESC (activity-based)
    // - Folders: sorted by created_at (client can re-sort by type)
    ListParticles(ctx context.Context, networkID string, parentID *string, requesterEmail string, filter ListFilter, cursor *Cursor) (*ParticleList, error)

    // Stream lifecycle
    OpenStream(ctx context.Context, id, requesterEmail string) error
    CloseStream(ctx context.Context, id, requesterEmail string) error

    // Returns current open stream count for a network (for capacity check)
    GetOpenStreamCount(ctx context.Context, networkID string) (int, error)

    // Membership (for custom visibility)
    SetVisibility(ctx context.Context, id string, mode VisibilityMode, requesterEmail string) error
    AddMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
    RemoveMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
    GetMembers(ctx context.Context, id string) ([]string, error)
}
```

### ListParticles Implementation Logic

```go
func (s *service) ListParticles(ctx context.Context, networkID string, parentID *string, email string, filter ListFilter, cursor *Cursor) (*ParticleList, error) {
    var parent *Particle
    var sortBy string

    // 1. Determine parent and sort strategy
    if parentID != nil {
        var err error
        parent, err = s.repo.getByID(ctx, *parentID)
        if err != nil {
            return nil, ErrNotFound
        }

        // Check visibility access to parent
        if !s.canAccess(ctx, *parentID, email) {
            return nil, ErrAccessDenied
        }

        // Sort based on parent type
        if parent.Type == TypeStream {
            sortBy = "updated_at DESC"  // Activity-based for streams
        } else {
            sortBy = "created_at DESC"  // Chronological for folders
        }
    } else {
        sortBy = "updated_at DESC"  // Root level: activity-based
    }

    // 2. Fetch particles with visibility filtering
    particles, hasMore, nextCursor, prevCursor := s.repo.listChildren(ctx, networkID, parentID, email, sortBy, filter, cursor)

    return &ParticleList{
        Parent:     parent,
        Particles:  particles,
        HasMore:    hasMore,
        NextCursor: nextCursor,
        PrevCursor: prevCursor,
    }, nil
}
```

### Simple Validation (Start Simple)

```go
// internal/particle/validation.go

func validateData(t ParticleType, data json.RawMessage) error {
    switch t {
    case TypeStream, TypeQuest, TypePaper, TypeThink:
        return requireField(data, "title")
    case TypeFolder:
        return requireField(data, "name")
    case TypeMedia, TypeFile:
        return requireFields(data, "url", "mime_type")
    case TypeText:
        return requireField(data, "content")
    default:
        return ErrInvalidParticleType
    }
}

func requireField(data json.RawMessage, field string) error {
    var m map[string]interface{}
    if err := json.Unmarshal(data, &m); err != nil {
        return fmt.Errorf("invalid data JSON: %w", err)
    }
    if _, ok := m[field]; !ok {
        return fmt.Errorf("missing required field: %s", field)
    }
    return nil
}
```

---

## Particle Visibility Logic

The particle service handles visibility filtering. It does **not** check network membership (handler does that).

### For `visibility = 'network_all'`
- All network members can see it
- Since handler already verified network membership, service returns it

### For `visibility = 'custom'`
- Only users in `particle_members` table can see it
- Service checks `particle_members` table

### Inheritance Rule
- Children can only **restrict** access, not expand
- If parent has `custom` visibility, child must also be `custom` (or more restrictive)
- Creating a child with `network_all` under a `custom` parent → error

### Access Check Query

```sql
-- Check if user can access a specific particle
-- Walks up the ancestor chain, verifies access at each level
WITH RECURSIVE ancestors AS (
  SELECT id, parent_id, visibility
  FROM particles
  WHERE id = $1

  UNION ALL

  SELECT p.id, p.parent_id, p.visibility
  FROM particles p
  JOIN ancestors a ON p.id = a.parent_id
)
SELECT bool_and(
  CASE
    WHEN visibility = 'network_all' THEN true  -- Handler already verified network membership
    WHEN visibility = 'custom' THEN (
      EXISTS (SELECT 1 FROM particle_members pm
              WHERE pm.particle_id = ancestors.id AND pm.email = $2)
    )
  END
) AS has_access
FROM ancestors;
```

---

## Network Service Updates

### New Methods (no changes to coupling)

```go
// In internal/network/service.go
type Service interface {
    // ... existing methods ...

    // Admin capacity management
    SetOpenStreamCapacity(ctx context.Context, networkID string, capacity int) error
    GetCapacityInfo(ctx context.Context, networkID string) (capacity int, current int, err error)

    // Stream count updates (called by handler, not particle service)
    IncrementOpenStreamCount(ctx context.Context, networkID string) error
    DecrementOpenStreamCount(ctx context.Context, networkID string) error
}
```

### Capacity Enforcement (in Handler)

```go
// Handler: Opening a stream
func (h *Handler) OpenStream(w http.ResponseWriter, r *http.Request) {
    // ... auth and validation ...

    // 1. Get particle to find its network
    particle, err := h.particleService.GetByID(ctx, particleID, email)

    // 2. Check capacity
    capacity, current, err := h.networkService.GetCapacityInfo(ctx, particle.NetworkID)
    if current >= capacity {
        return Error("stream capacity exceeded")
    }

    // 3. Open the stream
    err = h.particleService.OpenStream(ctx, particleID, email)

    // 4. Increment counter
    err = h.networkService.IncrementOpenStreamCount(ctx, particle.NetworkID)
}
```

---

## Files to Modify

| File | Changes |
|------|---------|
| `internal/network/models.go` | Add `OpenStreamCapacity`, `OpenStreamCount` fields |
| `internal/network/repository.go` | Add capacity CRUD methods |
| `internal/network/service.go` | Add `SetOpenStreamCapacity`, `GetCapacityInfo`, counter methods |
| `internal/handler/handler.go` | Wire up particle endpoints, add admin capacity endpoint |
| `migrations/` | Add 000003 and 000004 migration files |

## Files to Create

| File | Purpose |
|------|---------|
| `internal/particle/models.go` | Particle struct, type constants |
| `internal/particle/errors.go` | Domain errors |
| `internal/particle/repository.go` | Database operations |
| `internal/particle/service.go` | Business logic + visibility filtering |
| `internal/particle/validation.go` | Simple required field validation |
| `internal/particle/service_test.go` | Integration tests |

---

## Implementation Phases

### Phase 1: Network Capacity
1. Create migration `000003_network_capacity`
2. Update network models with capacity fields
3. Update network repository with capacity methods
4. Update network service with `SetOpenStreamCapacity`, `GetCapacityInfo`
5. Add admin endpoint in handler

### Phase 2: Core Particle CRUD
1. Create migration `000004_particles`
2. Create particle package with models, errors
3. Implement repository with basic CRUD
4. Implement simple validation
5. Write integration tests

### Phase 3: Particle Visibility
1. Implement visibility filtering in list queries
2. Implement access check for GetByID
3. Implement membership management (AddMembers, RemoveMembers)
4. Test visibility scenarios

### Phase 4: Stream Lifecycle
1. Implement `OpenStream`/`CloseStream` in particle service
2. Wire up capacity checks in handler
3. Test capacity enforcement

### Phase 5: Unified ListParticles
1. Implement unified `ListParticles` method
2. Add parent-type-aware sorting (streams: updated_at, folders: created_at)
3. Implement bidirectional cursor pagination
4. Include parent in response for breadcrumbs

### Phase 6: Handler Integration
1. Wire particle service to handler
2. Implement all particle endpoints
3. Add DTO transformations
4. End-to-end testing

---

## Verification Plan

1. **Integration tests:** Full flow with test database (following existing pattern in `network/service_test.go`)
2. **Manual testing:**
   - Create network with capacity 2
   - Open 2 streams → succeeds
   - Open 3rd stream → fails with capacity error
   - Close a stream → can open new one
   - Create nested particles, verify visibility inheritance
   - Add custom members, verify restricted access
   - Test handler data flow end-to-end