migrate orion repo into monorepo structure

2026-02-21 08:48:34 -08:00
parent 144596fcaa
commit b5f90709de
91 changed files with 19775 additions and 0 deletions
@@ -0,0 +1,694 @@
+# Particle System & Network Capacity Implementation Plan
+
+## Summary
+
+Implement a unified particle system where all content types (streams, folders, media, files, text, quests, papers, AI chats) are particles with a common structure but type-specific data. Add network capacity management for billing/limiting open streams.
+
+---
+
+## Key Design Decisions
+
+| Decision | Choice |
+|----------|--------|
+| Data model | Unified particle with `type` field + JSONB `data` |
+| Hierarchy | Arbitrary nesting (parent_id references another particle) |
+| Access control | Split: Handler checks network membership, Particle service checks particle visibility |
+| Stream membership | Auto-visible to all network members OR custom member list |
+| Open streams | Only "open" streams count against capacity |
+| Substream counting | All stream particles count (including nested) |
+| Service coupling | **Loose** - no FK constraints, particle service is independent |
+| Listing API | Unified `ListParticles(networkID, parentID, ...)` - file explorer style |
+| Sorting | Streams: `updated_at DESC` (activity), Folders: `created_at` (client sorts by type) |
+| Pagination | Bidirectional cursor for streams (chat-like); folders return all |
+| Network capacity | Stored on networks table (`open_stream_capacity`), updated via admin API |
+| Data validation | Start simple with required field validation in service code |
+
+---
+
+## Architecture: Access Control Split
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                           HANDLER LAYER                              │
+│  1. Extract user email from auth context                            │
+│  2. Check network membership (via network service)                  │
+│  3. Call particle service with (networkID, email)                   │
+│  4. Transform Particle structs → Response DTOs                      │
+└─────────────────────────────────────────────────────────────────────┘
+                                   │
+                                   ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│                        PARTICLE SERVICE                              │
+│  - Trusts that handler verified network membership                  │
+│  - Checks particle-level visibility (network_all vs custom)         │
+│  - Filters results to only particles user can see                   │
+│  - Returns Particle domain structs                                  │
+│  - NO dependency on network service                                 │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+**Why this split?**
+- Particle service stays decoupled from network service
+- Network membership is a cross-cutting concern (handler already knows user context)
+- Particle visibility is domain-specific (belongs in particle service)
+
+---
+
+## UI Data Flow Examples
+
+### Mental Model: File Explorer
+
+The API follows a file explorer pattern:
+- `parentID = nil` → root items of network
+- `parentID = "p_123"` → children of that particle
+- Same method works at every level
+- Response includes parent for breadcrumbs/context
+
+### Example 1: User Opens App → Network Sidebar
+
+```
+┌─────────────────────────────────────────────────────┐
+│  Streams in Acme Corp                               │
+│  ├─ 📂 Projects (folder)                           │
+│  │   ├─ 💬 Website Redesign (stream, open)         │
+│  │   └─ 💬 Mobile App (stream, closed)             │
+│  ├─ 💬 General Chat (stream, open)                 │
+│  └─ 💬 Support Tickets (stream, open)              │
+└─────────────────────────────────────────────────────┘
+```
+
+**API Calls:**
+
+```go
+// 1. Get root particles for selected network
+resp := particleService.ListParticles(ctx, "net_abc", nil, email, ListFilter{}, nil)
+// Returns:
+// {
+//   Parent: nil,  // No parent at root
+//   Particles: [
+//     {ID: "p_1", Type: "folder", Data: {"name": "Projects"}, ...},
+//     {ID: "p_2", Type: "stream", Data: {"title": "General Chat"}, StreamStatus: "open"},
+//     {ID: "p_3", Type: "stream", Data: {"title": "Support Tickets"}, StreamStatus: "open"},
+//   ]
+// }
+
+// 2. User clicks "Projects" folder → fetch children
+resp := particleService.ListParticles(ctx, "net_abc", ptr("p_1"), email, ListFilter{}, nil)
+// Returns:
+// {
+//   Parent: {ID: "p_1", Type: "folder", Data: {"name": "Projects"}},  // For breadcrumbs
+//   Particles: [
+//     {ID: "p_4", Type: "stream", Data: {"title": "Website Redesign"}, StreamStatus: "open"},
+//     {ID: "p_5", Type: "stream", Data: {"title": "Mobile App"}, StreamStatus: "closed"},
+//   ]
+// }
+```
+
+### Example 2: User Opens Stream → Chat View
+
+```
+┌─────────────────────────────────────────────────────┐
+│  💬 Website Redesign                     [Members]  │
+├─────────────────────────────────────────────────────┤
+│  ↑ Load older                                      │
+│  ─────────────────────────────────────────────────  │
+│  [Alice] Here's the new mockup                     │
+│  📎 mockup-v2.png (media particle)                 │
+│  ─────────────────────────────────────────────────  │
+│  [Bob] Looks great! Question about nav             │
+│  ─────────────────────────────────────────────────  │
+│  📋 Update navigation colors (quest)               │
+│  ─────────────────────────────────────────────────  │
+│  [Type a message...]                    [Send]      │
+└─────────────────────────────────────────────────────┘
+```
+
+**API Calls:**
+
+```go
+// 1. Initial load - most recent particles (sorted by updated_at DESC)
+resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{}, nil)
+// Returns:
+// {
+//   Parent: {ID: "stream_123", Type: "stream", Data: {"title": "Website Redesign"}},
+//   Particles: [newest...oldest],  // Sorted by updated_at DESC
+//   HasMore: true,
+//   PrevCursor: {Position: "p_oldest_in_batch", Direction: "before"},
+//   NextCursor: nil  // At newest
+// }
+
+// 2. User scrolls UP → load older messages
+resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
+    &Cursor{Position: "p_oldest_in_batch", Direction: "before"})
+
+// 3. User scrolls DOWN → load newer (after scrolling up)
+resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
+    &Cursor{Position: "some_id", Direction: "after"})
+
+// 4. User sends message
+newParticle := particleService.Create(ctx, CreateInput{
+    Type: TypeText,
+    NetworkID: networkID,
+    ParentID: ptr("stream_123"),
+    Data: json.RawMessage(`{"content": "My message"}`),
+}, email)
+// UI inserts at bottom
+```
+
+### Handler Implementation
+
+```go
+// Handler pseudocode
+func (h *Handler) ListParticles(w http.ResponseWriter, r *http.Request) {
+    email := getAuthEmail(r.Context())
+    networkID := r.URL.Query().Get("network_id")
+    parentID := r.URL.Query().Get("parent_id")  // Optional
+
+    // 1. Check network membership (handler responsibility)
+    network, err := h.networkService.GetByID(ctx, networkID)
+    if err != nil { return NotFound }
+
+    if !network.HasMember(email) && network.AdminEmail != email {
+        return Forbidden("not a network member")
+    }
+
+    // 2. Parse cursor if provided
+    var cursor *particle.Cursor
+    if r.URL.Query().Has("cursor") {
+        cursor = parseCursor(r.URL.Query().Get("cursor"))
+    }
+
+    // 3. Call particle service
+    var parentPtr *string
+    if parentID != "" {
+        parentPtr = &parentID
+    }
+
+    result, err := h.particleService.ListParticles(ctx, networkID, parentPtr, email, filter, cursor)
+    if err != nil { return err }
+
+    // 4. Transform to response
+    json.NewEncoder(w).Encode(toParticleListResponse(result))
+}
+```
+
+### Response DTO structure
+
+```go
+// Handler layer DTOs (in handler.go)
+type ParticleResponse struct {
+    ID          string          `json:"id"`
+    Type        string          `json:"type"`
+    NetworkID   string          `json:"network_id"`
+    ParentID    *string         `json:"parent_id,omitempty"`
+    Visibility  string          `json:"visibility"`
+    StreamStatus *string        `json:"stream_status,omitempty"`  // Only for streams
+    Data        json.RawMessage `json:"data"`
+    CreatedBy   string          `json:"created_by"`
+    UpdatedAt   string          `json:"updated_at"`
+    CreatedAt   string          `json:"created_at"`
+}
+
+// Transform function
+func toParticleResponse(p *particle.Particle) ParticleResponse {
+    return ParticleResponse{
+        ID:           p.ID,
+        Type:         string(p.Type),
+        NetworkID:    p.NetworkID,
+        ParentID:     p.ParentID,
+        Visibility:   string(p.Visibility),
+        StreamStatus: (*string)(p.StreamStatus),
+        Data:         p.Data,
+        CreatedBy:    p.CreatedByEmail,
+        UpdatedAt:    p.UpdatedAt.Format(time.RFC3339),
+        CreatedAt:    p.CreatedAt.Format(time.RFC3339),
+    }
+}
+```
+
+---
+
+## Particle Types
+
+| Type | Purpose | Required Data Fields |
+|------|---------|---------------------|
+| `stream` | Temporal container (chat-like) | `title` |
+| `folder` | Structural container | `name` |
+| `media` | Images, video, audio, clips | `url`, `mime_type` |
+| `file` | Documents, PDFs, attachments | `url`, `filename` |
+| `text` | Quick text messages | `content` |
+| `quest` | Tasks/requests | `title` |
+| `paper` | Rich documents | `title` |
+| `think` | AI chat container | `title` |
+
+**Note:** Quest with `assigned_to = current_user` is presented as a "request" in UI.
+
+---
+
+## Database Schema
+
+### Migration 1: Network Capacity
+
+```sql
+-- migrations/000003_network_capacity.up.sql
+ALTER TABLE networks
+  ADD COLUMN open_stream_capacity INTEGER NOT NULL DEFAULT 5,
+  ADD COLUMN open_stream_count INTEGER NOT NULL DEFAULT 0;
+```
+
+```sql
+-- migrations/000003_network_capacity.down.sql
+ALTER TABLE networks
+  DROP COLUMN open_stream_capacity,
+  DROP COLUMN open_stream_count;
+```
+
+### Migration 2: Particles
+
+```sql
+-- migrations/000004_particles.up.sql
+
+CREATE TYPE particle_type AS ENUM (
+  'stream', 'folder', 'media', 'file', 'text', 'quest', 'paper', 'think'
+);
+
+CREATE TYPE visibility_mode AS ENUM ('network_all', 'custom');
+
+CREATE TABLE particles (
+  id TEXT PRIMARY KEY,
+  type particle_type NOT NULL,
+  network_id TEXT NOT NULL,  -- NO FK constraint, loose coupling
+  parent_id TEXT,            -- NO FK constraint, loose coupling
+  created_by_email VARCHAR(255) NOT NULL,
+  visibility visibility_mode NOT NULL DEFAULT 'network_all',
+
+  -- Stream-specific (NULL for non-streams)
+  stream_status VARCHAR(20) CHECK (stream_status IN ('open', 'closed')),
+
+  -- Type-specific data
+  data JSONB NOT NULL DEFAULT '{}',
+
+  -- Timestamps
+  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+  CONSTRAINT stream_status_check CHECK (
+    (type = 'stream' AND stream_status IS NOT NULL) OR
+    (type != 'stream' AND stream_status IS NULL)
+  )
+);
+
+CREATE TABLE particle_members (
+  particle_id TEXT NOT NULL,  -- NO FK constraint
+  email VARCHAR(255) NOT NULL,
+  added_at TIMESTAMPTZ DEFAULT NOW(),
+  PRIMARY KEY (particle_id, email)
+);
+
+-- Indexes for query performance
+CREATE INDEX idx_particles_parent_updated ON particles(parent_id, updated_at DESC);
+CREATE INDEX idx_particles_network_root ON particles(network_id, updated_at DESC) WHERE parent_id IS NULL;
+CREATE INDEX idx_particles_open_streams ON particles(network_id) WHERE type = 'stream' AND stream_status = 'open';
+CREATE INDEX idx_particle_members_email ON particle_members(email, particle_id);
+CREATE INDEX idx_particles_network_id ON particles(network_id);
+```
+
+```sql
+-- migrations/000004_particles.down.sql
+DROP TABLE IF EXISTS particle_members;
+DROP TABLE IF EXISTS particles;
+DROP TYPE IF EXISTS visibility_mode;
+DROP TYPE IF EXISTS particle_type;
+```
+
+---
+
+## Service Layer Design
+
+### Package Structure
+
+```
+internal/particle/
+├── models.go       # Particle struct, type constants, data structs
+├── errors.go       # ErrNotFound, ErrCapacityExceeded, ErrAccessDenied
+├── repository.go   # Database operations (internal)
+├── service.go      # Public Service interface + implementation
+├── validation.go   # Type-specific validation (simple required fields)
+└── service_test.go # Integration tests
+```
+
+### Domain Models
+
+```go
+// internal/particle/models.go
+
+type ParticleType string
+
+const (
+    TypeStream ParticleType = "stream"
+    TypeFolder ParticleType = "folder"
+    TypeMedia  ParticleType = "media"
+    TypeFile   ParticleType = "file"
+    TypeText   ParticleType = "text"
+    TypeQuest  ParticleType = "quest"
+    TypePaper  ParticleType = "paper"
+    TypeThink  ParticleType = "think"
+)
+
+type VisibilityMode string
+
+const (
+    VisibilityNetworkAll VisibilityMode = "network_all"
+    VisibilityCustom     VisibilityMode = "custom"
+)
+
+type StreamStatus string
+
+const (
+    StreamOpen   StreamStatus = "open"
+    StreamClosed StreamStatus = "closed"
+)
+
+type Particle struct {
+    ID             string
+    Type           ParticleType
+    NetworkID      string
+    ParentID       *string
+    CreatedByEmail string
+    Visibility     VisibilityMode
+    StreamStatus   *StreamStatus  // Only for type=stream
+    Data           json.RawMessage
+    UpdatedAt      time.Time
+    CreatedAt      time.Time
+}
+
+type CreateInput struct {
+    Type           ParticleType
+    NetworkID      string
+    ParentID       *string
+    Visibility     VisibilityMode
+    Data           json.RawMessage
+    MemberEmails   []string  // For custom visibility
+}
+
+type ListFilter struct {
+    Types        []ParticleType
+    StreamStatus *StreamStatus
+}
+
+type Cursor struct {
+    Position  string  // particle ID or timestamp
+    Direction string  // "before" | "after"
+}
+
+type ParticleList struct {
+    Parent     *Particle   // The parent particle (nil if root level)
+    Particles  []*Particle
+    HasMore    bool
+    NextCursor *Cursor     // For loading more in same direction
+    PrevCursor *Cursor     // For bidirectional (streams)
+}
+```
+
+### Service Interface
+
+```go
+// internal/particle/service.go
+
+type Service interface {
+    // Core CRUD
+    // Note: Caller (handler) is responsible for verifying network membership
+    // Service handles particle-level visibility filtering
+
+    Create(ctx context.Context, input CreateInput, creatorEmail string) (*Particle, error)
+    GetByID(ctx context.Context, id, requesterEmail string) (*Particle, error)
+    Update(ctx context.Context, id string, data json.RawMessage, requesterEmail string) (*Particle, error)
+    Delete(ctx context.Context, id, requesterEmail string) error
+
+    // Unified listing - file explorer style
+    // - parentID = nil → root particles of network
+    // - parentID = "p_123" → children of that particle
+    // - Automatically filters by visibility
+    // - Streams: sorted by updated_at DESC (activity-based)
+    // - Folders: sorted by created_at (client can re-sort by type)
+    ListParticles(ctx context.Context, networkID string, parentID *string, requesterEmail string, filter ListFilter, cursor *Cursor) (*ParticleList, error)
+
+    // Stream lifecycle
+    OpenStream(ctx context.Context, id, requesterEmail string) error
+    CloseStream(ctx context.Context, id, requesterEmail string) error
+
+    // Returns current open stream count for a network (for capacity check)
+    GetOpenStreamCount(ctx context.Context, networkID string) (int, error)
+
+    // Membership (for custom visibility)
+    SetVisibility(ctx context.Context, id string, mode VisibilityMode, requesterEmail string) error
+    AddMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
+    RemoveMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
+    GetMembers(ctx context.Context, id string) ([]string, error)
+}
+```
+
+### ListParticles Implementation Logic
+
+```go
+func (s *service) ListParticles(ctx context.Context, networkID string, parentID *string, email string, filter ListFilter, cursor *Cursor) (*ParticleList, error) {
+    var parent *Particle
+    var sortBy string
+
+    // 1. Determine parent and sort strategy
+    if parentID != nil {
+        var err error
+        parent, err = s.repo.getByID(ctx, *parentID)
+        if err != nil {
+            return nil, ErrNotFound
+        }
+
+        // Check visibility access to parent
+        if !s.canAccess(ctx, *parentID, email) {
+            return nil, ErrAccessDenied
+        }
+
+        // Sort based on parent type
+        if parent.Type == TypeStream {
+            sortBy = "updated_at DESC"  // Activity-based for streams
+        } else {
+            sortBy = "created_at DESC"  // Chronological for folders
+        }
+    } else {
+        sortBy = "updated_at DESC"  // Root level: activity-based
+    }
+
+    // 2. Fetch particles with visibility filtering
+    particles, hasMore, nextCursor, prevCursor := s.repo.listChildren(ctx, networkID, parentID, email, sortBy, filter, cursor)
+
+    return &ParticleList{
+        Parent:     parent,
+        Particles:  particles,
+        HasMore:    hasMore,
+        NextCursor: nextCursor,
+        PrevCursor: prevCursor,
+    }, nil
+}
+```
+
+### Simple Validation (Start Simple)
+
+```go
+// internal/particle/validation.go
+
+func validateData(t ParticleType, data json.RawMessage) error {
+    switch t {
+    case TypeStream, TypeQuest, TypePaper, TypeThink:
+        return requireField(data, "title")
+    case TypeFolder:
+        return requireField(data, "name")
+    case TypeMedia, TypeFile:
+        return requireFields(data, "url", "mime_type")
+    case TypeText:
+        return requireField(data, "content")
+    default:
+        return ErrInvalidParticleType
+    }
+}
+
+func requireField(data json.RawMessage, field string) error {
+    var m map[string]interface{}
+    if err := json.Unmarshal(data, &m); err != nil {
+        return fmt.Errorf("invalid data JSON: %w", err)
+    }
+    if _, ok := m[field]; !ok {
+        return fmt.Errorf("missing required field: %s", field)
+    }
+    return nil
+}
+```
+
+---
+
+## Particle Visibility Logic
+
+The particle service handles visibility filtering. It does **not** check network membership (handler does that).
+
+### For `visibility = 'network_all'`
+- All network members can see it
+- Since handler already verified network membership, service returns it
+
+### For `visibility = 'custom'`
+- Only users in `particle_members` table can see it
+- Service checks `particle_members` table
+
+### Inheritance Rule
+- Children can only **restrict** access, not expand
+- If parent has `custom` visibility, child must also be `custom` (or more restrictive)
+- Creating a child with `network_all` under a `custom` parent → error
+
+### Access Check Query
+
+```sql
+-- Check if user can access a specific particle
+-- Walks up the ancestor chain, verifies access at each level
+WITH RECURSIVE ancestors AS (
+  SELECT id, parent_id, visibility
+  FROM particles
+  WHERE id = $1
+
+  UNION ALL
+
+  SELECT p.id, p.parent_id, p.visibility
+  FROM particles p
+  JOIN ancestors a ON p.id = a.parent_id
+)
+SELECT bool_and(
+  CASE
+    WHEN visibility = 'network_all' THEN true  -- Handler already verified network membership
+    WHEN visibility = 'custom' THEN (
+      EXISTS (SELECT 1 FROM particle_members pm
+              WHERE pm.particle_id = ancestors.id AND pm.email = $2)
+    )
+  END
+) AS has_access
+FROM ancestors;
+```
+
+---
+
+## Network Service Updates
+
+### New Methods (no changes to coupling)
+
+```go
+// In internal/network/service.go
+type Service interface {
+    // ... existing methods ...
+
+    // Admin capacity management
+    SetOpenStreamCapacity(ctx context.Context, networkID string, capacity int) error
+    GetCapacityInfo(ctx context.Context, networkID string) (capacity int, current int, err error)
+
+    // Stream count updates (called by handler, not particle service)
+    IncrementOpenStreamCount(ctx context.Context, networkID string) error
+    DecrementOpenStreamCount(ctx context.Context, networkID string) error
+}
+```
+
+### Capacity Enforcement (in Handler)
+
+```go
+// Handler: Opening a stream
+func (h *Handler) OpenStream(w http.ResponseWriter, r *http.Request) {
+    // ... auth and validation ...
+
+    // 1. Get particle to find its network
+    particle, err := h.particleService.GetByID(ctx, particleID, email)
+
+    // 2. Check capacity
+    capacity, current, err := h.networkService.GetCapacityInfo(ctx, particle.NetworkID)
+    if current >= capacity {
+        return Error("stream capacity exceeded")
+    }
+
+    // 3. Open the stream
+    err = h.particleService.OpenStream(ctx, particleID, email)
+
+    // 4. Increment counter
+    err = h.networkService.IncrementOpenStreamCount(ctx, particle.NetworkID)
+}
+```
+
+---
+
+## Files to Modify
+
+| File | Changes |
+|------|---------|
+| `internal/network/models.go` | Add `OpenStreamCapacity`, `OpenStreamCount` fields |
+| `internal/network/repository.go` | Add capacity CRUD methods |
+| `internal/network/service.go` | Add `SetOpenStreamCapacity`, `GetCapacityInfo`, counter methods |
+| `internal/handler/handler.go` | Wire up particle endpoints, add admin capacity endpoint |
+| `migrations/` | Add 000003 and 000004 migration files |
+
+## Files to Create
+
+| File | Purpose |
+|------|---------|
+| `internal/particle/models.go` | Particle struct, type constants |
+| `internal/particle/errors.go` | Domain errors |
+| `internal/particle/repository.go` | Database operations |
+| `internal/particle/service.go` | Business logic + visibility filtering |
+| `internal/particle/validation.go` | Simple required field validation |
+| `internal/particle/service_test.go` | Integration tests |
+
+---
+
+## Implementation Phases
+
+### Phase 1: Network Capacity
+1. Create migration `000003_network_capacity`
+2. Update network models with capacity fields
+3. Update network repository with capacity methods
+4. Update network service with `SetOpenStreamCapacity`, `GetCapacityInfo`
+5. Add admin endpoint in handler
+
+### Phase 2: Core Particle CRUD
+1. Create migration `000004_particles`
+2. Create particle package with models, errors
+3. Implement repository with basic CRUD
+4. Implement simple validation
+5. Write integration tests
+
+### Phase 3: Particle Visibility
+1. Implement visibility filtering in list queries
+2. Implement access check for GetByID
+3. Implement membership management (AddMembers, RemoveMembers)
+4. Test visibility scenarios
+
+### Phase 4: Stream Lifecycle
+1. Implement `OpenStream`/`CloseStream` in particle service
+2. Wire up capacity checks in handler
+3. Test capacity enforcement
+
+### Phase 5: Unified ListParticles
+1. Implement unified `ListParticles` method
+2. Add parent-type-aware sorting (streams: updated_at, folders: created_at)
+3. Implement bidirectional cursor pagination
+4. Include parent in response for breadcrumbs
+
+### Phase 6: Handler Integration
+1. Wire particle service to handler
+2. Implement all particle endpoints
+3. Add DTO transformations
+4. End-to-end testing
+
+---
+
+## Verification Plan
+
+1. **Integration tests:** Full flow with test database (following existing pattern in `network/service_test.go`)
+2. **Manual testing:**
+   - Create network with capacity 2
+   - Open 2 streams → succeeds
+   - Open 3rd stream → fails with capacity error
+   - Close a stream → can open new one
+   - Create nested particles, verify visibility inheritance
+   - Add custom members, verify restricted access
+   - Test handler data flow end-to-end