Files
llink/go/docs/particle-system-plan.md
T
2026-02-21 08:48:34 -08:00

695 lines
24 KiB
Markdown

# Particle System & Network Capacity Implementation Plan
## Summary
Implement a unified particle system where all content types (streams, folders, media, files, text, quests, papers, AI chats) are particles with a common structure but type-specific data. Add network capacity management for billing/limiting open streams.
---
## Key Design Decisions
| Decision | Choice |
|----------|--------|
| Data model | Unified particle with `type` field + JSONB `data` |
| Hierarchy | Arbitrary nesting (parent_id references another particle) |
| Access control | Split: Handler checks network membership, Particle service checks particle visibility |
| Stream membership | Auto-visible to all network members OR custom member list |
| Open streams | Only "open" streams count against capacity |
| Substream counting | All stream particles count (including nested) |
| Service coupling | **Loose** - no FK constraints, particle service is independent |
| Listing API | Unified `ListParticles(networkID, parentID, ...)` - file explorer style |
| Sorting | Streams: `updated_at DESC` (activity), Folders: `created_at` (client sorts by type) |
| Pagination | Bidirectional cursor for streams (chat-like); folders return all |
| Network capacity | Stored on networks table (`open_stream_capacity`), updated via admin API |
| Data validation | Start simple with required field validation in service code |
---
## Architecture: Access Control Split
```
┌─────────────────────────────────────────────────────────────────────┐
│ HANDLER LAYER │
│ 1. Extract user email from auth context │
│ 2. Check network membership (via network service) │
│ 3. Call particle service with (networkID, email) │
│ 4. Transform Particle structs → Response DTOs │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ PARTICLE SERVICE │
│ - Trusts that handler verified network membership │
│ - Checks particle-level visibility (network_all vs custom) │
│ - Filters results to only particles user can see │
│ - Returns Particle domain structs │
│ - NO dependency on network service │
└─────────────────────────────────────────────────────────────────────┘
```
**Why this split?**
- Particle service stays decoupled from network service
- Network membership is a cross-cutting concern (handler already knows user context)
- Particle visibility is domain-specific (belongs in particle service)
---
## UI Data Flow Examples
### Mental Model: File Explorer
The API follows a file explorer pattern:
- `parentID = nil` → root items of network
- `parentID = "p_123"` → children of that particle
- Same method works at every level
- Response includes parent for breadcrumbs/context
### Example 1: User Opens App → Network Sidebar
```
┌─────────────────────────────────────────────────────┐
│ Streams in Acme Corp │
│ ├─ 📂 Projects (folder) │
│ │ ├─ 💬 Website Redesign (stream, open) │
│ │ └─ 💬 Mobile App (stream, closed) │
│ ├─ 💬 General Chat (stream, open) │
│ └─ 💬 Support Tickets (stream, open) │
└─────────────────────────────────────────────────────┘
```
**API Calls:**
```go
// 1. Get root particles for selected network
resp := particleService.ListParticles(ctx, "net_abc", nil, email, ListFilter{}, nil)
// Returns:
// {
// Parent: nil, // No parent at root
// Particles: [
// {ID: "p_1", Type: "folder", Data: {"name": "Projects"}, ...},
// {ID: "p_2", Type: "stream", Data: {"title": "General Chat"}, StreamStatus: "open"},
// {ID: "p_3", Type: "stream", Data: {"title": "Support Tickets"}, StreamStatus: "open"},
// ]
// }
// 2. User clicks "Projects" folder → fetch children
resp := particleService.ListParticles(ctx, "net_abc", ptr("p_1"), email, ListFilter{}, nil)
// Returns:
// {
// Parent: {ID: "p_1", Type: "folder", Data: {"name": "Projects"}}, // For breadcrumbs
// Particles: [
// {ID: "p_4", Type: "stream", Data: {"title": "Website Redesign"}, StreamStatus: "open"},
// {ID: "p_5", Type: "stream", Data: {"title": "Mobile App"}, StreamStatus: "closed"},
// ]
// }
```
### Example 2: User Opens Stream → Chat View
```
┌─────────────────────────────────────────────────────┐
│ 💬 Website Redesign [Members] │
├─────────────────────────────────────────────────────┤
│ ↑ Load older │
│ ───────────────────────────────────────────────── │
│ [Alice] Here's the new mockup │
│ 📎 mockup-v2.png (media particle) │
│ ───────────────────────────────────────────────── │
│ [Bob] Looks great! Question about nav │
│ ───────────────────────────────────────────────── │
│ 📋 Update navigation colors (quest) │
│ ───────────────────────────────────────────────── │
│ [Type a message...] [Send] │
└─────────────────────────────────────────────────────┘
```
**API Calls:**
```go
// 1. Initial load - most recent particles (sorted by updated_at DESC)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{}, nil)
// Returns:
// {
// Parent: {ID: "stream_123", Type: "stream", Data: {"title": "Website Redesign"}},
// Particles: [newest...oldest], // Sorted by updated_at DESC
// HasMore: true,
// PrevCursor: {Position: "p_oldest_in_batch", Direction: "before"},
// NextCursor: nil // At newest
// }
// 2. User scrolls UP → load older messages
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
&Cursor{Position: "p_oldest_in_batch", Direction: "before"})
// 3. User scrolls DOWN → load newer (after scrolling up)
resp := particleService.ListParticles(ctx, networkID, ptr("stream_123"), email, ListFilter{},
&Cursor{Position: "some_id", Direction: "after"})
// 4. User sends message
newParticle := particleService.Create(ctx, CreateInput{
Type: TypeText,
NetworkID: networkID,
ParentID: ptr("stream_123"),
Data: json.RawMessage(`{"content": "My message"}`),
}, email)
// UI inserts at bottom
```
### Handler Implementation
```go
// Handler pseudocode
func (h *Handler) ListParticles(w http.ResponseWriter, r *http.Request) {
email := getAuthEmail(r.Context())
networkID := r.URL.Query().Get("network_id")
parentID := r.URL.Query().Get("parent_id") // Optional
// 1. Check network membership (handler responsibility)
network, err := h.networkService.GetByID(ctx, networkID)
if err != nil { return NotFound }
if !network.HasMember(email) && network.AdminEmail != email {
return Forbidden("not a network member")
}
// 2. Parse cursor if provided
var cursor *particle.Cursor
if r.URL.Query().Has("cursor") {
cursor = parseCursor(r.URL.Query().Get("cursor"))
}
// 3. Call particle service
var parentPtr *string
if parentID != "" {
parentPtr = &parentID
}
result, err := h.particleService.ListParticles(ctx, networkID, parentPtr, email, filter, cursor)
if err != nil { return err }
// 4. Transform to response
json.NewEncoder(w).Encode(toParticleListResponse(result))
}
```
### Response DTO structure
```go
// Handler layer DTOs (in handler.go)
type ParticleResponse struct {
ID string `json:"id"`
Type string `json:"type"`
NetworkID string `json:"network_id"`
ParentID *string `json:"parent_id,omitempty"`
Visibility string `json:"visibility"`
StreamStatus *string `json:"stream_status,omitempty"` // Only for streams
Data json.RawMessage `json:"data"`
CreatedBy string `json:"created_by"`
UpdatedAt string `json:"updated_at"`
CreatedAt string `json:"created_at"`
}
// Transform function
func toParticleResponse(p *particle.Particle) ParticleResponse {
return ParticleResponse{
ID: p.ID,
Type: string(p.Type),
NetworkID: p.NetworkID,
ParentID: p.ParentID,
Visibility: string(p.Visibility),
StreamStatus: (*string)(p.StreamStatus),
Data: p.Data,
CreatedBy: p.CreatedByEmail,
UpdatedAt: p.UpdatedAt.Format(time.RFC3339),
CreatedAt: p.CreatedAt.Format(time.RFC3339),
}
}
```
---
## Particle Types
| Type | Purpose | Required Data Fields |
|------|---------|---------------------|
| `stream` | Temporal container (chat-like) | `title` |
| `folder` | Structural container | `name` |
| `media` | Images, video, audio, clips | `url`, `mime_type` |
| `file` | Documents, PDFs, attachments | `url`, `filename` |
| `text` | Quick text messages | `content` |
| `quest` | Tasks/requests | `title` |
| `paper` | Rich documents | `title` |
| `think` | AI chat container | `title` |
**Note:** Quest with `assigned_to = current_user` is presented as a "request" in UI.
---
## Database Schema
### Migration 1: Network Capacity
```sql
-- migrations/000003_network_capacity.up.sql
ALTER TABLE networks
ADD COLUMN open_stream_capacity INTEGER NOT NULL DEFAULT 5,
ADD COLUMN open_stream_count INTEGER NOT NULL DEFAULT 0;
```
```sql
-- migrations/000003_network_capacity.down.sql
ALTER TABLE networks
DROP COLUMN open_stream_capacity,
DROP COLUMN open_stream_count;
```
### Migration 2: Particles
```sql
-- migrations/000004_particles.up.sql
CREATE TYPE particle_type AS ENUM (
'stream', 'folder', 'media', 'file', 'text', 'quest', 'paper', 'think'
);
CREATE TYPE visibility_mode AS ENUM ('network_all', 'custom');
CREATE TABLE particles (
id TEXT PRIMARY KEY,
type particle_type NOT NULL,
network_id TEXT NOT NULL, -- NO FK constraint, loose coupling
parent_id TEXT, -- NO FK constraint, loose coupling
created_by_email VARCHAR(255) NOT NULL,
visibility visibility_mode NOT NULL DEFAULT 'network_all',
-- Stream-specific (NULL for non-streams)
stream_status VARCHAR(20) CHECK (stream_status IN ('open', 'closed')),
-- Type-specific data
data JSONB NOT NULL DEFAULT '{}',
-- Timestamps
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT stream_status_check CHECK (
(type = 'stream' AND stream_status IS NOT NULL) OR
(type != 'stream' AND stream_status IS NULL)
)
);
CREATE TABLE particle_members (
particle_id TEXT NOT NULL, -- NO FK constraint
email VARCHAR(255) NOT NULL,
added_at TIMESTAMPTZ DEFAULT NOW(),
PRIMARY KEY (particle_id, email)
);
-- Indexes for query performance
CREATE INDEX idx_particles_parent_updated ON particles(parent_id, updated_at DESC);
CREATE INDEX idx_particles_network_root ON particles(network_id, updated_at DESC) WHERE parent_id IS NULL;
CREATE INDEX idx_particles_open_streams ON particles(network_id) WHERE type = 'stream' AND stream_status = 'open';
CREATE INDEX idx_particle_members_email ON particle_members(email, particle_id);
CREATE INDEX idx_particles_network_id ON particles(network_id);
```
```sql
-- migrations/000004_particles.down.sql
DROP TABLE IF EXISTS particle_members;
DROP TABLE IF EXISTS particles;
DROP TYPE IF EXISTS visibility_mode;
DROP TYPE IF EXISTS particle_type;
```
---
## Service Layer Design
### Package Structure
```
internal/particle/
├── models.go # Particle struct, type constants, data structs
├── errors.go # ErrNotFound, ErrCapacityExceeded, ErrAccessDenied
├── repository.go # Database operations (internal)
├── service.go # Public Service interface + implementation
├── validation.go # Type-specific validation (simple required fields)
└── service_test.go # Integration tests
```
### Domain Models
```go
// internal/particle/models.go
type ParticleType string
const (
TypeStream ParticleType = "stream"
TypeFolder ParticleType = "folder"
TypeMedia ParticleType = "media"
TypeFile ParticleType = "file"
TypeText ParticleType = "text"
TypeQuest ParticleType = "quest"
TypePaper ParticleType = "paper"
TypeThink ParticleType = "think"
)
type VisibilityMode string
const (
VisibilityNetworkAll VisibilityMode = "network_all"
VisibilityCustom VisibilityMode = "custom"
)
type StreamStatus string
const (
StreamOpen StreamStatus = "open"
StreamClosed StreamStatus = "closed"
)
type Particle struct {
ID string
Type ParticleType
NetworkID string
ParentID *string
CreatedByEmail string
Visibility VisibilityMode
StreamStatus *StreamStatus // Only for type=stream
Data json.RawMessage
UpdatedAt time.Time
CreatedAt time.Time
}
type CreateInput struct {
Type ParticleType
NetworkID string
ParentID *string
Visibility VisibilityMode
Data json.RawMessage
MemberEmails []string // For custom visibility
}
type ListFilter struct {
Types []ParticleType
StreamStatus *StreamStatus
}
type Cursor struct {
Position string // particle ID or timestamp
Direction string // "before" | "after"
}
type ParticleList struct {
Parent *Particle // The parent particle (nil if root level)
Particles []*Particle
HasMore bool
NextCursor *Cursor // For loading more in same direction
PrevCursor *Cursor // For bidirectional (streams)
}
```
### Service Interface
```go
// internal/particle/service.go
type Service interface {
// Core CRUD
// Note: Caller (handler) is responsible for verifying network membership
// Service handles particle-level visibility filtering
Create(ctx context.Context, input CreateInput, creatorEmail string) (*Particle, error)
GetByID(ctx context.Context, id, requesterEmail string) (*Particle, error)
Update(ctx context.Context, id string, data json.RawMessage, requesterEmail string) (*Particle, error)
Delete(ctx context.Context, id, requesterEmail string) error
// Unified listing - file explorer style
// - parentID = nil → root particles of network
// - parentID = "p_123" → children of that particle
// - Automatically filters by visibility
// - Streams: sorted by updated_at DESC (activity-based)
// - Folders: sorted by created_at (client can re-sort by type)
ListParticles(ctx context.Context, networkID string, parentID *string, requesterEmail string, filter ListFilter, cursor *Cursor) (*ParticleList, error)
// Stream lifecycle
OpenStream(ctx context.Context, id, requesterEmail string) error
CloseStream(ctx context.Context, id, requesterEmail string) error
// Returns current open stream count for a network (for capacity check)
GetOpenStreamCount(ctx context.Context, networkID string) (int, error)
// Membership (for custom visibility)
SetVisibility(ctx context.Context, id string, mode VisibilityMode, requesterEmail string) error
AddMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
RemoveMembers(ctx context.Context, id string, emails []string, requesterEmail string) error
GetMembers(ctx context.Context, id string) ([]string, error)
}
```
### ListParticles Implementation Logic
```go
func (s *service) ListParticles(ctx context.Context, networkID string, parentID *string, email string, filter ListFilter, cursor *Cursor) (*ParticleList, error) {
var parent *Particle
var sortBy string
// 1. Determine parent and sort strategy
if parentID != nil {
var err error
parent, err = s.repo.getByID(ctx, *parentID)
if err != nil {
return nil, ErrNotFound
}
// Check visibility access to parent
if !s.canAccess(ctx, *parentID, email) {
return nil, ErrAccessDenied
}
// Sort based on parent type
if parent.Type == TypeStream {
sortBy = "updated_at DESC" // Activity-based for streams
} else {
sortBy = "created_at DESC" // Chronological for folders
}
} else {
sortBy = "updated_at DESC" // Root level: activity-based
}
// 2. Fetch particles with visibility filtering
particles, hasMore, nextCursor, prevCursor := s.repo.listChildren(ctx, networkID, parentID, email, sortBy, filter, cursor)
return &ParticleList{
Parent: parent,
Particles: particles,
HasMore: hasMore,
NextCursor: nextCursor,
PrevCursor: prevCursor,
}, nil
}
```
### Simple Validation (Start Simple)
```go
// internal/particle/validation.go
func validateData(t ParticleType, data json.RawMessage) error {
switch t {
case TypeStream, TypeQuest, TypePaper, TypeThink:
return requireField(data, "title")
case TypeFolder:
return requireField(data, "name")
case TypeMedia, TypeFile:
return requireFields(data, "url", "mime_type")
case TypeText:
return requireField(data, "content")
default:
return ErrInvalidParticleType
}
}
func requireField(data json.RawMessage, field string) error {
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return fmt.Errorf("invalid data JSON: %w", err)
}
if _, ok := m[field]; !ok {
return fmt.Errorf("missing required field: %s", field)
}
return nil
}
```
---
## Particle Visibility Logic
The particle service handles visibility filtering. It does **not** check network membership (handler does that).
### For `visibility = 'network_all'`
- All network members can see it
- Since handler already verified network membership, service returns it
### For `visibility = 'custom'`
- Only users in `particle_members` table can see it
- Service checks `particle_members` table
### Inheritance Rule
- Children can only **restrict** access, not expand
- If parent has `custom` visibility, child must also be `custom` (or more restrictive)
- Creating a child with `network_all` under a `custom` parent → error
### Access Check Query
```sql
-- Check if user can access a specific particle
-- Walks up the ancestor chain, verifies access at each level
WITH RECURSIVE ancestors AS (
SELECT id, parent_id, visibility
FROM particles
WHERE id = $1
UNION ALL
SELECT p.id, p.parent_id, p.visibility
FROM particles p
JOIN ancestors a ON p.id = a.parent_id
)
SELECT bool_and(
CASE
WHEN visibility = 'network_all' THEN true -- Handler already verified network membership
WHEN visibility = 'custom' THEN (
EXISTS (SELECT 1 FROM particle_members pm
WHERE pm.particle_id = ancestors.id AND pm.email = $2)
)
END
) AS has_access
FROM ancestors;
```
---
## Network Service Updates
### New Methods (no changes to coupling)
```go
// In internal/network/service.go
type Service interface {
// ... existing methods ...
// Admin capacity management
SetOpenStreamCapacity(ctx context.Context, networkID string, capacity int) error
GetCapacityInfo(ctx context.Context, networkID string) (capacity int, current int, err error)
// Stream count updates (called by handler, not particle service)
IncrementOpenStreamCount(ctx context.Context, networkID string) error
DecrementOpenStreamCount(ctx context.Context, networkID string) error
}
```
### Capacity Enforcement (in Handler)
```go
// Handler: Opening a stream
func (h *Handler) OpenStream(w http.ResponseWriter, r *http.Request) {
// ... auth and validation ...
// 1. Get particle to find its network
particle, err := h.particleService.GetByID(ctx, particleID, email)
// 2. Check capacity
capacity, current, err := h.networkService.GetCapacityInfo(ctx, particle.NetworkID)
if current >= capacity {
return Error("stream capacity exceeded")
}
// 3. Open the stream
err = h.particleService.OpenStream(ctx, particleID, email)
// 4. Increment counter
err = h.networkService.IncrementOpenStreamCount(ctx, particle.NetworkID)
}
```
---
## Files to Modify
| File | Changes |
|------|---------|
| `internal/network/models.go` | Add `OpenStreamCapacity`, `OpenStreamCount` fields |
| `internal/network/repository.go` | Add capacity CRUD methods |
| `internal/network/service.go` | Add `SetOpenStreamCapacity`, `GetCapacityInfo`, counter methods |
| `internal/handler/handler.go` | Wire up particle endpoints, add admin capacity endpoint |
| `migrations/` | Add 000003 and 000004 migration files |
## Files to Create
| File | Purpose |
|------|---------|
| `internal/particle/models.go` | Particle struct, type constants |
| `internal/particle/errors.go` | Domain errors |
| `internal/particle/repository.go` | Database operations |
| `internal/particle/service.go` | Business logic + visibility filtering |
| `internal/particle/validation.go` | Simple required field validation |
| `internal/particle/service_test.go` | Integration tests |
---
## Implementation Phases
### Phase 1: Network Capacity
1. Create migration `000003_network_capacity`
2. Update network models with capacity fields
3. Update network repository with capacity methods
4. Update network service with `SetOpenStreamCapacity`, `GetCapacityInfo`
5. Add admin endpoint in handler
### Phase 2: Core Particle CRUD
1. Create migration `000004_particles`
2. Create particle package with models, errors
3. Implement repository with basic CRUD
4. Implement simple validation
5. Write integration tests
### Phase 3: Particle Visibility
1. Implement visibility filtering in list queries
2. Implement access check for GetByID
3. Implement membership management (AddMembers, RemoveMembers)
4. Test visibility scenarios
### Phase 4: Stream Lifecycle
1. Implement `OpenStream`/`CloseStream` in particle service
2. Wire up capacity checks in handler
3. Test capacity enforcement
### Phase 5: Unified ListParticles
1. Implement unified `ListParticles` method
2. Add parent-type-aware sorting (streams: updated_at, folders: created_at)
3. Implement bidirectional cursor pagination
4. Include parent in response for breadcrumbs
### Phase 6: Handler Integration
1. Wire particle service to handler
2. Implement all particle endpoints
3. Add DTO transformations
4. End-to-end testing
---
## Verification Plan
1. **Integration tests:** Full flow with test database (following existing pattern in `network/service_test.go`)
2. **Manual testing:**
- Create network with capacity 2
- Open 2 streams → succeeds
- Open 3rd stream → fails with capacity error
- Close a stream → can open new one
- Create nested particles, verify visibility inheritance
- Add custom members, verify restricted access
- Test handler data flow end-to-end