golib/monitor/pool

Pool Package

Monitor pool management for coordinating multiple health monitors with batch operations and metrics aggregation.

AI Disclaimer: AI tools are used solely to assist with testing, documentation, and bug fixes under human supervision, in compliance with EU AI Act Article 50.4.

---

Overview

The pool package provides management capabilities for coordinating multiple monitors as a group. It enables batch lifecycle operations, aggregated metrics collection, Prometheus integration, and operational control through shell commands.

Key Features

• Monitor Management: Add, remove, update, and query monitors by name
• Batch Operations: Start, stop, and restart all monitors as a unit
• Metrics Aggregation: Collect and export metrics from all monitors
• Prometheus Integration: Built-in support for metrics registration and collection
• Shell Commands: CLI-style commands for operational control (list, info, start, stop, restart, status)
• Thread-Safe: All operations are safe for concurrent access

---

Installation

go get github.com/nabbar/golib/monitor/pool

---

Quick Start

import (
    "context"
    "time"
    "github.com/nabbar/golib/monitor/pool"
)

// Create pool
ctx := context.Background()
p := pool.New(func() context.Context { return ctx })

// Add monitors
p.MonitorAdd(createDatabaseMonitor())
p.MonitorAdd(createAPIMonitor())
p.MonitorAdd(createCacheMonitor())

// Register Prometheus metrics
err := p.RegisterMetrics(prometheusFunc, loggerFunc)
if err != nil {
    panic(err)
}
defer p.UnregisterMetrics()

// Start all monitors
if err := p.Start(ctx); err != nil {
    panic(err)
}
defer p.Stop(ctx)

// Trigger periodic metrics collection
go p.TriggerCollectMetrics(ctx, 30*time.Second)

// Query individual monitor
mon := p.MonitorGet("database")
fmt.Printf("DB Status: %s\n", mon.Status())

// List all monitors
names := p.MonitorList()
fmt.Printf("Monitors: %v\n", names)

---

Architecture

Component Structure

pool/
├── interface.go     # Pool interface definition
├── model.go         # Pool state management
├── pool.go          # Monitor CRUD operations
├── server.go        # Lifecycle (start/stop/restart)
├── metrics.go       # Prometheus metrics integration
├── shell.go         # Shell commands implementation
└── encode.go        # Encoding support (JSON, Text)

Data Flow

┌──────────────────────────────────────────┐
│              Pool                         │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  │
│  │Monitor 1│  │Monitor 2│  │Monitor N│  │
│  └────┬────┘  └────┬────┘  └────┬────┘  │
└───────┼────────────┼────────────┼────────┘
        │            │            │
        ▼            ▼            ▼
   Health Check   Health Check  Health Check
        │            │            │
        └────────────┴────────────┘
                     │
                     ▼
        ┌─────────────────────────┐
        │  Prometheus Metrics      │
        │  - Latency per monitor   │
        │  - Uptime per monitor    │
        │  - Status per monitor    │
        └─────────────────────────┘

---

API Reference

Pool Interface

type Pool interface {
    // Monitor Management
    MonitorAdd(mon Monitor) error
    MonitorGet(name string) Monitor
    MonitorSet(mon Monitor) error
    MonitorDel(name string)
    MonitorList() []string
    MonitorWalk(func(name string, mon Monitor) bool, validName ...string)
    
    // Lifecycle
    Start(ctx context.Context) error
    Stop(ctx context.Context) error
    Restart(ctx context.Context) error
    IsRunning() bool
    Uptime() time.Duration
    
    // Metrics
    RegisterMetrics(prom FuncGetPrometheus, log FuncLog) error
    UnregisterMetrics()
    RegisterFctProm(prom FuncGetPrometheus)
    RegisterFctLogger(log FuncLog)
    TriggerCollectMetrics(ctx context.Context, interval time.Duration)
    
    // Shell Commands
    GetShellCommand(ctx context.Context) []Command
    
    // Encoding
    MarshalText() ([]byte, error)
    MarshalJSON() ([]byte, error)
}

---

Usage Examples

Basic Pool Management

// Create pool
pool := pool.New(contextFunc)

// Add monitors
db := createDatabaseMonitor("postgres")
api := createAPIMonitor("user-service")
cache := createCacheMonitor("redis")

pool.MonitorAdd(db)
pool.MonitorAdd(api)
pool.MonitorAdd(cache)

// Start all
if err := pool.Start(ctx); err != nil {
    log.Fatal(err)
}
defer pool.Stop(ctx)

// Query status
for _, name := range pool.MonitorList() {
    mon := pool.MonitorGet(name)
    log.Printf("%s: %s\n", name, mon.Status())
}

Prometheus Integration

import "github.com/nabbar/golib/prometheus"

// Create Prometheus instance
prom := prometheus.New(contextFunc)

// Register metrics
err := pool.RegisterMetrics(
    func() prometheus.Prometheus { return prom },
    loggerFunc,
)
if err != nil {
    panic(err)
}

// Always cleanup
defer pool.UnregisterMetrics()

// Trigger periodic collection
go pool.TriggerCollectMetrics(ctx, 30*time.Second)

// Metrics are exported as:
// monitor_latency{monitor="postgres"}
// monitor_uptime{monitor="postgres"}
// monitor_status{monitor="postgres"}
// (and similar for each monitor)

Shell Commands

// Get shell commands
commands := pool.GetShellCommand(ctx)

// Commands available:
// 0. list    - List all monitors
// 1. info    - Show monitor information
// 2. start   - Start monitors
// 3. stop    - Stop monitors
// 4. restart - Restart monitors
// 5. status  - Show monitor status

// Execute list command
var stdout bytes.Buffer
commands[0].Run(&stdout, nil, []string{})
fmt.Println(stdout.String())

// Execute status for specific monitor
stdout.Reset()
commands[5].Run(&stdout, nil, []string{"postgres"})
fmt.Println(stdout.String())

// Execute status for all monitors
stdout.Reset()
commands[5].Run(&stdout, nil, []string{})
fmt.Println(stdout.String())

Dynamic Monitor Management

// Add monitor to running pool
if pool.IsRunning() {
    newMon := createMonitor("new-service")
    pool.MonitorAdd(newMon)  // Automatically starts if pool is running
}

// Update monitor configuration
mon := pool.MonitorGet("postgres")
mon.SetConfig(ctx, newConfig)
pool.MonitorSet(mon)

// Remove monitor
pool.MonitorDel("old-service")

// Walk through monitors
pool.MonitorWalk(func(name string, mon Monitor) bool {
    log.Printf("%s: %s (uptime: %s)\n", name, mon.Status(), mon.Uptime())
    return true  // Continue iteration
})

Batch Operations

// Start all monitors
if err := pool.Start(ctx); err != nil {
    log.Printf("Failed to start some monitors: %v\n", err)
    // Individual monitor errors are aggregated
}

// Stop all monitors
if err := pool.Stop(ctx); err != nil {
    log.Printf("Failed to stop some monitors: %v\n", err)
}

// Restart all monitors
if err := pool.Restart(ctx); err != nil {
    log.Printf("Failed to restart some monitors: %v\n", err)
}

// Check if any monitor is running
if pool.IsRunning() {
    log.Println("At least one monitor is running")
}

// Get pool uptime (longest monitor uptime)
uptime := pool.Uptime()
log.Printf("Pool uptime: %s\n", uptime)

---

Metrics Collection

Available Metrics

Per-monitor metrics exported to Prometheus:

Metric	Type	Labels	Description
monitor_latency	Histogram	monitor	Health check duration
monitor_uptime	Gauge	monitor	Total uptime in seconds
monitor_downtime	Gauge	monitor	Total downtime in seconds
monitor_risetime	Gauge	monitor	Time in rising state
monitor_falltime	Gauge	monitor	Time in falling state
monitor_status	Gauge	monitor	Current status (0=KO, 0.5=Warn, 1=OK)
monitor_rise	Gauge	monitor	Rising flag (0 or 1)
monitor_fall	Gauge	monitor	Falling flag (0 or 1)
monitor_sli	Gauge	monitor	Service Level Indicator (0-1)

Metric Names

Metrics are prefixed with a customizable name:

// Default naming
monitor_latency{monitor="postgres"}

// Custom naming (set in individual monitors)
mon.RegisterMetricsName("database_health")
// Results in: database_health_latency{monitor="postgres"}

---

Shell Commands

Available Commands

1. list: List all monitors

   list
   # Output: postgres, redis, user-service
   

2. info: Show monitor information

   info postgres
   # Output: OK: postgres (version: 14.5) | 5ms / 1h / 0s
   
   info
   # Output: (info for all monitors)
   

3. start: Start monitors

   start postgres
   # Starts specific monitor
   
   start
   # Starts all monitors
   

4. stop: Stop monitors

   stop postgres
   stop  # All monitors
   

5. restart: Restart monitors

   restart postgres
   restart  # All monitors
   

6. status: Show monitor status

   status postgres
   # Output: postgres: OK
   
   status
   # Output: (status for all monitors)
   

Integration Example

// CLI tool integration
func runCommand(pool Pool, cmdName string, args []string) {
    commands := pool.GetShellCommand(context.Background())
    
    // Find command by name
    var cmd Command
    for _, c := range commands {
        if c.Name() == cmdName {
            cmd = c
            break
        }
    }
    
    if cmd == nil {
        fmt.Printf("Unknown command: %s\n", cmdName)
        return
    }
    
    // Execute
    var stdout, stderr bytes.Buffer
    cmd.Run(&stdout, &stderr, args)
    
    fmt.Print(stdout.String())
    if stderr.Len() > 0 {
        fmt.Fprint(os.Stderr, stderr.String())
    }
}

// Usage
runCommand(pool, "list", nil)
runCommand(pool, "status", []string{"postgres"})

---

Performance

Benchmarks

Measured on: AMD Ryzen 9 5950X, 64GB RAM, Go 1.21

Operation	Time	Memory	Notes
Pool Creation	2.5 µs	500 B	One-time cost
MonitorAdd	1.8 µs	150 B	Per monitor
MonitorGet	45 ns	0 B	Lock-free read
MonitorList	2.1 µs	240 B	Iteration
Start (10 monitors)	85 µs	8 KB	Parallel start
Stop (10 monitors)	42 µs	2 KB	Parallel stop
Metrics Collection	25 µs	1 KB	All monitors

Scalability

• Monitors per pool: Tested up to 1000 monitors
• Concurrent operations: Lock-free reads, synchronized writes
• Memory overhead: ~500B base + (monitors × 150B)
• Startup time: O(n) where n = number of monitors (parallel)

---

Best Practices

1. Pool Lifecycle

// DO: Always cleanup
pool := pool.New(ctxFunc)
err := pool.RegisterMetrics(promFunc, logFunc)
if err != nil {
    return err
}
defer pool.UnregisterMetrics()  // Cleanup Prometheus metrics

if err := pool.Start(ctx); err != nil {
    return err
}
defer pool.Stop(ctx)  // Stop all monitors

// DON'T: Forget cleanup
pool := pool.New(ctxFunc)
pool.RegisterMetrics(promFunc, logFunc)
pool.Start(ctx)
// Missing UnregisterMetrics() and Stop()

2. Monitor Management

// DO: Check if monitor exists before operations
if mon := pool.MonitorGet("postgres"); mon != nil {
    // Use monitor
}

// DO: Handle errors from batch operations
if err := pool.Start(ctx); err != nil {
    log.Printf("Some monitors failed to start: %v\n", err)
}

// DON'T: Assume operations always succeed
pool.MonitorGet("nonexistent").Status()  // Panic!
pool.Start(ctx)  // Ignore errors

3. Metrics Registration

// DO: Register early, unregister on cleanup
pool.RegisterMetrics(promFunc, logFunc)
defer pool.UnregisterMetrics()

// DO: Use separate function for periodic collection
go pool.TriggerCollectMetrics(ctx, 30*time.Second)

// DON'T: Register multiple times without unregistering
pool.RegisterMetrics(promFunc, logFunc)
pool.RegisterMetrics(promFunc, logFunc)  // Duplicate registration error

4. Concurrent Access

// DO: Pool is thread-safe, use freely
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
    wg.Add(1)
    go func(idx int) {
        defer wg.Done()
        pool.MonitorGet(fmt.Sprintf("mon-%d", idx))
    }(i)
}
wg.Wait()

// DO: But be careful with monitor instances
mon := pool.MonitorGet("postgres")
// Don't share 'mon' across goroutines without synchronization
// Use pool.MonitorGet() in each goroutine instead

---

Testing

Test Coverage

Current coverage: 76.7%

Run tests:

# All tests
go test ./...

# With coverage
go test -coverprofile=coverage.out ./...

# With race detection
CGO_ENABLED=1 go test -race ./...

# Coverage script
./test_coverage.sh --html

Test Files

• pool_test.go: Core functionality
• pool_metrics_test.go: Metrics collection (extended coverage)
• pool_shell_test.go: Shell command execution
• pool_coverage_test.go: Edge cases and error paths
• pool_errors_test.go: Error handling scenarios

See ../TESTING.md for detailed testing documentation.

---

Contributing

See ../README.md#contributing for contribution guidelines.

AI Usage Reminder: Do not use AI to generate package code. AI may assist with tests, documentation, and bug fixes only.

---

API Migration

Deprecated Methods

For backward compatibility, deprecated methods are still available:

// Old API (deprecated)
pool.InitMetrics(promFunc, logFunc)
defer pool.ShutDown()

// New API (recommended)
pool.RegisterMetrics(promFunc, logFunc)
defer pool.UnregisterMetrics()

The deprecated methods will be removed in a future version. Please migrate to the new API.

---

License

MIT License - Copyright (c) 2022 Nicolas JUHEL

---

Resources

• Parent Package: github.com/nabbar/golib/monitor
• GoDoc: pkg.go.dev/github.com/nabbar/golib/monitor/pool
• Coverage Report: Run ./test_coverage.sh --html