golib/ioutils

IOUtils Package

Production-ready I/O utilities collection providing specialized tools for stream processing, resource management, progress tracking, and concurrent I/O operations with comprehensive testing and thread-safe implementations.

---

Table of Contents

• Overview
  • Design Philosophy
  • Key Features
• Architecture
  • Package Organization
• Performance
  • Benchmark Summary
  • Coverage Statistics
• Subpackages
  • aggregator
  • bufferReadCloser
  • delim
  • fileDescriptor
  • ioprogress
  • iowrapper
  • mapCloser
  • maxstdio
  • multi
  • nopwritecloser
• Root-Level Utilities
  • PathCheckCreate
• Use Cases
• Quick Start
  • Installation
  • Basic Examples
• Best Practices
• Testing
• Contributing
• Improvements & Security
• Resources
• AI Transparency
• License

---

Overview

The ioutils package is a comprehensive collection of I/O utilities for Go applications, providing 10 specialized subpackages plus root-level helper functions. Each subpackage addresses specific I/O challenges encountered in production environments: concurrent write aggregation, stream multiplexing, progress tracking, resource lifecycle management, and more.

Design Philosophy

1. Standard Interface Compliance: All implementations conform to Go's io package interfaces (Reader, Writer, Closer, ReadCloser, WriteCloser)
2. Thread Safety First: Atomic operations, proper locking, and race-free implementations verified with -race detector
3. Streaming-Oriented: Designed for continuous data flow with constant memory usage, not batch processing
4. Resource Management: Proper cleanup with defer, context cancellation support, and automatic lifecycle handling
5. Zero External Dependencies: Only standard library and internal golib packages
6. Production Hardened: 776 test specs, 90.4% average coverage, extensive documentation

Key Features

✅ Thread-Safe Operations: All subpackages safe for concurrent use
✅ Context Integration: Cancellation and deadline propagation throughout
✅ Comprehensive Testing: 776 specs, zero race conditions
✅ Rich Subpackage Ecosystem: 10 specialized packages for different I/O needs
✅ High Performance: Benchmarked and optimized for throughput and latency
✅ Well Documented: GoDoc comments, examples, README per subpackage

---

Architecture

Package Organization

ioutils/
├── tools.go                    Root-level utilities (PathCheckCreate)
│
├── aggregator/                 Thread-safe write aggregator that buffers and serializes concurrent writes
│   └── [115 specs, 84.9% coverage]
│
├── bufferReadCloser/           io.Closer wrappers for bytes.Buffer and bufio types
│   └── [44 specs, 100% coverage]
│
├── delim/                      Buffered reader for reading delimiter-separated data streams
│   └── [95 specs, 98.6% coverage]
│
├── fileDescriptor/             Cross-platform file descriptor limit management
│   └── [28 specs, 85.7% coverage]
│
├── ioprogress/                 Thread-safe I/O progress tracking wrappers
│   └── [54 specs, 84.7% coverage]
│
├── iowrapper/                  Flexible I/O wrapper for customization and interception
│   └── [88 specs, 100% coverage]
│
├── mapCloser/                  Thread-safe, context-aware manager for multiple io.Closer instances
│   └── [82 specs, 80.8% coverage]
│
├── maxstdio/                   Standard I/O limit management
│   └── [No specs - utility package]
│
├── multi/                      Thread-safe adaptive multi-writer extending io.MultiWriter
│   └── [112 specs, 80.8% coverage]
│
└── nopwritecloser/             io.WriteCloser wrapper with no-op Close()
    └── [54 specs, 100% coverage]

Total: 776 test specs, 90.4% average coverage

---

Performance

Benchmark Summary

Based on actual test execution (776 specs, ~41 seconds total):

Package	Specs	Coverage	Execution Time	Notable Metrics
aggregator	115	84.9%	~33.3s	Start: 10.7ms, Throughput: 5000-10000/s
bufferReadCloser	44	100%	~0.03s	Read: <1ms, Buffer: configurable
delim	95	98.6%	~6.68s	Read: <500µs, Scan: <1ms
fileDescriptor	28	85.7%	~0.01s	Limit check: <1µs
ioprogress	54	84.7%	~0.02s	Callback: <10µs overhead
iowrapper	88	100%	~0.08s	Wrap: <1µs, Pass-through
mapCloser	82	80.8%	~0.05s	Add/Remove: <1µs, Close all: <1ms
multi	112	80.8%	~0.69s	Write to N: O(N), Copy: <100µs
nopwritecloser	54	100%	~0.24s	No-op: <1ns
ioutils (root)	31	88.2%	~0.04s	PathCheckCreate: varies

Aggregate Performance:

• Total execution: ~41 seconds (including setup/teardown)
• Average spec time: ~53ms (dominated by aggregator's timing tests)
• Zero race conditions: All tests pass with -race detector
• Memory efficiency: Constant memory for streaming operations

Coverage Statistics

Overall Coverage:       90.4% (weighted average)
Packages at 100%:       3/10 (bufferReadCloser, iowrapper, nopwritecloser)
Packages >85%:          5/10
Lowest Coverage:        80.8% (mapCloser, multi - primarily error paths)

Coverage Breakdown:

Coverage Range	Count	Packages
100%	3	bufferReadCloser, iowrapper, nopwritecloser
85-99%	4	delim (98.6%), ioutils (88.2%), fileDescriptor (85.7%), ioprogress (84.7%)
80-89%	3	aggregator (84.9%), mapCloser (80.8%), multi (80.8%)

---

Subpackages

aggregator

Purpose: Thread-safe write aggregator that buffers and serializes concurrent write operations to a single writer function.

Key Features:

• Concurrent writes from multiple goroutines
• Configurable buffer for backpressure handling
• Optional periodic callbacks (async/sync)
• Real-time metrics (count and size based)
• Context integration for lifecycle management

Performance:

• Throughput: 5,000-10,000 writes/second (depends on FctWriter)
• Latency: Start 10.7ms, Stop 12.1ms, Write <1ms
• Metrics read: <5µs

Use Case: Socket server logging, database write pooling, network stream multiplexing

Documentation: aggregator/README.md

---

bufferReadCloser

Purpose: Lightweight wrappers around Go's standard buffered I/O types (bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter) that add io.Closer support with automatic resource cleanup and custom close callbacks.

Key Features:

• Wraps bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter
• Adds io.Closer interface to non-closeable buffers
• Automatic reset/flush on close
• Optional custom close callbacks
• Zero-copy passthrough to underlying buffers

Performance:

• Wrapper overhead: ~5.7ms per 10,000 operations
• I/O operations: 0-100ns (direct delegation)
• Memory: 24 bytes per wrapper

Use Case: Adding lifecycle management to standard buffers, automatic cleanup with defer, composable close operations

Documentation: bufferReadCloser/README.md

---

delim

Purpose: Buffered reader for reading delimiter-separated data streams.

Key Features:

• Read until any delimiter character
• Buffered scanning for efficiency
• Handles any byte delimiter
• Line and token reading

Performance:

• Read latency: <500µs
• Scan latency: <1ms
• Memory: constant per buffer

Use Case: CSV parsing, log file processing, protocol parsing

Documentation: delim/README.md

---

fileDescriptor

Purpose: Cross-platform utilities for managing file descriptor limits, offering a unified API for querying and modifying the maximum number of open files or I/O resources allowed for a process.

Key Features:

• Check system-wide FD limits
• Validate current usage
• Preemptive resource checks
• Cross-platform support

Performance:

• Limit check: <1µs
• No overhead on I/O operations

Use Case: High-concurrency servers, connection pooling, resource planning

Documentation: fileDescriptor/README.md

---

ioprogress

Purpose: Thread-safe I/O progress tracking wrappers for monitoring read and write operations in real-time through customizable callbacks.

Key Features:

• Reader/Writer progress callbacks
• Byte count tracking
• Real-time notification
• Progress bar integration

Performance:

• Callback overhead: <10µs per operation
• Thread-safe atomic counters
• Minimal allocation

Use Case: File uploads/downloads, progress bars, bandwidth monitoring, ETL pipelines

Documentation: ioprogress/README.md

---

iowrapper

Purpose: Flexible I/O wrapper that enables customization and interception of read, write, seek, and close operations on any underlying I/O object without modifying its implementation.

Key Features:

• Wrap readers and writers
• Add custom behavior
• Preserve interface semantics
• Zero-allocation pass-through

Performance:

• Wrap overhead: <1µs
• Pass-through: no measurable impact

Use Case: Logging wrappers, compression, encryption, transformation

Documentation: iowrapper/README.md

---

mapCloser

Purpose: Thread-safe, context-aware manager for multiple io.Closer instances.

Key Features:

• Manage multiple closers
• Automatic cleanup on context cancellation
• Error aggregation
• Add/remove closers dynamically

Performance:

• Add/Remove: <1µs
• Close all: <1ms (for moderate counts)
• Thread-safe operations

Use Case: Resource pools, connection management, cleanup coordination

Documentation: mapCloser/README.md

---

maxstdio

Purpose: Standard I/O (stdin/stdout/stderr) limit enforcement.

Key Features:

• Protect against excessive stdio usage
• Redirect overflow to alternatives
• Configurable thresholds

Use Case: Daemon processes, service wrappers, log management

Documentation: maxstdio/README.md

---

multi

Purpose: Thread-safe, adaptive multi-writer that extends Go's standard io.MultiWriter with advanced features including adaptive sequential/parallel execution, latency monitoring, and comprehensive concurrency support.

Key Features:

• Write to multiple writers atomically
• Dynamic writer addition/removal
• Error handling per writer
• Zero-allocation for single writer
• Copy to multiple outputs

Performance:

• Write to N writers: O(N) time
• Copy operation: <100µs
• Atomic operations

Use Case: Logging to multiple files, network fanout, tee operations

Documentation: multi/README.md

---

nopwritecloser

Purpose: Wrapper that implements io.WriteCloser for an io.Writer by adding a no-op Close() method.

Key Features:

• Implements io.WriteCloser for any io.Writer
• Write operations delegate to underlying writer unchanged
• Close() is no-op (always returns nil, never affects writer)
• Thread-safe if underlying writer is thread-safe

Performance:

• Write: Direct delegation (no overhead)
• Close: <1ns (no-op)
• Zero allocation

Use Case: Protecting stdout/stderr from closure, API compatibility (io.Writer → io.WriteCloser), testing with inspectable buffers after close, shared resource management

Documentation: nopwritecloser/README.md

---

Root-Level Utilities

PathCheckCreate

Function: PathCheckCreate(isFile bool, path string, permFile os.FileMode, permDir os.FileMode) error

Purpose: Ensures a file or directory exists with correct permissions.

Features:

• Creates files or directories as needed
• Creates parent directories automatically
• Validates and updates permissions
• Type checking (file vs directory)
• Atomic creation with proper error handling

Example:

// Ensure config directory exists
err := ioutils.PathCheckCreate(false, "/etc/app/config", 0644, 0755)

// Ensure log file exists
err := ioutils.PathCheckCreate(true, "/var/log/app.log", 0644, 0755)

Use Case: Application initialization, config file setup, log directory creation

---

Use Cases

1. High-Concurrency Logging

Problem: Multiple goroutines writing to a single log file (filesystem doesn't support concurrent writes).

Solution: Use aggregator to serialize writes.

logFile, _ := os.Create("app.log")
agg, _ := aggregator.New(ctx, aggregator.Config{
    BufWriter: 1000,
    FctWriter: func(p []byte) (int, error) {
        return logFile.Write(p)
    },
}, logger)

// All goroutines write through aggregator
for i := 0; i < 100; i++ {
    go func(id int) {
        agg.Write([]byte(fmt.Sprintf("[%d] Log message\n", id)))
    }(i)
}

2. Fan-Out Data Broadcasting

Problem: Send data to multiple destinations (files, network, stdout).

Solution: Use multi for write multiplexing.

mw := multi.New()
mw.AddWriter(os.Stdout)
mw.AddWriter(logFile)
mw.AddWriter(networkConn)

// One write reaches all destinations
mw.Write([]byte("Broadcast message\n"))

3. Upload Progress Tracking

Problem: Show upload progress to user during file transfer.

Solution: Use ioprogress wrapper.

file, _ := os.Open("large-file.dat")
progressReader := ioprogress.NewReader(file, func(bytes int64) {
    percent := float64(bytes) / float64(fileSize) * 100
    fmt.Printf("Uploaded: %.1f%%\r", percent)
})

http.Post(url, "application/octet-stream", progressReader)

4. Resource Management

Problem: Manage multiple connections/files with automatic cleanup.

Solution: Use mapCloser.

closer := mapcloser.New(ctx)

// Add resources
conn1, _ := net.Dial("tcp", "host1:port")
closer.Add("conn1", conn1)

conn2, _ := net.Dial("tcp", "host2:port")
closer.Add("conn2", conn2)

// Automatic cleanup on context cancel or explicit close
defer closer.Close()

5. Protocol Parsing

Problem: Parse delimited protocol messages efficiently.

Solution: Use delim scanner.

conn, _ := net.Dial("tcp", "server:port")
scanner := delim.NewScanner(conn, '\n')

for scanner.Scan() {
    message := scanner.Text()
    processMessage(message)
}

---

Quick Start

Installation

# Install entire package
go get github.com/nabbar/golib/ioutils

# Import specific subpackages as needed
import (
    "github.com/nabbar/golib/ioutils"
    "github.com/nabbar/golib/ioutils/aggregator"
    "github.com/nabbar/golib/ioutils/multi"
    "github.com/nabbar/golib/ioutils/ioprogress"
)

Basic Examples

Path Management:

import "github.com/nabbar/golib/ioutils"

// Create config directory
if err := ioutils.PathCheckCreate(false, "/etc/myapp", 0644, 0755); err != nil {
    log.Fatal(err)
}

// Create log file
if err := ioutils.PathCheckCreate(true, "/var/log/myapp.log", 0644, 0755); err != nil {
    log.Fatal(err)
}

Write Aggregation (aggregator):

import "github.com/nabbar/golib/ioutils/aggregator"

cfg := aggregator.Config{
    BufWriter: 100,
    FctWriter: func(p []byte) (int, error) {
        return logFile.Write(p)
    },
}

agg, _ := aggregator.New(ctx, cfg, logger)
agg.Start(ctx)
defer agg.Close()

// Write from multiple goroutines safely
for i := 0; i < 10; i++ {
    go func(id int) {
        agg.Write([]byte(fmt.Sprintf("Log from goroutine %d\n", id)))
    }(i)
}

Write Multiplexing (multi):

import "github.com/nabbar/golib/ioutils/multi"

// Create multi-writer
mw := multi.New()

// Add multiple destinations
file1, _ := os.Create("output1.txt")
file2, _ := os.Create("output2.txt")

mw.AddWriter(file1)
mw.AddWriter(file2)

// Write to both files at once
mw.Write([]byte("This goes to both files\n"))

Progress Tracking (ioprogress):

import "github.com/nabbar/golib/ioutils/ioprogress"

// Wrap reader with progress callback
reader := ioprogress.NewReader(file, func(bytes int64) {
    fmt.Printf("Read %d bytes so far\n", bytes)
})

// Read operations trigger callbacks
io.Copy(dest, reader)

Delimiter Reading (delim):

import "github.com/nabbar/golib/ioutils/delim"

scanner := delim.NewScanner(file, '\n')

for scanner.Scan() {
    line := scanner.Text()
    fmt.Println(line)
}

---

Best Practices

✅ DO

Choose the Right Subpackage:

// For concurrent writes → aggregator
// For broadcast writes → multi
// For progress tracking → ioprogress
// For resource management → mapCloser
// For delimiter parsing → delim

Proper Resource Cleanup:

// Always close resources
agg, _ := aggregator.New(ctx, cfg, logger)
defer agg.Close()

// Use context for lifecycle
ctx, cancel := context.WithCancel(parent)
defer cancel()

Buffer Sizing:

// Size buffers based on workload
cfg := aggregator.Config{
    BufWriter: writeRate * maxLatency * 1.5,  // Safety margin
    // ...
}

Error Handling:

// Check errors from I/O operations
if n, err := writer.Write(data); err != nil {
    log.Error("Write failed:", err)
    return err
}

Thread Safety:

// All subpackages are thread-safe
for i := 0; i < 10; i++ {
    go func() {
        agg.Write(data)  // Safe concurrent access
    }()
}

❌ DON'T

Don't Ignore Buffer Limits:

// ❌ BAD: No buffer with slow writer
cfg := aggregator.Config{
    BufWriter: 1,  // Will block immediately
    FctWriter: slowWriter,
}

// ✅ GOOD: Sized for throughput
cfg := aggregator.Config{
    BufWriter: 1000,
    FctWriter: slowWriter,
}

Don't Mix Interfaces:

// ❌ BAD: Writing directly bypasses aggregator
agg.Start(ctx)
logFile.Write(data)  // Bypasses serialization

// ✅ GOOD: All writes through aggregator
agg.Write(data)

Don't Forget Context:

// ❌ BAD: No cancellation
agg, _ := aggregator.New(context.Background(), cfg, logger)

// ✅ GOOD: Cancellable context
ctx, cancel := context.WithCancel(parent)
defer cancel()
agg, _ := aggregator.New(ctx, cfg, logger)

Don't Leave Resources Open:

// ❌ BAD: No cleanup
closer := mapcloser.New(ctx)
closer.Add("conn", conn)
// Program exits, resources leak

// ✅ GOOD: Explicit cleanup
defer closer.Close()

---

Testing

Comprehensive test suite with 776 specs across all subpackages.

Quick Test:

# Run all tests
go test ./...

# With race detector
CGO_ENABLED=1 go test -race ./...

# Coverage report
go test -cover ./...

Expected Results:

• 776 specs passed
• 90.4% average coverage
• Zero race conditions
• ~41 seconds execution time

See TESTING.md for detailed testing documentation including:

• Running tests (standard, race, coverage, profiling)
• Performance benchmarks per subpackage
• Writing new tests
• CI integration examples

---

Contributing

Contributions are welcome! Please follow these guidelines:

1. Code Quality

   • Follow Go best practices and idioms
   • Maintain or improve code coverage (target: >85%)
   • Pass all tests including race detector
   • Use gofmt and golint

2. AI Usage Policy

   • ❌ Do NOT use AI for implementing package functionality or core logic
   • ✅ AI may assist with:
     • Writing and improving tests
     • Documentation and comments
     • Debugging and troubleshooting
   • All AI-assisted contributions must be reviewed and validated by humans

3. Testing

   • Add tests for new features
   • Use Ginkgo v2 / Gomega for test framework
   • Ensure zero race conditions
   • Maintain coverage above 85%

4. Documentation

   • Update GoDoc comments for public APIs
   • Add examples for new features
   • Update README.md if adding subpackages
   • Update TESTING.md if changing test structure

5. Pull Request Process

   • Fork the repository
   • Create a feature branch
   • Write clear commit messages
   • Ensure all tests pass
   • Update documentation
   • Submit PR with description of changes

---

Improvements & Security

Current Status

The package is production-ready with no urgent improvements or security vulnerabilities identified across all subpackages.

Code Quality Metrics

• ✅ 90.4% average test coverage (target: >85%)
• ✅ Zero race conditions detected with -race flag
• ✅ Thread-safe implementations across all subpackages
• ✅ Memory-safe with proper resource cleanup
• ✅ Standard interfaces for maximum compatibility
• ✅ 776 comprehensive test specs ensuring reliability

Future Enhancements (Non-urgent)

The following enhancements could be considered for future versions:

New Subpackages:

1. iozip: Streaming compression/decompression wrappers
2. iocrypto: Encryption/decryption stream wrappers
3. ioratelimit: Bandwidth throttling and rate limiting
4. iocache: Write-through/write-back caching layers

Performance Optimizations:

1. SIMD-accelerated delimiter scanning (delim)
2. Lock-free queues for aggregator
3. Memory pool for buffer allocation
4. Zero-copy operations where possible

Monitoring Enhancements:

1. Prometheus metrics integration
2. OpenTelemetry tracing
3. Structured logging throughout
4. Performance profiling hooks

Advanced Features:

1. Async I/O support (io_uring on Linux)
2. Adaptive buffer sizing based on load
3. Priority queuing in aggregator
4. Circuit breaker patterns for reliability

These are optional improvements and not required for production use. The current implementation is stable, performant, and feature-complete for its intended use cases.

Suggestions and contributions are welcome via GitHub issues.

---

Resources

Internal Documentation

• GoDoc - Complete API documentation
• TESTING.md - Test suite documentation
• Individual subpackage READMEs (linked in Subpackages)

Related Packages

• github.com/nabbar/golib/logger - Logging interface used by subpackages
• github.com/nabbar/golib/runner/startStop - Lifecycle management
• github.com/nabbar/golib/socket/server - Socket server (uses aggregator)

External References

• Go io package - Standard library I/O interfaces
• Effective Go - Go best practices
• Go Concurrency Patterns - Official Go blog

---

AI Transparency

In compliance with EU AI Act Article 50.4: AI assistance was used for testing, documentation, and bug resolution under human supervision. All core functionality is human-designed and validated.

---

License

MIT License - See LICENSE file for details.

Copyright (c) 2025 Nicolas JUHEL

---

Maintained by: Nicolas JUHEL
Package: github.com/nabbar/golib/ioutils
Version: See releases for versioning