From 737f174177784489be49f7af6280c4c6a77b579b Mon Sep 17 00:00:00 2001 From: nabbar Date: Sun, 21 Dec 2025 17:34:27 +0100 Subject: [PATCH] Documentation: - FIX: package ioutils main documentation md & GoDoc - FIX: root documentation --- README.md | 24 +++---- ioutils/README.md | 162 ++++++++++++++++++---------------------------- ioutils/doc.go | 36 +++++------ 3 files changed, 92 insertions(+), 130 deletions(-) diff --git a/README.md b/README.md index 426a79d..50c62d2 100644 --- a/README.md +++ b/README.md @@ -349,17 +349,17 @@ The table below lists all 165 Go packages with their test coverage and links to | **file/bandwidth** | 77.8% | 25 | Bandwidth-limited I/O | [README](file/) | | **file/perm** | 88.9% | 141 | File permission utilities | - | | **file/progress** | 71.1% | 90 | File operation progress tracking | - | -| **ioutils** | 87.7% | 31 | I/O utilities base | [README](ioutils/) | -| **ioutils/aggregator** | 86.0% | 115 | Write operation aggregator | [README](ioutils/aggregator/) | -| **ioutils/bufferReadCloser** | 100% | 57 | Buffered reader with closer | [README](ioutils/bufferReadCloser/) | -| **ioutils/delim** | 100% | 198 | Delimiter-based stream processing | [README](ioutils/delim/) | -| **ioutils/fileDescriptor** | 85.7% | 23 | File descriptor management | [README](ioutils/fileDescriptor/) | -| **ioutils/ioprogress** | 84.7% | 42 | I/O progress tracking | [README](ioutils/ioprogress/) | -| **ioutils/iowrapper** | 100% | 114 | Generic I/O wrappers | [README](ioutils/iowrapper/) | -| **ioutils/mapCloser** | 77.5% | 29 | Multiple closer management | [README](ioutils/mapCloser/) | +| **ioutils** | 88.2% | 31 | I/O utilities base | [README](ioutils/) | +| **ioutils/aggregator** | 84.9% | 115 | Thread-safe write aggregator that buffers and serializes concurrent writes | [README](ioutils/aggregator/) | +| **ioutils/bufferReadCloser** | 100% | 44 | Lightweight wrappers adding io.Closer to bytes.Buffer and bufio types | [README](ioutils/bufferReadCloser/) | +| **ioutils/delim** | 98.6% | 95 | Delimiter-based stream processing | [README](ioutils/delim/) | +| **ioutils/fileDescriptor** | 85.7% | 28 | Cross-platform file descriptor limit management | [README](ioutils/fileDescriptor/) | +| **ioutils/ioprogress** | 84.7% | 54 | I/O progress tracking | [README](ioutils/ioprogress/) | +| **ioutils/iowrapper** | 100% | 88 | Generic I/O wrappers | [README](ioutils/iowrapper/) | +| **ioutils/mapCloser** | 80.8% | 82 | Thread-safe, context-aware manager for multiple io.Closer instances | [README](ioutils/mapCloser/) | | **ioutils/maxstdio** | No tests | - | Stdio limit management | [README](ioutils/maxstdio/) | -| **ioutils/multi** | 81.7% | 113 | Write multiplexing | [README](ioutils/multi/) | -| **ioutils/nopwritecloser** | 100% | 54 | No-op writer closer | [README](ioutils/nopwritecloser/) | +| **ioutils/multi** | 80.8% | 112 | Thread-safe adaptive multi-writer extending io.MultiWriter | [README](ioutils/multi/) | +| **ioutils/nopwritecloser** | 100% | 54 | Wrapper implementing io.WriteCloser with no-op Close() | [README](ioutils/nopwritecloser/) | ### Security & Communication @@ -652,7 +652,7 @@ Race Conditions: 0 (verified with -race flag) |----------------|-------|------------|------------------| | **100%** | 14 | 11.0% | errors/pool, logger/gorm, router/authheader, semaphore/sem | | **90-99%** | 24 | 18.9% | atomic (91.8%), version (93.8%), size (95.4%) | -| **80-89%** | 29 | 22.8% | ioutils (87.7%), mail/queuer (90.8%), static (82.6%), context (87.5%) | +| **80-89%** | 29 | 22.8% | ioutils (88.2%), mail/queuer (90.8%), static (82.6%), context (87.5%) | | **70-79%** | 19 | 15.0% | cobra (76.7%), viper (73.3%), file/bandwidth (77.8%) | | **60-69%** | 10 | 7.9% | config (61.9%), logger (68.0%), database/kvmap (66.7%) | | **<60%** | 31 | 24.4% | archive (8.6%), aws (5.4%), httpserver (52.5%) | @@ -671,7 +671,7 @@ Race Conditions: 0 (verified with -race flag) **Well-Tested Core Packages:** -- **ioutils**: 87.7% average across 10 subpackages, 772 specs +- **ioutils**: 90.4% average across 10 subpackages, 776 specs - **mail**: 89.0% average across 6 subpackages, 970 specs - **logger**: 74.7% base + 85%+ in config/entry, 573 specs - **errors**: 87.6% base + 100% pool, 305 specs diff --git a/ioutils/README.md b/ioutils/README.md index 5d4c0e3..9da1e7e 100644 --- a/ioutils/README.md +++ b/ioutils/README.md @@ -2,7 +2,7 @@ [![Go Version](https://img.shields.io/badge/Go-%3E%3D%201.18-blue)](https://go.dev/doc/install) [![License](https://img.shields.io/badge/License-MIT-green.svg)](../../../LICENSE) -[![Coverage](https://img.shields.io/badge/Coverage-88.2%25-brightgreen)](TESTING.md) +[![Coverage](https://img.shields.io/badge/Coverage-90.4%25-brightgreen)](TESTING.md) Production-ready I/O utilities collection providing specialized tools for stream processing, resource management, progress tracking, and concurrent I/O operations with comprehensive testing and thread-safe implementations. @@ -15,7 +15,6 @@ Production-ready I/O utilities collection providing specialized tools for stream - [Key Features](#key-features) - [Architecture](#architecture) - [Package Organization](#package-organization) - - [Dependency Graph](#dependency-graph) - [Performance](#performance) - [Benchmark Summary](#benchmark-summary) - [Coverage Statistics](#coverage-statistics) @@ -57,13 +56,13 @@ The **ioutils** package is a comprehensive collection of I/O utilities for Go ap 3. **Streaming-Oriented**: Designed for continuous data flow with constant memory usage, not batch processing 4. **Resource Management**: Proper cleanup with `defer`, context cancellation support, and automatic lifecycle handling 5. **Zero External Dependencies**: Only standard library and internal golib packages -6. **Production Hardened**: 772 test specs, 90.7% average coverage, extensive documentation +6. **Production Hardened**: 776 test specs, 90.4% average coverage, extensive documentation ### Key Features ✅ **Thread-Safe Operations**: All subpackages safe for concurrent use ✅ **Context Integration**: Cancellation and deadline propagation throughout -✅ **Comprehensive Testing**: 772 specs, zero race conditions +✅ **Comprehensive Testing**: 776 specs, zero race conditions ✅ **Rich Subpackage Ecosystem**: 10 specialized packages for different I/O needs ✅ **High Performance**: Benchmarked and optimized for throughput and latency ✅ **Well Documented**: GoDoc comments, examples, README per subpackage @@ -78,77 +77,39 @@ The **ioutils** package is a comprehensive collection of I/O utilities for Go ap ioutils/ ├── tools.go Root-level utilities (PathCheckCreate) │ -├── aggregator/ Concurrent write aggregation -│ └── [115 specs, 86.0% coverage] +├── aggregator/ Thread-safe write aggregator that buffers and serializes concurrent writes +│ └── [115 specs, 84.9% coverage] │ -├── bufferReadCloser/ Buffered readers with closer +├── bufferReadCloser/ io.Closer wrappers for bytes.Buffer and bufio types │ └── [44 specs, 100% coverage] │ -├── delim/ Delimiter-based stream processing -│ └── [95 specs, 100% coverage] +├── delim/ Buffered reader for reading delimiter-separated data streams +│ └── [95 specs, 98.6% coverage] │ -├── fileDescriptor/ File descriptor management +├── fileDescriptor/ Cross-platform file descriptor limit management │ └── [28 specs, 85.7% coverage] │ -├── ioprogress/ Progress tracking wrappers +├── ioprogress/ Thread-safe I/O progress tracking wrappers │ └── [54 specs, 84.7% coverage] │ -├── iowrapper/ Generic I/O wrappers +├── iowrapper/ Flexible I/O wrapper for customization and interception │ └── [88 specs, 100% coverage] │ -├── mapCloser/ Multiple closer management -│ └── [82 specs, 77.5% coverage] +├── mapCloser/ Thread-safe, context-aware manager for multiple io.Closer instances +│ └── [82 specs, 80.8% coverage] │ -├── maxstdio/ Stdio limit management +├── maxstdio/ Standard I/O limit management │ └── [No specs - utility package] │ -├── multi/ Write multiplexing -│ └── [112 specs, 81.7% coverage] +├── multi/ Thread-safe adaptive multi-writer extending io.MultiWriter +│ └── [112 specs, 80.8% coverage] │ -└── nopwritecloser/ No-op writer closers +└── nopwritecloser/ io.WriteCloser wrapper with no-op Close() └── [54 specs, 100% coverage] ``` -**Total**: 772 test specs, 90.7% average coverage +**Total**: 776 test specs, 90.4% average coverage -### Dependency Graph - -``` - ┌──────────────────────┐ - │ ioutils (root) │ - │ PathCheckCreate │ - └──────────────────────┘ - │ - ┌─────────────────────┼─────────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ aggregator │ │ ioprogress │ │ multi │ -│ (concurrent) │ │ (tracking) │ │ (multiplexing) │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ │ │ - └─────────────────────┴─────────────────────┘ - │ - ┌─────────────────────┼─────────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ iowrapper │ │ mapCloser │ │ nopwritecloser │ -│ (generic) │ │ (lifecycle) │ │ (no-op) │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ │ │ - └─────────────────────┴─────────────────────┘ - │ - ┌─────────────────────┼─────────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│bufferReadCloser │ │ delim │ │ fileDescriptor │ -│ (buffering) │ │ (delimiter) │ │ (limits) │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ -``` - -**Note**: Arrows represent conceptual grouping by complexity, not actual import dependencies. --- @@ -156,43 +117,43 @@ ioutils/ ### Benchmark Summary -Based on actual test execution (772 specs, ~33 seconds total): +Based on actual test execution (776 specs, ~41 seconds total): | Package | Specs | Coverage | Execution Time | Notable Metrics | |---------|-------|----------|----------------|-----------------| -| **aggregator** | 115 | 86.0% | ~30.8s | Start: 10.7ms, Throughput: 5000-10000/s | +| **aggregator** | 115 | 84.9% | ~33.3s | Start: 10.7ms, Throughput: 5000-10000/s | | **bufferReadCloser** | 44 | 100% | ~0.03s | Read: <1ms, Buffer: configurable | -| **delim** | 95 | 100% | ~0.19s | Read: <500µs, Scan: <1ms | +| **delim** | 95 | 98.6% | ~6.68s | Read: <500µs, Scan: <1ms | | **fileDescriptor** | 28 | 85.7% | ~0.01s | Limit check: <1µs | | **ioprogress** | 54 | 84.7% | ~0.02s | Callback: <10µs overhead | | **iowrapper** | 88 | 100% | ~0.08s | Wrap: <1µs, Pass-through | -| **mapCloser** | 82 | 77.5% | ~0.02s | Add/Remove: <1µs, Close all: <1ms | -| **multi** | 112 | 81.7% | ~0.15s | Write to N: O(N), Copy: <100µs | +| **mapCloser** | 82 | 80.8% | ~0.05s | Add/Remove: <1µs, Close all: <1ms | +| **multi** | 112 | 80.8% | ~0.69s | Write to N: O(N), Copy: <100µs | | **nopwritecloser** | 54 | 100% | ~0.24s | No-op: <1ns | -| **ioutils (root)** | - | 88.2% | ~0.02s | PathCheckCreate: varies | +| **ioutils (root)** | 31 | 88.2% | ~0.04s | PathCheckCreate: varies | **Aggregate Performance:** -- **Total execution**: ~33 seconds (including setup/teardown) -- **Average spec time**: ~43ms (dominated by aggregator's timing tests) +- **Total execution**: ~41 seconds (including setup/teardown) +- **Average spec time**: ~53ms (dominated by aggregator's timing tests) - **Zero race conditions**: All tests pass with `-race` detector - **Memory efficiency**: Constant memory for streaming operations ### Coverage Statistics ``` -Overall Coverage: 90.7% (weighted average) -Packages at 100%: 5/10 (bufferReadCloser, delim, iowrapper, nopwritecloser, root) -Packages >85%: 8/10 -Lowest Coverage: 77.5% (mapCloser - primarily error paths) +Overall Coverage: 90.4% (weighted average) +Packages at 100%: 3/10 (bufferReadCloser, iowrapper, nopwritecloser) +Packages >85%: 5/10 +Lowest Coverage: 80.8% (mapCloser, multi - primarily error paths) ``` **Coverage Breakdown:** | Coverage Range | Count | Packages | |----------------|-------|----------| -| 100% | 5 | bufferReadCloser, delim, iowrapper, nopwritecloser, root | -| 85-99% | 3 | aggregator (86%), fileDescriptor (85.7%), ioprogress (84.7%) | -| 75-84% | 2 | mapCloser (77.5%), multi (81.7%) | +| 100% | 3 | bufferReadCloser, iowrapper, nopwritecloser | +| 85-99% | 4 | delim (98.6%), ioutils (88.2%), fileDescriptor (85.7%), ioprogress (84.7%) | +| 80-89% | 3 | aggregator (84.9%), mapCloser (80.8%), multi (80.8%) | --- @@ -200,7 +161,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### aggregator -**Purpose**: Thread-safe write aggregator that serializes concurrent write operations to a single output function. +**Purpose**: Thread-safe write aggregator that buffers and serializes concurrent write operations to a single writer function. **Key Features**: - Concurrent writes from multiple goroutines @@ -222,20 +183,21 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### bufferReadCloser -**Purpose**: Buffered reader implementation with proper closer interface. +**Purpose**: Lightweight wrappers around Go's standard buffered I/O types (bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter) that add io.Closer support with automatic resource cleanup and custom close callbacks. **Key Features**: -- Wraps any `io.Reader` with buffering -- Implements `io.ReadCloser` -- Configurable buffer size -- Proper resource cleanup +- Wraps bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter +- Adds io.Closer interface to non-closeable buffers +- Automatic reset/flush on close +- Optional custom close callbacks +- Zero-copy passthrough to underlying buffers **Performance**: -- Read overhead: <1ms -- Buffer size: configurable (default 4KB) -- Zero-copy when possible +- Wrapper overhead: ~5.7ms per 10,000 operations +- I/O operations: 0-100ns (direct delegation) +- Memory: 24 bytes per wrapper -**Use Case**: Network streams, file reading with buffering, pipe wrapping +**Use Case**: Adding lifecycle management to standard buffers, automatic cleanup with defer, composable close operations **Documentation**: [bufferReadCloser/README.md](bufferReadCloser/README.md) @@ -243,7 +205,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### delim -**Purpose**: High-performance delimiter-based stream processing. +**Purpose**: Buffered reader for reading delimiter-separated data streams. **Key Features**: - Read until any delimiter character @@ -264,7 +226,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### fileDescriptor -**Purpose**: File descriptor limit management and validation. +**Purpose**: Cross-platform utilities for managing file descriptor limits, offering a unified API for querying and modifying the maximum number of open files or I/O resources allowed for a process. **Key Features**: - Check system-wide FD limits @@ -284,7 +246,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### ioprogress -**Purpose**: Thread-safe I/O progress tracking wrappers. +**Purpose**: Thread-safe I/O progress tracking wrappers for monitoring read and write operations in real-time through customizable callbacks. **Key Features**: - Reader/Writer progress callbacks @@ -305,7 +267,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### iowrapper -**Purpose**: Generic I/O wrappers for extending Reader/Writer functionality. +**Purpose**: Flexible I/O wrapper that enables customization and interception of read, write, seek, and close operations on any underlying I/O object without modifying its implementation. **Key Features**: - Wrap readers and writers @@ -361,7 +323,7 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### multi -**Purpose**: Thread-safe I/O multiplexer for broadcasting writes to multiple destinations. +**Purpose**: Thread-safe, adaptive multi-writer that extends Go's standard io.MultiWriter with advanced features including adaptive sequential/parallel execution, latency monitoring, and comprehensive concurrency support. **Key Features**: - Write to multiple writers atomically @@ -383,20 +345,20 @@ Lowest Coverage: 77.5% (mapCloser - primarily error paths) ### nopwritecloser -**Purpose**: No-op writer closer for testing and interface satisfaction. +**Purpose**: Wrapper that implements io.WriteCloser for an io.Writer by adding a no-op Close() method. **Key Features**: -- Implements `io.WriteCloser` -- Write discards all data -- Close is always successful -- Useful for testing +- Implements `io.WriteCloser` for any `io.Writer` +- Write operations delegate to underlying writer unchanged +- Close() is no-op (always returns nil, never affects writer) +- Thread-safe if underlying writer is thread-safe **Performance**: -- Write: <1ns (no-op) +- Write: Direct delegation (no overhead) - Close: <1ns (no-op) - Zero allocation -**Use Case**: Testing, benchmarking, interface mocking, discard sinks +**Use Case**: Protecting stdout/stderr from closure, API compatibility (io.Writer → io.WriteCloser), testing with inspectable buffers after close, shared resource management **Documentation**: [nopwritecloser/README.md](nopwritecloser/README.md) @@ -738,7 +700,7 @@ defer closer.Close() ## Testing -Comprehensive test suite with 772 specs across all subpackages. +Comprehensive test suite with 776 specs across all subpackages. **Quick Test:** @@ -754,10 +716,10 @@ go test -cover ./... ``` **Expected Results:** -- 772 specs passed -- 90.7% average coverage +- 776 specs passed +- 90.4% average coverage - Zero race conditions -- ~33 seconds execution time +- ~41 seconds execution time See [TESTING.md](TESTING.md) for detailed testing documentation including: - Running tests (standard, race, coverage, profiling) @@ -815,12 +777,12 @@ The package is **production-ready** with no urgent improvements or security vuln ### Code Quality Metrics -- ✅ **90.7% average test coverage** (target: >85%) +- ✅ **90.4% average test coverage** (target: >85%) - ✅ **Zero race conditions** detected with `-race` flag - ✅ **Thread-safe** implementations across all subpackages - ✅ **Memory-safe** with proper resource cleanup - ✅ **Standard interfaces** for maximum compatibility -- ✅ **772 comprehensive test specs** ensuring reliability +- ✅ **776 comprehensive test specs** ensuring reliability ### Future Enhancements (Non-urgent) diff --git a/ioutils/doc.go b/ioutils/doc.go index 4cd03a5..9bb86f1 100644 --- a/ioutils/doc.go +++ b/ioutils/doc.go @@ -43,16 +43,16 @@ The package is organized into a root-level utility and several specialized subpa Root Package (ioutils) ├── PathCheckCreate - File/directory creation with permission management │ - ├── aggregator - Buffered write aggregation for concurrent operations - ├── bufferReadCloser - Memory-efficient buffer management with io.ReadCloser - ├── delim - Delimiter-based stream processing - ├── fileDescriptor - Low-level file descriptor operations - ├── ioprogress - Progress tracking for I/O operations - ├── iowrapper - Generic I/O wrappers and decorators - ├── mapCloser - Collection-based closer management + ├── aggregator - Thread-safe write aggregator that buffers and serializes concurrent writes to a single writer function + ├── bufferReadCloser - io.Closer wrappers for bytes.Buffer and bufio types + ├── delim - Buffered reader for reading delimiter-separated data streams + ├── fileDescriptor - Cross-platform utilities for managing file descriptor limits + ├── ioprogress - Thread-safe I/O progress tracking wrappers for monitoring read and write operations + ├── iowrapper - Flexible I/O wrapper enabling customization and interception of I/O operations + ├── mapCloser - Thread-safe, context-aware manager for multiple io.Closer instances ├── maxstdio - Standard I/O redirection and capture - ├── multi - Multiple writer/reader composition - └── nopwritecloser - No-op writer/closer implementations + ├── multi - Thread-safe, adaptive multi-writer extending io.MultiWriter with advanced features + └── nopwritecloser - io.WriteCloser wrapper with no-op Close() Each subpackage is designed to solve specific I/O challenges while maintaining compatibility with standard library interfaces. @@ -210,15 +210,15 @@ validation in a single call. For detailed documentation, see individual subpackage docs: - - aggregator: Concurrent-safe write buffering and aggregation - - bufferReadCloser: Memory-efficient buffer with ReadCloser interface - - delim: Delimiter-based stream parsing and processing - - fileDescriptor: Low-level file descriptor operations and management - - ioprogress: Progress tracking and reporting for I/O operations - - iowrapper: Generic wrappers for io.Reader/Writer interfaces - - mapCloser: Collection management for multiple io.Closer instances + - aggregator: Thread-safe write aggregator that buffers and serializes concurrent write operations to a single writer function + - bufferReadCloser: io.Closer wrappers for bytes.Buffer and bufio types with cleanup + - delim: Buffered reader for reading delimiter-separated data streams + - fileDescriptor: Cross-platform utilities for managing file descriptor limits in Go applications + - ioprogress: Thread-safe I/O progress tracking wrappers for monitoring read and write operations in real-time through customizable callbacks + - iowrapper: Flexible I/O wrapper that enables customization and interception of read, write, seek, and close operations on any underlying I/O object + - mapCloser: Thread-safe, context-aware manager for multiple io.Closer instances - maxstdio: Standard I/O stream redirection and capture - - multi: Composite readers/writers for multiple sources/destinations - - nopwritecloser: No-op implementations for testing and composition + - multi: Thread-safe, adaptive multi-writer that extends Go's standard io.MultiWriter with advanced features + - nopwritecloser: No-op Close() wrapper for io.Writer (delegates Write unchanged) */ package ioutils