Users are looking for a library that implements an SSD-optimized Write-Ahead Log for databases or message brokers, focusing on performance improvements during write operations.
A [library](https://github.com/alialaee/logfile) I extracted from a custom database engine I built sometime ago. It ran in production, so it's reasonably "battle-tested". It's a simple, optimized append-only log file, that can be used to build a write-ahead log for a database or a message broker. The design is very much inspired by the write-ahead log in ScyllaDB. The main features that differentiate it from other Go log file libraries are: - SSD-Optimized: Writes are explicitly padded to 4KB sector boundaries to maximize SSD performance. - No flush timers: Flushes are triggered immediately after every write, but as it uses group commit, under heavy load, multiple concurrent writes are batched into a single fsync. - Group Commit: under concurrency, writes are batched into a single IO + fsync using pingpong buffers. Basically one batch is being flushed, new writes keep accumulating. - Very simple API: `Write()` is blocking and safe to call from hundreds of goroutines. Actually with more `goroutines`, you get better performance (until a certain point). But it can come close to saturating the disk. I intentionally left out file management and rotation since every database handles that differently. But I might add it. Also The reader is very basic right now. I'm working on it. If you are interested in this kind of thing or have any suggestion, shoot me a message or comment. I'd appreciate it. There's another library from the same codebase for fast `Marshal`/`Unmarshal` into a binary format that is very suitable for log files and databases. It's called [RAF](https://github.com/alialaee/raf). It has zero heap allocations and random access to fields without unmarshalling the whole record. It's also schema-less. I did lots of tricks to have a reasonable performance using `reflect` package. But I have some more improvements that I'm working on (`op-codes` and internal interpreters).