Hey everyone, I’ve been tackling a bit of a hardware puzzle lately and could use some collective wisdom. I recently moved my main storage over to an NVMe-based array for my home lab, and while the local benchmarks are absolutely screaming, I’m seeing some really weird performance drop-offs when I access the data over the fabric.
One specific point I’ve noticed during my testing is that the latency spikes significantly during sequential writes once the initial buffer fills up. I’ve gone down the rabbit hole of checking CPU interrupts and NVMe driver versions, but I’m starting to look at the physical layer now. I’m currently using some older 8Gb Fibre Channel transceivers that I picked up second-hand a few years ago. I know that sounds a bit "old school" compared to modern 10GbE or 25GbE setups, but I’ve always preferred the dedicated nature of a SAN for my virtual machines and backup storage.
It’s a bit frustrating because you spend all this time and money on high-end SSDs and cooling, only to realize the "pipes" might be the real issue. It reminds me of when I first started tinkering with servers; I’d focus so much on the processor speed and totally ignore the interconnects. I’m starting to wonder if these older Fibre Channel modules are just hitting their thermal limits or if they simply can’t keep up with the burstiness of modern NVMe drives. I even checked the SFP logs, and everything seems "clean," but the throughput just isn't there.
Has anyone else here tried to mix "legacy" SAN hardware with newer flash storage? I’m trying to decide if I should just bite the bullet and upgrade to 16Gb or 32Gb transceivers, or if the overhead of Fibre Channel itself is becoming the bottleneck for consumer-grade NVMe.
Does the stability of an older FC fabric still beat out a modern iSCSI setup in your experience, or is it time for me to finally retire the transceivers and move to a different protocol entirely?
