Operations Engineer, HPC Networking
The Operations Engineer, HPC Networking role at fal focuses on maintaining and scaling the company's high-performance computing (HPC) network infrastructure. This position is integral to ensuring the health and performance of InfiniBand and Ethernet fabrics, which are critical components of fal's generative media ecosystem. The engineer will collaborate closely with data center operations and customer-facing teams to support the deployment and operation of these networks.
Key responsibilities include monitoring the health and performance of InfiniBand and Ethernet fabrics, investigating and resolving network issues such as connectivity problems, congestion, and performance regressions, and supporting the bring-up of new network fabrics alongside data center operations. The role also involves conducting maintenance and upgrades on switches and control plane components, partnering with cluster operations on cross-domain incidents, and improving tooling and runbooks to enhance incident resolution efficiency.
Candidates should have experience operating InfiniBand fabrics in production environments, including subnet management, routing, partitioning, and monitoring. Proficiency in debugging across the full stack—cables, transceivers, switch firmware, host channel adapters (HCAs), drivers, and NCCL—is essential. Experience in bringing up new network fabrics from cable installation through validation, as well as scripting skills in languages such as Bash, Python, or Go, are required. Familiarity with Ethernet RoCE, Spectrum-X, or large-scale GPU cluster networking is considered a plus.
The ideal candidate is detail-oriented, maintains high standards for cable plant hygiene, remains calm under pressure during network incidents, stays informed by reading vendor release notes, and prioritizes identifying root causes over temporary fixes.
This position offers the opportunity to work remotely, providing flexibility and the chance to contribute to a leading company in the generative media industry.