OFI BTL and FI_DELIVERY_COMPLETE #7982
Description
Starting an issue so we can discuss the FI_DELIVERY_COMPLETE requirements of the OFI BTL and the state of the world with the various libfabric providers that claim support for it today.
No currently existing hardware can provide the FI_DELIVERY_COMPLETE semantic. The semantic is that when local completion has occurred at the initiator, “a completion guarantees that the result of the operation is available”, which is impossible to guarantee at the remote application layer when you’re sitting on the PCIe bus. However, the guarantee that hardware supporting FI_DELIVERY_COMPLETE actually provides could be read more as “the remote NIC has initiated the transaction to target memory”, which is a looser semantic but also exactly the semantic needed by upper layer protocols which require FI_DELIVER_COMPLETE.
A number of libfabric providers using bounce buffers for communication operations, especially RDMA operations, provide a FI_DELIVERY_COMPLETE semantic significantly looser than that promised by hardware. This list apparently includes RxD, EFA, and likely others. In this case, we’re seeing that the erroneous FI_DELIVERY_COMPLETE support is causing failures in Open MPI.
Creating this issue to use as a reference during the dev meeting on 8/4. Will update with more information and references.