Automatic Cleanup of Stale, Non-Finite Transfer Processes to Prevent Vault Overflow

[marctuerke]

This discussion proposes a new feature to improve the lifecycle management of non-finite transfer processes and address their impact on the Vault due to stale Endpoint Data References (EDRs).

The Problem

Our Vault is progressively filling up with secrets associated with EDRs generated by non-finite transfer processes.

To provide context, the DSP specification makes a clear distinction between finite and non-finite transfer processes:

Data may be finite or non-finite. This applies to either push and pull transfers. Finite data is data that is defined by a finite set, for example, machine learning data or images. After finite data transmission has finished, the Transfer Process is completed. Non-finite data is data that is defined by an infinite set or has no specified end, for example, streams or an API endpoint. With non-finite data, a Transfer Process will continue indefinitely until either the Consumer or Provider explicitly terminates the transmission.

This design works perfectly for finite transfers. Once the data is transmitted, the process completes, and the associated EDR secret is correctly removed from the Vault.

The challenge arises with non-finite transfers. As per the spec, these processes remain active indefinitely. In practice, a consumer might abandon a stream without explicitly terminating the transfer process. In this scenario, the process remains active, and its corresponding EDR secret is never automatically cleaned up from the Vault.

This leads to an accumulation of "stale" but technically active processes and their unused EDR secrets, which can eventually cause the Vault to reach its storage limit, requiring manual intervention.

The Proposal

I propose the introduction of a mechanism to automatically complete stale, non-finite transfer processes.

The core idea is as follows:

A non-finite transfer process could be considered "stale" if there has been no data transfer activity associated with it for a configurable period of time (e.g., 24 hours).

A background task in the EDC would periodically scan for these stale processes.

Once a process is identified as stale, the EDC would automatically transition it to a final state (e.g., COMPLETED or TERMINATED). This transition would trigger the existing cleanup logic, removing the associated EDR secret from the Vault. This feature should ideally be configurable, allowing operators to:

• Enable or disable the automatic cleanup functionality.
• Define the inactivity timeout period (e.g., 1h, 24h, 7d).

Benefits

• Automated Vault Hygiene: Prevents the Vault from filling up with unused EDR secrets, reducing operational risk and manual cleanup.
• Improved Resource Management: Frees up resources in both the EDC (by closing idle processes) and the Vault.
• Reduced Operational Overhead: Eliminates the need for custom scripts or manual API calls to clean up old processes.

I'm looking forward to hearing the community's thoughts and feedback on this proposal.

Thank you

[wolf4ood]

Hi @marctuerke

thanks for raising this discussion. This functionality is already available in EDC in the form of policy monitor which checks the open transfers and run the policy engine in the policy monitor context.

A bundled policy function that runs in the policy monitor context is already available in EDC docs here which can be used to set the validity of a contract. Once it expires
it automatically closes all TP related to that contract negotiation and thus cleaning up resources.

If it's not suitable you can develop a custom policy function that works in the policy monitor context and do other custom checks

Let me know if this helps

Thanks

[jimmarino]

I would add that having long-running transfer processes is a best practice. It should be more common for transfers to be non-finite (streaming, API access) than finite and open for an extended period of time.

Closing these processes based on an arbitrary timeout should not be done. Instead, track and close them when finished or no longer needed. @wolf4ood outlined one way to do that. Another way is to have a user or client application do this. Either way, if resources are being leaked, it is likely the transfers are not being modeled correctly.