Why Does Neon Track relpersistence in SMgrRelation Instead of Just Checking Local Disk?

[sujanacc]

I'm trying to understand the design decisions behind how Neon handles unlogged and temporary tables.

From what I see, Neon persists unlogged tables using local storage. To support this, there are some changes in the PostgreSQL codebase:

• The smgr open function has an extra parameter.
• The SMgrRelation structure now includes a relpersistence field.
• There are hooks for smgr_start_unlogged_build and related functions.

My question is:
Why does Neon need to track relpersistence in SMgrRelation and add these hooks? Wouldn't it be simpler to just create the local files for unlogged tables in RelationCreateStorage (for all four forks), and then, for every subsequent smgr call, check if the file exists on disk? If the file exists, we can treat it as an unlogged table.

If checking the disk is considered too expensive, couldn't we use the Relation Cache hashtable (which PostgreSQL already uses to cache relation structs)? This cache includes the relpersistence value, and if a lookup misses, PostgreSQL already scans the pg_class table to populate it. To avoid the table scan, we could fall back to checking for the file on disk.

In summary:
Why didn't Neon choose this kind of approach, which would require fewer changes to PostgreSQL? Is there a technical reason for maintaining the relpersistence state in SMgrRelation and introducing new hooks, rather than relying on file existence or the relation cache?

[hlinnaka]

Wouldn't it be simpler to just create the local files for unlogged tables in RelationCreateStorage (for all four forks), and then, for every subsequent smgr call, check if the file exists on disk? If the file exists, we can treat it as an unlogged table.

It would be expensive

If checking the disk is considered too expensive, couldn't we use the Relation Cache hashtable (which PostgreSQL already uses to cache relation structs)? This cache includes the relpersistence value, and if a lookup misses, PostgreSQL already scans the pg_class table to populate it. To avoid the table scan, we could fall back to checking for the file on disk.

Isn't that exactly what the code does today?

[sujanacc]

Isn't that exactly what the code does today?

I noticed that Neon stores the relpersistence value in the SMgrRelation struct. But PostgreSQL already has the RelationIdCache (relcache), which also stores relpersistence.

Wouldn’t it be simpler to just use the relcache to get the persistence type, instead of adding an extra field to SMgrRelation? Is there a reason why Neon doesn’t just rely on the relcache for this?

[knizhnik]

Postgres relcache is private to backend and is filled when relation is opened. And in this case do not need to guess relation persistence - it is passed to SMGR. The only problem is when page is evicted from shared buffer but this relation wasn't yet accessed by this backend. In this case persistence is unknown and this is why we have to use our own relopersistence cache.

[sujanacc]

Thanks a lot!!!.

Since Neon maintains the persistence state (relpersistence) in SMgrRelation, can't it directly handle two-phase index builds based on that information? Why does Neon still need a separate flow with neon_start_unlogged_build and related functions for unlogged index builds? What specific problem does this extra mechanism solve that isn't already covered by tracking persistence in SMgrRelation?