From 8b6e6348160676996efb8beee14a81a2a4ccd088 Mon Sep 17 00:00:00 2001 From: Bear Giles Date: Wed, 28 May 2025 16:59:51 -0600 Subject: [PATCH 1/2] Prototype based on blackhole_fdw and c -> java calls This is prototype based on the blackhole_fdw. That is - the required method are the minimum required for the FDW to be loaded. There's quite a bit of extra code required if before you can use a real backend that's just happens to also return nothing. I have already implemented that elsewhere but for now it's fine to skip it. That said most of the provided methods call the appropriate java method. THESE ARE NOT STATIC METHODS - it is assumed that everything except for the FDWValidator is associated with a Java object - FDWForeignDataWrapper, FDWServer, FDWTable, or the various internal states. The `fwd_private` fields contain a `jobject` for the java `instance`. It is passed through to java via the JNI call in the same way that we can use `method.invoke(Object obj,...)` in Java. There is also a dummy implementation of the java side of the FDW. It does nothing but log an acknowledgement that the method was called. (Is there a way to call trigger `elog(NOTIFY,...)` ? BONUS - and probably a separate pull request The pljava-so directory contains a Dockerfile and docker-compose.yml file that can be used to create a test image. The makefile(?) needs to add a stanza that calls `docker build ...` - I can provide the details later. For this project it may make more sense to move this to the packaging module - for the other project I'm working on it's entirely standalone so I can do everything in that module. Once there's a docker image I can add an 'example' that uses TestContainers to run some actual java-based tests. KNOWN ISSUES There are a lot. The most basic is that I'm not reusing existing bits of JNI. This should be easy to fix. The most important, and the reason I haven't tried to actually build and run the .so, is that most of these objects need to stay in the persistent memory context since they live beyond the lifetime of the query. I know there's already one in the existing backend code but I don't know if I should use it or a per-FDW one. It's also clear that there needs to be an OID associated with the Foreign Data Wrapper, Server, and Foreign Table objects since they're persistence database objects. It should be possible to reuse existing ones, or at least have a clean way to access them. For now I'm using a simple tree - actually a one-to-one-to-one tree - but while the hooks are there it's not actually executing all of the steps yet. However there is enough that we can create a standalone object for testing. However this has raised a second issue - the method signatures to create a new Server or Foreign Table are identical but both Foreign Data Wrappers and Servers can support multiple children. I'm sure there's an easy way to get the jclass from a jobject but I wanted to focus on other things since we can always fall back to assuming a single matching class. There also needs to be correct mapping between int and jint, char * and jstring, etc. Once we have a solid foundation we can start adding conversions for the rest of the functions/methods. Finally the user OID is available to us but I don't know how to retrieve the desired information from it. For maximum flexibility (and security) we want at least three things: - the database username - the authentication method used - the connection method used (via unix named file, TCP/IP (IP address), etc.) The last item is important in highly secured environments, e.g., some critical operations may be limited to local users - and then the java class may still require additional authentication with something like a Yubico key. (Can you tell my other project is related to encryption keys and signing using harded external devices?) So, overall, this is enough to give a decent representation of what's required to have the C-based FDW call java classes for the actual work. There's quite a bit more work to be done on the FDW side but it's internal bookkeeping and doesn't affect the java SPI. However I don't know what prep work needs to be done beyond what's already done for UDF and UDT, and there's definitely the question of how the different persistent nodes are created and used. I've looked at a few other implementations but this goes a few steps beyond them due to the desire to eventually support multiple servers and tables. FDW Handler?... I've been a bit confused about a single handler when there has always been a desire to provide multiple servers and foreign tables. I think I have an answer though - the two functions are persistent and will have nodes and OIDs associated with them. The initial definition can point to a 'blackhole' Handler but it the handler could be replaced at the Server and Foreign table level. That's the only thing that makes sense since different tables will require different implementations of the planner, scanner, modification methods, etc. It's likely that the Validator can also change since the meaningful values for a Foreign Table may depend on the the specific Server used. BONUS ITEM #2 This is WAAAY out there but while digging through the deeply nested structures I came across a function pointer for performance analysis. I don't think it's limited to just FDW - it was pretty deep in the standard structures at this point. On one hand I shudder to think of the impact of proving a JNI binding to allow a java class to collect performance metrics. On the other hand.... That said... I've looked at similar situations in the past and have largely concluded that the best solution - on a single-node system - is to use a named pipe, or IPC if you're more comfortable with it. As long as you're willing to accept data loss if the system is overwhelmed there's not much risk to slamming information into either pseudodevice and relying on a consumer to move the information somewhere else. For instance something like ApacheMQ so that the analysis can be performed on one or more remote machines. However it still got me wondering... and I'm sure there's something similar for row and column-level authorization, auditing, perhaps even encryption. --- .../pljava/fdw/FDWForeignDataWrapper.java | 5 + .../pljava/fdw/FDWForeignTable.java | 12 + .../postgresql/pljava/fdw/FDWPlanState.java | 9 + .../postgresql/pljava/fdw/FDWScanState.java | 10 + .../org/postgresql/pljava/fdw/FDWServer.java | 10 + .../postgresql/pljava/fdw/FDWValidator.java | 9 + .../fdw/BlackholeForeignDataWrapper.java | 32 + .../example/fdw/BlackholeForeignTable.java | 46 ++ .../example/fdw/BlackholePlanState.java | 29 + .../example/fdw/BlackholeScanState.java | 39 + .../pljava/example/fdw/BlackholeServer.java | 39 + .../example/fdw/BlackholeValidator.java | 47 ++ pljava-so/Dockerfile | 27 + pljava-so/docker-compose.yml | 17 + pljava-so/src/main/c/BlackholeFDW.c | 704 ++++++++++++++++++ pljava-so/src/main/include/pljava/FDW.h | 23 + .../main/resources/fdw/blackhole_fdw.control | 5 + .../main/resources/fdw/sql/blackhole_fdw.sql | 31 + 18 files changed, 1094 insertions(+) create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignDataWrapper.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholePlanState.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeServer.java create mode 100644 pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeValidator.java create mode 100644 pljava-so/Dockerfile create mode 100644 pljava-so/docker-compose.yml create mode 100644 pljava-so/src/main/c/BlackholeFDW.c create mode 100644 pljava-so/src/main/include/pljava/FDW.h create mode 100644 pljava-so/src/main/resources/fdw/blackhole_fdw.control create mode 100644 pljava-so/src/main/resources/fdw/sql/blackhole_fdw.sql diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java new file mode 100644 index 000000000..519f31488 --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java @@ -0,0 +1,5 @@ +package org.postgresql.pljava.fdw; + +public interface FDWForeignDataWrapper { + FDWServer getServer(); +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java new file mode 100644 index 000000000..513696f1d --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java @@ -0,0 +1,12 @@ +package org.postgresql.pljava.fdw; + +public interface FDWForeignTable { + FDWPlanState newPlanState(); + FDWScanState ScanState(); + + default boolean updatable() { return false; } + + default void analyze() { }; +// +// default void vacuum() { }; +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java new file mode 100644 index 000000000..f9c00a69f --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java @@ -0,0 +1,9 @@ +package org.postgresql.pljava.fdw; + +public interface FDWPlanState { + void open(); + + void close(); + + // int rows(); +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java new file mode 100644 index 000000000..e32d6a4fb --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java @@ -0,0 +1,10 @@ +package org.postgresql.pljava.fdw; + +public interface FDWScanState { + void open(); + void next(Object slot); + void reset(); + void close(); + + // void explain(); ?? +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java new file mode 100644 index 000000000..d9168e97f --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java @@ -0,0 +1,10 @@ +package org.postgresql.pljava.fdw; + +import java.sql.ResultSetMetaData; + +public interface FDWServer { + FDWForeignTable getForeignTable(); + + // For 'importSchemaStmt() + ResultSetMetaData getMetaData(); +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java new file mode 100644 index 000000000..ff9a0de42 --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java @@ -0,0 +1,9 @@ +package org.postgresql.pljava.fdw; + +public interface FDWValidator { + void addOption(int relid, String key, String value); + + boolean validate(); + + FDWForeignDataWrapper getForeignDataWrapper(); +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignDataWrapper.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignDataWrapper.java new file mode 100644 index 000000000..f1e8bf44a --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignDataWrapper.java @@ -0,0 +1,32 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWForeignDataWrapper; +import org.postgresql.pljava.fdw.FDWServer; + +import java.util.Collections; +import java.util.Map; +import java.util.logging.Logger; + +/** + * A ForeignDataWrapper. (Persistent) + * + * Note: a single ForeignDataWrapper may contain multiple servers + * so there should be caching somewhere. + */ +public class BlackholeForeignDataWrapper implements FDWForeignDataWrapper { + private static final Logger LOG = Logger.getLogger(BlackholeForeignDataWrapper.class.getName()); + + public BlackholeForeignDataWrapper() { + this(Collections.emptyMap()); + } + + public BlackholeForeignDataWrapper(Map options) { + LOG.info("constructor"); + } + + @Override + public FDWServer getServer() { + LOG.info("getServer()"); + return new BlackholeServer(); + } +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java new file mode 100644 index 000000000..89e3f477b --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java @@ -0,0 +1,46 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWForeignDataWrapper; +import org.postgresql.pljava.fdw.FDWPlanState; +import org.postgresql.pljava.fdw.FDWScanState; +import org.postgresql.pljava.fdw.FDWForeignTable; + +import java.util.Collections; +import java.util.Map; +import java.util.logging.Logger; + +/** + * A Foreign Table (persistent) + * + * A single ForeignTable may have multiple PlanStates and ScanStates + * however they are transient and unlikely to be reused. + */ +public class BlackholeForeignTable implements FDWForeignTable { + private static final Logger LOG = Logger.getLogger(BlackholeForeignTable.class.getName()); + + public BlackholeForeignTable() { + this(Collections.emptyMap()); + } + + public BlackholeForeignTable(Map options) { + LOG.info("constructor"); + } + + @Override + public FDWPlanState newPlanState() { + LOG.info("getPlanState()"); + return new BlackholePlanState(this); + } + + @Override + public FDWScanState newScanState() { + LOG.info("newScanState()"); + return new BlackholeScanState(this); + } + + @Override + public boolean updatable() { + LOG.info("updatable()"); + return false; + } +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholePlanState.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholePlanState.java new file mode 100644 index 000000000..865912759 --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholePlanState.java @@ -0,0 +1,29 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWPlanState; + +import java.util.logging.Logger; + +/** + * A ForeignTable plan state. (Temporary) + */ +public class BlackholePlanState implements FDWPlanState { + private static final Logger LOG = Logger.getLogger(BlackholePlanState.class.getName()); + + private final BlackholeForeignTable table; + + public BlackholePlanState(BlackholeForeignTable table) { + LOG.info("constructor()"); + this.table = table; + } + + @Override + public void open() { + LOG.info("open()"); + } + + @Override + public void close() { + LOG.info("close()"); + } +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java new file mode 100644 index 000000000..98a2c924b --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java @@ -0,0 +1,39 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWScanState; + +import java.util.logging.Logger; + +/** + * A ForeignTable scan state. (Temporary) + */ +public class BlackholeScanState implements FDWScanState { + private static final Logger LOG = Logger.getLogger(BlackholeScanState.class.getName()); + + private final BlackholeForeignTable table; + + public BlackholeScanState(BlackholeForeignTable table) { + LOG.info("constructor()"); + this.table = table; + } + + @Override + public void open() { + LOG.info("open()"); + } + + @Override + public void next(Object slot) { + LOG.info("next()"); + } + + @Override + public void reset() { + LOG.info("reset()"); + } + + @Override + public void close() { + LOG.info("close()"); + } +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeServer.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeServer.java new file mode 100644 index 000000000..0abb486f1 --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeServer.java @@ -0,0 +1,39 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWForeignTable; +import org.postgresql.pljava.fdw.FDWServer; + +import java.sql.ResultSetMetaData; +import java.util.Collections; +import java.util.Map; +import java.util.logging.Logger; + +/** + * A Foreign Server (persistent) + * + * Note: a single Server may contain multiple ForeignTables + * so there should be caching somewhere. + */ +public class BlackholeServer implements FDWServer { + private static final Logger LOG = Logger.getLogger(BlackholeServer.class.getName()); + + public BlackholeServer() { + this(Collections.emptyMap()); + } + + public BlackholeServer(Map options) { + LOG.info("constructor()"); + } + + @Override + public FDWForeignTable getForeignTable() { + LOG.info("getForeignTable()"); + return new BlackholeForeignTable() {}; + } + + @Override + public ResultSetMetaData getMetaData() { + LOG.info("getMetaData()"); + return null; + } +} diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeValidator.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeValidator.java new file mode 100644 index 000000000..863987ffb --- /dev/null +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeValidator.java @@ -0,0 +1,47 @@ +package org.postgresql.pljava.example.fdw; + +import org.postgresql.pljava.fdw.FDWForeignDataWrapper; +import org.postgresql.pljava.fdw.FDWValidator; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.logging.Logger; + +public class BlackholeValidator implements FDWValidator { + private static final Logger LOG = Logger.getLogger(BlackholeValidator.class.getName()); + + // note: we know there's only five possible integer values. + private final Map> options = new HashMap<>(); + + public BlackholeValidator() { + this(Collections.emptyMap()); + } + + public BlackholeValidator(Map options) { + LOG.info("constructor"); + } + + @Override + public void addOption(int relid, String key, String value) { + LOG.info(String.format("addOption(%d, %s, %s)", relid, key, value)); + + if (!options.containsKey(relid)) { + options.put(relid, new HashMap<>()); + } + + options.get(relid).put(key, value); + } + + @Override + public boolean validate() { + LOG.info("validate()"); + return true; + } + + @Override + public FDWForeignDataWrapper getForeignDataWrapper() { + LOG.info("getForeignDataWrapper()"); + return new BlackholeForeignDataWrapper(); + } +} diff --git a/pljava-so/Dockerfile b/pljava-so/Dockerfile new file mode 100644 index 000000000..1229418d6 --- /dev/null +++ b/pljava-so/Dockerfile @@ -0,0 +1,27 @@ +# TODO: pull various VERS from the environment... + +FROM postgres:17.2-bookworm +LABEL authors="Bear Giles " + +ENV TARGET=target +ENV RESOURCES=src/main/resources + +# can/should be set as build property... +ENV PG_VERS=17 +ENV LIBDIR=/usr/lib/postgresql/${PG_VERS}/lib +ENV EXTDIR=/usr/share/postgresql/${PG_VERS}/extension + +ENV SO_NAME=pljava.so + +ENV FDW_NAME=blackhole_fdw +ENV FDW_VERS=1.9.6 + +# this will install the standard version. It can be updated once the docker image is running in a test environment. +RUN apt-get update && apt-get install -y postgresql-${PG_VERS}-pljava postgresql-${PG_VERS}-pljava-dbgsym + +COPY ${TARGET}/${SO_NAME}.so ${LIBDIR}/${SO_NAME}.so + +COPY ${RESOURCES}/fdw/${FDW_NAME}.control ${EXTDIR}/ +COPY ${RESOURCES}/fdw/sql/${FDW_NAME}*.sql ${EXTDIR}/ + +# ENTRYPOINT ["top", "-b"] \ No newline at end of file diff --git a/pljava-so/docker-compose.yml b/pljava-so/docker-compose.yml new file mode 100644 index 000000000..d432a00d8 --- /dev/null +++ b/pljava-so/docker-compose.yml @@ -0,0 +1,17 @@ +services: + postgresql: + image: blackhole:latest + container_name: blackhole + ports: + - '5432:5432' + environment: + POSTGRES_PASSWORD: password + networks: + - frontend + +networks: + frontend: + # Specify driver options + driver: bridge + driver_opts: + com.docker.network.bridge.host_binding_ipv4: "127.0.1.1" diff --git a/pljava-so/src/main/c/BlackholeFDW.c b/pljava-so/src/main/c/BlackholeFDW.c new file mode 100644 index 000000000..9e1e04051 --- /dev/null +++ b/pljava-so/src/main/c/BlackholeFDW.c @@ -0,0 +1,704 @@ +/** + * Actual implementation of minimal FDW based on the 'blackhole_fdw' + * project. For simplicity all of the comments and unused functions + * have been removed. + * + * The purpose of this file is to demonstrate the ability of C-based + * FDW implementation to successfully interact with a java object + * that implements the FDW interfaces. + * + * The first milestone is simply sending a NOTICE from the java: + * method. + */ +#include "postgres.h" + +#include "access/reloptions.h" +#include "foreign/fdwapi.h" +#include "foreign/foreign.h" +#include "optimizer/pathnode.h" +#include "optimizer/planmain.h" +#include "optimizer/restrictinfo.h" + +#include "../include/pljava/FDW.h" + +PG_MODULE_MAGIC; + +#if (PG_VERSION_NUM < 90500) +// fail... +#endif + +/* + * SQL functions + */ +extern Datum blackhole_fdw_handler(PG_FUNCTION_ARGS); + +extern Datum blackhole_fdw_validator(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(blackhole_fdw_handler); +PG_FUNCTION_INFO_V1(blackhole_fdw_validator); + + +/* callback functions */ +static void blackholeGetForeignRelSize(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid); + +static void blackholeGetForeignPaths(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid); + +static FdwPlan *blackholePlanForeignScan(Oid foreigntableid, PlannerInfo *root, RelOptInfo *baserel); + +static void blackholeBeginForeignScan(ForeignScanState *node, + int eflags); + +static TupleTableSlot *blackholeIterateForeignScan(ForeignScanState *node); + +static void blackholeReScanForeignScan(ForeignScanState *node); + +static void blackholeEndForeignScan(ForeignScanState *node); + +/* everything below here is optional */ + +static int blackholeIsForeignRelUpdatable(Relation rel); + +// TODO: locate 'ExplainState' +// static void blackholeExplainForeignScan(ForeignScanState *node, struct ExplainState *es); + +#if (PG_VERSION_NUM >= 120000) +static void blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + TupleTableSlot *slot, + bool *updated); +#else + +static HeapTuple blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + bool *updated); + +#endif + +static List *blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, + Oid serverOid); + +#endif + +static bool blackholeAnalyzeForeignTable(Relation relation, + AcquireSampleRowsFunc *func, + BlockNumber *totalpages); + +#if (PG_VERSION_NUM >= 120000) +static void blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + TupleTableSlot *slot, + bool *updated); +#else + +static HeapTuple blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + bool *updated); + +#endif + +/* ------------------------------------------------------------ + * The POSTGRESQL Functions + * -----------------------------------------------------------*/ +// this needs to be known BEFORE we execute 'CREATE FOREIGN DATA WRAPPER...' +static const char FDW_validator_classname = "org/postgresql/pljava/fdw/BlackholeValidator"; +static const char FDW_handler_classname = "org/postgresql/pljava/fdw/BlackholeHandler"; // ??? + +Datum +blackhole_fdw_handler(PG_FUNCTION_ARGS) { + FdwRoutine *fdwroutine = makeNode(FdwRoutine); + + elog(DEBUG1, "entering function %s", __func__); + + /* + * assign the handlers for the FDW + * + * This function might be called a number of times. In particular, it is + * likely to be called for each INSERT statement. For an explanation, see + * core postgres file src/optimizer/plan/createplan.c where it calls + * GetFdwRoutineByRelId((). + */ + + /* Required by notations: S=SELECT I=INSERT U=UPDATE D=DELETE */ + + /* these are required */ + fdwroutine->GetForeignRelSize = blackholeGetForeignRelSize; /* S U D */ + fdwroutine->GetForeignPaths = blackholeGetForeignPaths; /* S U D */ + fdwroutine->GetForeignPlan = blackholeGetForeignPlan; /* S U D */ + fdwroutine->BeginForeignScan = blackholeBeginForeignScan; /* S U D */ + fdwroutine->IterateForeignScan = blackholeIterateForeignScan; /* S */ + fdwroutine->ReScanForeignScan = blackholeReScanForeignScan; /* S */ + fdwroutine->EndForeignScan = blackholeEndForeignScan; /* S U D */ + + /* remainder are optional - use NULL if not required */ + /* support for insert / update / delete */ + fdwroutine->IsForeignRelUpdatable = blackholeIsForeignRelUpdatable; + fdwroutine->AddForeignUpdateTargets = NULL; /* U D */ + fdwroutine->PlanForeignModify = NULL; /* I U D */ + fdwroutine->BeginForeignModify = NULL; /* I U D */ + fdwroutine->ExecForeignInsert = NULL; /* I */ + fdwroutine->ExecForeignUpdate = NULL; /* U */ + fdwroutine->ExecForeignDelete = NULL; /* D */ + fdwroutine->EndForeignModify = NULL; /* I U D */ + + /* support for EXPLAIN */ + // fdwroutine->ExplainForeignScan = blackholeExplainForeignScan; /* EXPLAIN S U D */ + fdwroutine->ExplainForeignScan = NULL; /* EXPLAIN S U D */ + fdwroutine->ExplainForeignModify = NULL; /* EXPLAIN I U D */ + + /* support for ANALYSE */ + fdwroutine->AnalyzeForeignTable = blackholeAnalyzeForeignTable; /* ANALYZE only */ + + /* Support functions for IMPORT FOREIGN SCHEMA */ + fdwroutine->ImportForeignSchema = blackholeImportForeignSchema; + + /* Support for scanning foreign joins */ + fdwroutine->GetForeignJoinPaths = NULL; + + /* Support for locking foreign rows */ + fdwroutine->GetForeignRowMarkType = NULL: + fdwroutine->RefetchForeignRow = blackholeRefetchForeignRow; + + // none of the newer functions are handled yet - they deal with 'direct' access, concurrency, and async. + + PG_RETURN_POINTER(fdwroutine); +} + +Datum +blackhole_fdw_validator(PG_FUNCTION_ARGS) { + List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); + + elog(DEBUG1, "entering function %s", __func__); + + /* make sure the options are valid */ + + /* no options are supported */ + JNIEnv *env = NULL; + JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname); + + errhint("Blackhole FDW does not support any options"))); + + PG_RETURN_POINTER(fdwroutine); + + PG_RETURN_VOID(); +} + +/* + + - I know from context that the Foreign Data Wrapper, Server, and Foreign Table have OIDs associated with them... + +Datum +blackhole_fdw_server(PG_FUNCTION_ARGS) { + FdwServer *fdwserver = makeNode(FdwServer); + PG_RETURN_POINTER(fdwserver); +} + +Datum +blackhole_fdw_table(PG_FUNCTION_ARGS) { + FdwTable *fdwtable = makeNode(FdwTable); + PG_RETURN_POINTER(fdwtable); +} + */ + +/* ------------------------------------------------------------ + * The JNI headers + * ------------------------------------------------------------*/ + +static typedef struct JNI_FDW_Wrapper JNI_FDW_Wrapper; +static typedef struct JNI_FDW_Server JNI_FDW_Server; +static typedef struct JNI_FDW_Table JNI_FDW_Table; +static typedef struct JNI_FDW_PlanState JNI_FDW_PlanState; +static typedef struct JNI_FDW_ScanState JNI_FDW_ScanState; + +static JNI_FDW_Wrapper *validator_get_wrapper(JNI_FDW_Validator *validator); +static JNI_FDW_Server *wrapper_get_server(JNI_FDW_Wrapper *wrapper); +static JNI_FDW_Table *server_get_table(JNI_FDW_Server *server); +static JNI_FDW_PlanState *table_new_plan(JNI_FDW_Table *table); +static JNI_FDW_ScanState *table_new_scan(JNI_FDW_Table *table + +// not all functions... + +static void validator_add_option(JNI_FDW_Validator *, int relid, String key, String value); +static bool validator_validate(JNI_FDW_Validator *); + +static JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table); +static void plan_open(JNI_FDW_Table *table, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid); +static void plan_close(JNI_FDW_PlanState *plan_state); + +static void scan_open(JNI_FDW_ScanState *scan_state, ForeignScanState *node, int eflag); +static void scan_next(JNI_FDW_ScanState *scan_state, Slot *slot); +static void scan_reset(JNI_FDW_ScanState *scan_state); +static void scan_close(JNI_FDW_ScanState *scan_state); + +static JNI_FDW_ScanState *table_new_scanPlan(JNI_FDW_Table *table); + +// Note: this does not do memory management yet! + +static +jmethodId getMethodId(JNIEnv *env, jclass class, ...) +{ + return env->GetMethodIdw(class, vargargs); +} + +/** + * Public: fdwvalidator method + */ +static +typedef struct { + void (*addOption)(JNI_FDW_Validator *, int, const char *, const char *) = validator_add_option; + bool (*validate)(JNI_FDW_Validator *) = validator_validate; + JNI_FDW_Wrapper *(*get_wrapper)(JNI_FDW_Validator *) = validator_get_wrapper; + + const JNIEnv *env; + const jclass validatorClass; + const jobject instance; +} JNI_FDW_Validator; + +JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname) { + JNI_FDW_Validator *validator = (JNI_FDW_Validator *) palloc0(sizeof JNI_FDW_Validator); + + validator->env = env; + validator->validatorClass = env->FindClass(validator_classname); + validator->instance = env->AllocObject(fdw->validatorClass); + + return validator; +} + +/* FIXME - how to handle 'handler' since the 'create foreign wrapper' requires it but we start with the validator? */ + +/* FIXME - how to we pass wrapper, server, and table options to each? We have them in the Validator function...*/ + +/* FIXME - plus shouldn't the validation already know the associated wrapper, server, and table?... */ + + +/** + * Public - Foreign Data Wrapper + */ +static +typedef struct { + jobject (*newServer)(JNI_FDW_Wrapper *) = wrapper_new_server; + + JNIEnv *env; + jclass wrapperClass + jobject *instance; + + JNI_FDW_Validator *validator; +} JNI_FDW_Wrapper; + +static +JNI_FDW_Wrapper *validator_new_wrapper(JNI_FDW_Validator *validator, const char *handler_classname) +{ + const JNIEnv *env = validator->env; + jmethodId validateMethodId = env->GetMethodID(validator->validatorClass, "validate", "(V)[org.postgresql.pljava.fdw.Wrapper;"); + + const JNI_FDW_Wrapper *wrapper = (JNI_FDW_Wrapper *) palloc0(sizeof JNI_FDW_Wrapper); + + wrapper->env = validator->env; + wrapper->wrapperClass = env->FindClass(validator_classname); + wrapper->instance = env->CallObjectMethod(validator->instance, validator->validateMethodId); + wrapper->validator = validator; + + return wrapper; +} + +/* + * Public - Server + */ +static +typedef struct { + void* (*newTable)(void) = server_new_table; + + JNIEnv *env; + jclass serverClass; + jobject *instance; + + JNI_FDW_Table *wrqpper; +} JNI_FDW_Server; + +static +JNI_FDW_Server *wrapper_new_server(JNI_FDW_Wrapper *wrapper) +{ + const JNIEnv *env = wrapper->env; + jmethodId newServerMethodId = env->GetMethodID(wrapper->wrapperClass, "newServer", "(V)[org.postgresql.pljava.fdw.Server;"); + + const JNI_FDW_Server *server = (JNI_FDW_Server *) palloc0(sizeof JNI_FDW_Server); + server->env = env; +// server->serverClass = env->FindClass(validator_classname); + server->instance = env->CallObjectMethod(wrapper->instance, newServerMethodId); + server->wrapper = wrapper; + + return server; +} + +/* + * Public - Foreign Table + */ +static +typedef struct { + void* (*newPlanState)(JNI_FDW_Table *table); + void *(*newScanState)(JNI_FDW_Table *table); + void (*analyze)(JNI_FDW_Table *table); + + JNIEnv *env; + jclass tableClass; + jobject instance; + + JNI_FDW_Table *server; +} JNI_FDW_Table; + +static +JNI_FDW_Table *server_new_table(JNI_FDW_Server *server) { + const JNIEnv *env = server->env; + jmethodId newTableMethodId = env->GetMethodID(server->serverClass, "newTable", + "(V)[org.postgresql.pljava.fdw.Table;"); + + const JNI_FDW_Table *table = (JNI_FDW_Table *) palloc0(sizeof JNI_FDW_Table); + table->env = env; + // table->tableClass = env->FindClass(table_classname); + table->instance = env->CallObjectMethod(wrapper->instance, newTableMethodId); + table->server = server; + + return table; +} + +/* + * Private - plan state + */ +static +typedef struct { + void (*open)(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foregntableid) = plan_open; + void (*close)(JNI_FDW_PlanState *planState); + + JNIEnv *env; + jobject *instance; + + JNI_FDW_Table *table; + +} JNI_FDW_PlanState; + +static +JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table) { + const JNIEnv *env = table->env; + + jmethodId openPlanMethodId = env->GetMethodID(table->tableClass, "newPlanState", + "(V)[org.postgresql.pljava.fdw.PlanState;"); + + const JNI_FDW_PlanState *planState = (JNI_FDW_PlanState *) palloc0(sizeof JNI_FDW_PlanState); + planState->env = env; + // table->planStateClass = env->FindClass(planstate_classname); + planState->instance = env->CallObjectMethod(table->instance, newPlanStateMethodId); + planState->table = table; +} + +static +void *plan_open(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid) { + const JNIEnv *env = table->env; + + // FIXME: for now we don't pass anything through. However we could after a bit of conversions... + const jmethodId openPlanMethodId = env->GetMethodID(planState->planStateClass, "open", "(V)V"); + env->CallObjectMethod(planState->instance, openPlanStateMethodId); +} + +/* + * Private - scan state + */ +static +typedef struct { + void (*open)(JNI_FDW_ScanState *scanState, ForeignScanState *node, int eflags) = open_scan; + void (*next)(JNI_FDW_ScanState *scanState, TableTupleSlot *slot); + void (*reset)(JNI_FDW_ScanState *scanState); + void (*close)(JNI_FDW_ScanState *scanState); + void (*explain)(JNI_FDW_ScanState *scanState); + + JNIEnv *env; + jobject *instance; + + JNI_FDW_Table *table; +} JNI_FDW_ScanState; + +static +JNI_FDW_ScanState *table_new_scan_state(JNI_FDW_Table *table, ForeignScanState *node, int eflag) { + const JNIEnv *env = table->env; + + // for now we ignore the extra parameters. + const jmethodId newScanStateMethodId = env->GetMethodID(table->tableClass, "newScanState", + "(V)[org.postgresql.pljava.fdw.ScanState;"); + + const JNI_FDW_ScanState *scanState = (JNI_FDW_ScanState *) palloc0(sizeof JNI_FDW_ScanState); + scanState->env = env; + + // for now we ignore the extra parameters. + scanState->instance = env->CallObjectMethod(table->instance, newScanStateMethodId); + scanState->table = table; + + return planState; +} + +static +void scan_open(JNI_FDW_ScanState *scanState, ForeignScanState *node, int eflag) { + +} + +/* ------------------------------------------------------------ + * Rest of JNI Implementation - does not use memory management yet! + * ------------------------------------------------------------*/ +static void validator_add_option(JNI_FDW_Validator *validator, int relid, String key, String value) +{ + const JNIEnv *env = validator->env; + const jmethodId addOptionMethodId = env->GetMethodID(validator->validatorClass, "addOption", "(int, String, String)V"); + + const jint jrelid = NULL; + const jstring jkey = NULL; + const jstring jvalue = NULL; + + env->CallObjectMethod(validator->instance, addOptionMethodId, jrelid, jkey, jvalue); +} + +/* ------------------------------------------------------------ + * The POSTGRESQL implementations + * -----------------------------------------------------------*/ + +/* ------------------------------------------------------------ + * FIXME: How do we get to specific JNI_FDW_Table ?? + * ------------------------------------------------------------*/ + + +/** + * Called to get an estimated size of the foreign table. + * + * Note: this can be a no-op. + */ +static void +blackholeGetForeignRelSize(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid) { + + // FIXME - this should be available... somewhere... + const JNI_FDW_Table table = NULL; + JNI_FDW_Plan plan_state; + + elog(DEBUG1, "entering function %s", __func__); + + plan_state = table->newPlan(root, baserel, foreigntableid); + baserel->fdw_private = (void *) plan_state; + + baserel->rows = plan_state->rows; + + /* initialize required state in plan_state */ + +} + +/** + * SELECT: Called to find the location of the foreign table's resources. + */ +static void +blackholeGetForeignPaths(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid) { + + /* + * BlackholeFdwPlanState *plan_state = baserel->fdw_private; + */ + + Cost startup_cost, + total_cost; + + elog(DEBUG1, "entering function %s", __func__); + + startup_cost = 0; + total_cost = startup_cost + baserel->rows; + + /* Create a ForeignPath node and add it as only possible path */ + add_path(baserel, (Path *) + create_foreignscan_path(root, baserel, + NULL, /* default pathtarget */ + baserel->rows, +#if (PG_VERSION_NUM >= 180000) + 0, /* no disabled nodes */ +#endif + startup_cost, + total_cost, + NIL, /* no pathkeys */ + NULL, /* no outer rel either */ + NULL, /* no extra plan */ +#if (PG_VERSION_NUM >= 170000) + NIL, /* no fdw_restrictinfo list */ +#endif + NIL)); /* no fdw_private data */ +} + +/** + * SELECT: Called to plan a foreign scan. + */ +static FdwPlan * +blackholePlanForeignScan(Oid foreigntableid, PlannerInfo *root, RelOptInfo *baserel) { + FdwPlan *fdwplan; + fdwplan = makeNode(FdwPlan); + fdwplan->fdw_private = NIL; + fdwplan->startup_cost = 0; + fdwplan->total_cost = 0; + return fdwplan; +} + +/** + * SELECT: Called before the first tuple has been retrieved. It allows + * last-second validation of the parameters. + */ +static void +blackholeBeginForeignScan(ForeignScanState *node, int eflags) { + // FIXME: how to get JNI_FDW_table? + const JNI_FDW_Table *table = NULL; + const JNI_FDW_ScanState table->newScan(table, node, eflags); + + elog(DEBUG1, "entering function %s", __func__); + + // I'm not sure if this is called before or after test below... + scan_state = table->begin(table, node, eflags); + + if (eflags & EXEC_FLAG_EXPLAIN_ONLY) { + return; + } + + node->fdw_state = scan_state; +} + +/** + * SELECT: Called to retrieve each tuple in the foreign table. + * Note: the external resource must be opened in this function. + */ +static TupleTableSlot * +blackholeIterateForeignScan(ForeignScanState *node) { + const TableTupleType slot = node->ss.ss_ScanTupleSlot(); + const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); + + elog(DEBUG1, "entering function %s", __func__); + + // is this EXPLAIN_ONLY ? + if (scan_state == NULL) { + return; + } + + ExecClearTuple(slot); + scan_state->next(scan_state, slot); + + // additional processing? + + return slot; +} + +/** + * SELECT: Called to reset internal state to initial conditions. + */ +static void +blackholeReScanForeignScan(ForeignScanState *node) { + const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); + + elog(DEBUG1, "entering function %s", __func__); + + // is this EXPLAIN_ONLY ? + if (scan_state == null) { + return; + } + + scan_state->reset(scan_state); +} + +/* + * SELECT: Called after the last row has been returned. + */ +static void +blackholeEndForeignScan(ForeignScanState *node) { + const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); + + elog(DEBUG1, "entering function %s", __func__); + + scan_state->close(scan_state); + // scan->table.removeScan(scan); ??? + + pfree(scan_state); +} + +/** + * Called when EXPLAIN is executed. This allows us to provide + * Wrapper, Server, and Table options like URLs, etc. + */ +static void +blackholeExplainForeignScan(ForeignScanState *node, + struct ExplainState *es) { + + elog(DEBUG1, "entering function %s", __func__); + +} + +/** + * Called when ANALYZE is executed on a foreign table. + */ +static bool +blackholeAnalyzeForeignTable(Relation relation, + AcquireSampleRowsFunc *func, + BlockNumber *totalpages) { + + elog(DEBUG1, "entering function %s", __func__); + + return false; +} + +/** + * Called when two or more foreign tables are on the same foreign server. + */ +static void +blackholeGetForeignJoinPaths(PlannerInfo *root, + RelOptInfo *joinrel, + RelOptInfo *outerrel, + RelOptInfo *innerrel, + JoinType jointype, + JoinPathExtraData *extra) { + + elog(DEBUG1, "entering function %s", __func__); +} + +/** + * LOCK-AWARE - called to re-fetch a tuple from a foreign table + */ +#if (PG_VERSION_NUM >= 120000) +static void blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + TupleTableSlot *slot, + bool *updated) +#else + +static HeapTuple +blackholeRefetchForeignRow(EState *estate, + ExecRowMark *erm, + Datum rowid, + bool *updated) +#endif +{ + + elog(DEBUG1, "entering function %s", __func__); + +#if (PG_VERSION_NUM < 120000) + return NULL; +#endif +} + + +/* + * Called when IMPORT FOREIGN SCHEMA is executed. + */ +static List * +blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, + Oid serverOid) { + + elog(DEBUG1, "entering function %s", __func__); + + return NULL; +} diff --git a/pljava-so/src/main/include/pljava/FDW.h b/pljava-so/src/main/include/pljava/FDW.h new file mode 100644 index 000000000..f0a6eeeb0 --- /dev/null +++ b/pljava-so/src/main/include/pljava/FDW.h @@ -0,0 +1,23 @@ +/** + * Actual implementation of minimal FDW based on the 'blackhole_fdw' + * project. For simplicity all of the comments and unused functions + * have been removed. + * + * The purpose of this file is to demonstrate the ability of C-based + * FDW implementation to successfully interact with a java object + * that implements the FDW interfaces. + * + * The first milestone is simply sending a NOTICE from the java + * method. + */ +#ifndef PLJAVA_SO_BLACKHOLEFDW_H +#define PLJAVA_SO_BLACKHOLEFDW_H + +// temporary name... +extern Datum blackhole_fdw_handler(PG_FUNCTION_ARGS); +extern Datum blackhole_fdw_validator(PG_FUNCTION_ARGS); + +typedef struct JNI_FDW_Validator JNI_FDW_Validator; +JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname); + +#endif //PLJAVA_SO_BLACKHOLEFDW_H diff --git a/pljava-so/src/main/resources/fdw/blackhole_fdw.control b/pljava-so/src/main/resources/fdw/blackhole_fdw.control new file mode 100644 index 000000000..913cede60 --- /dev/null +++ b/pljava-so/src/main/resources/fdw/blackhole_fdw.control @@ -0,0 +1,5 @@ +# blackhole FDW +comment = 'Blackhole Foreign Data Wrapper' +default_version = '1.9.6' +module_pathname = '$libdir/libpljava.so' +relocatable = true diff --git a/pljava-so/src/main/resources/fdw/sql/blackhole_fdw.sql b/pljava-so/src/main/resources/fdw/sql/blackhole_fdw.sql new file mode 100644 index 000000000..82356afde --- /dev/null +++ b/pljava-so/src/main/resources/fdw/sql/blackhole_fdw.sql @@ -0,0 +1,31 @@ +* +* foreign-data wrapper blackhole + * + * Copyright (c) 2013, PostgreSQL Global Development Group + * + * This software is released under the PostgreSQL Licence + * + * Author: Andrew Dunstan + * + * IDENTIFICATION + * blackhole_fdw/=sql/blackhole_fdw.sql + * + *------------------------------------------------------------------------- + */ + +CREATE FUNCTION blackhole_fdw_handler() + RETURNS fdw_handler +AS '$libdir/blackhole_fdw' +LANGUAGE C STRICT; + +CREATE FUNCTION blackhole_fdw_validator(text[], oid) + RETURNS void +AS '$libdir/blackhole_fdw' +LANGUAGE C STRICT; + +CREATE +FOREIGN DATA WRAPPER blackhole_fdw + HANDLER blackhole_fdw_handler + VALIDATOR blackhole_fdw_validator; + +-- CREATE EXTENSION blackhole_fdw; From da105381b1cc5430897fd08e211d4df38f7d54fd Mon Sep 17 00:00:00 2001 From: Bear Giles Date: Sun, 1 Jun 2025 20:23:11 -0600 Subject: [PATCH 2/2] Fleshed out FDW a bit. Fleshed out FDW a bit. On the pljava-so side it fails to load (in the docker test) because of a dependency on GLIBC_2.38. I hadn't hit this with my sister project because it was much simplier and didn't introduce additional dependencies. I also found the bits of backend code that lets us access all five types of options at the points where we need them. It's forcing me to continue to rethink some of my assumptions, e.g., there's definitely a need to have a conceptual difference between FDW, server, and table, but it looks like nearly all of the required functions only require a relation or table. Could the solution be having the java classes responsible for executing 'CREATE FOREIGN DATA WRAPPER ...' etc? I had assumed the constructors would need to be called from the backend - like all of the callbacks - but now think that's backwards. This would makes it much easier to ensure the options contain the correct classnames and method signatures. :-) On the java side I have a big chunk of the API and thin implementation written. I haven't had a chance to introduce some changes I mentioned earlier - ones that will provide a much better abstraction between the FDW-specific bits and the actual implementation. That was because I hoped to start seeing immediate feedback as I modified a working (but minimal) FDW. The Dockerfile under pljava-so had always been a bit of a stopgap measure - a way to do a really quick sanity check when modifying the backend code - but with the extension additions to the api and examples modules the docker creation should probably be moved to 'packaging' anyway. At that point it can continue to use the official postgresql images but use the locally build .so, jars, etc., instead of trying to sneak in changes to a preconfigured docker image. --- .../pljava/fdw/FDWExplainState.java | 9 + .../pljava/fdw/FDWForeignDataWrapper.java | 36 +- .../pljava/fdw/FDWForeignTable.java | 117 ++- .../postgresql/pljava/fdw/FDWPlanState.java | 7 +- .../postgresql/pljava/fdw/FDWScanState.java | 51 +- .../org/postgresql/pljava/fdw/FDWServer.java | 48 +- .../org/postgresql/pljava/fdw/FDWUser.java | 118 +++ .../postgresql/pljava/fdw/FDWValidator.java | 56 +- pljava-examples/pom.xml | 8 +- .../example/fdw/BlackholeForeignTable.java | 47 +- .../example/fdw/BlackholeScanState.java | 61 +- pljava-so/Dockerfile | 12 +- pljava-so/src/main/c/BlackholeFDW.c | 988 +++++++++--------- pljava-so/src/main/c/BlackholeFDWJNI.c | 253 +++++ pljava-so/src/main/include/pljava/FDW.h | 25 +- 15 files changed, 1300 insertions(+), 536 deletions(-) create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWExplainState.java create mode 100644 pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWUser.java create mode 100644 pljava-so/src/main/c/BlackholeFDWJNI.c diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWExplainState.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWExplainState.java new file mode 100644 index 000000000..db1fbbd1a --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWExplainState.java @@ -0,0 +1,9 @@ +package org.postgresql.pljava.fdw; + +/** + * Additional content for `EXPLAIN x...` + * + * No details yet. + */ +public interface FDWExplainState { +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java index 519f31488..97a26efe3 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignDataWrapper.java @@ -1,5 +1,39 @@ package org.postgresql.pljava.fdw; +import java.util.Collections; +import java.util.Map; + +/** + * The Foreign Data Wrapper. + * + * This is the highest-level abstraction, e.g., for information + * contained in S3 files. + * + * It could also capture an abstract concept, e.g., one FDW + * to capture multiple authentication implementations. + * + * There may be multiple instances of a single FOREIGN DATA WRAPPER. + */ public interface FDWForeignDataWrapper { - FDWServer getServer(); + + /** + * The instances unique ID. It should be used to maintain a cache. + * @return + */ + default Long getId() { return null; } + + /** + * Return a copy of the options provided to `CREATE FOREIGN DATA WRAPPER...` + * @return + */ + default Map getOptions() { return Collections.emptyMap(); }; + + /** + * Validate a set of options against an existing instance. There should be + * a similar static method before creating a new instance. + * + * @param options + * @return + */ + default boolean validateOptions(Map options) { return true; }; } diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java index 513696f1d..9cdfe3d5c 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWForeignTable.java @@ -1,12 +1,117 @@ package org.postgresql.pljava.fdw; +import java.sql.ResultSetMetaData; +import java.util.Collections; +import java.util.Map; + +/** + * The Foreign Table + * + * This is the lowest-level abstraction, e.g., a specific + * S3 file. + * + * There may be multiple instances of a Foreign Table + * for a single Foreign Server. + */ public interface FDWForeignTable { - FDWPlanState newPlanState(); - FDWScanState ScanState(); - default boolean updatable() { return false; } + /** + * The instances unique ID. It should be used to maintain a cache. + */ + default Long getId() { return null; } + + /** + * Return a copy of the options provided to `CREATE FOREIGN TABLE...` + * @return + */ + default Map getOptions() { return Collections.emptyMap(); }; + + /** + * Validate a set of options against an existing instance. There should be + * a similar static method before creating a new instance. + * + * @param options + * @return + */ + default boolean validateOptions(Map options) { return true; }; + + /** + * Create an object used for query planning. + * + * @param user + * @return + */ + FDWPlanState newPlanState(FDWUser user); + + /** + * Create an object used for SELECT statements. + * @param user + * @return + */ + FDWScanState newScanState(FDWUser user, boolean explainOnly); + + // values from PlannerInfo *root, RelOptInfo *baserel + // BUT NOT foreigntableoid + void getRelSize(); + +/* + static void blackholeGetForeignPaths(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid); +*/ + + // values from PlannerInfo *root, RelOptInfo *baserel + // BUT NOT foreigntableoid + void getForeignPaths(); + + /** + * Is this table updatable by this user? + * + * @param user + * @return + */ + default boolean isUupdatable(FDWUser user) { return false; } + + /** + * Does this table support concurrent access? + * @return + */ + default boolean supportsConcurrency() { return false; } + + /** + * Does this table support asynchronous queries? + * @return + */ + default boolean supportsAsyncOperations() { return false; } + + /** + * Collect statistics used by the query optimizer. + * This can be supported for read-only tables. + * + * Details TBD + */ + default void analyze() { } + + /** + * Compact the data, if appropriate. + * This should be a noop for read-only tables. + * + * Details TBD. + */ + default void vacuum() { } + + /** + * Get the table's schema. This information + * will be used when executing `IMPORT FOREIGN SCHEMA...` + * + * @return + */ + default ResultSetMetaData getMetaData() { return null; } - default void analyze() { }; -// -// default void vacuum() { }; + /** + * Estimate the number of rows. + * + * @return + */ + default long getRows() { return 0; } } diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java index f9c00a69f..6ffbee457 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWPlanState.java @@ -1,9 +1,14 @@ package org.postgresql.pljava.fdw; public interface FDWPlanState { + + // values from PlannerInfo *root, RelOptInfo *baserel. + // the PlannerInfo is only used in advanced queries. void open(); void close(); - // int rows(); + default long getRows() { return 0; } + + default FDWUser getUser() { return null; } } diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java index e32d6a4fb..cd0dfaffd 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWScanState.java @@ -1,10 +1,55 @@ package org.postgresql.pljava.fdw; +import java.util.Map; + public interface FDWScanState { - void open(); - void next(Object slot); + /** + * The database has already performed an initial check for + * the user's permission to execute SELECT - but our java + * code may impose additional requirements. + * + * The minimal implementation just adds the ability to check + * whether * the user is authorized. (The current FDWUser is + * transparently * passed to the appropriate method.) + * + * A more advanced implementation would allow us to + * add row and column filtering beyond what will already + * be done by the database. + */ + default boolean isAuthorizedUser() { return true; } + + /** + * Verify that we have a valid configuration. + * + * No external resources should be accessed if + * the `explainOnly` flag is true. (It's okay to + * check a file exists and is readable but it should + * not be opened. A REST service can have its hostname + * verified but it should not be called. + * + * If the `explainOnly` flag is false than external + * resources can be accessed in order to verify + * that it's a valid URL and we have valid credentials. + * However all external resources should be released + * before this method exits. + */ + void open(boolean explainOnly); + + // values from TableTupleType. It is an element + // of the ForeignScanState mentioned above. + Map next(); + + /** + * Reset scan to initial state. + */ void reset(); + + /** + * Release resources + */ void close(); - // void explain(); ?? + default FDWExplainState explain() { return null; } + + default FDWUser getUser() { return null; } } diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java index d9168e97f..70aa4b754 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWServer.java @@ -1,10 +1,50 @@ package org.postgresql.pljava.fdw; -import java.sql.ResultSetMetaData; +import java.sql.DatabaseMetaData; +import java.util.Collections; +import java.util.Map; +/** + * The Foreign Server + * + * This is the middle-level abstraction, e.g., a specific + * AWS account with access to the required resources. + * + * There may be multiple instances of a Foreign Server + * for a single Foreign Data Wrapper. + */ public interface FDWServer { - FDWForeignTable getForeignTable(); - // For 'importSchemaStmt() - ResultSetMetaData getMetaData(); + /** + * The instances unique ID. It should be used to maintain a cache. + */ + default Long getId() { return null; } + + /** + * Return a copy of the options provided to `CREATE FOREIGN SERVER...` + * @return + */ + default Map getOptions() { return Collections.emptyMap(); }; + + /** + * Validate a set of options against an existing instance. There should be + * a similar static method before creating a new instance. + * + * @param options + * @return + */ + default boolean validateOptions(Map options) { return true; }; + + /** + * Get the server's entire schema. This can be useful + * information even if the backend only gets the + * schema for individual tables. + * + * (It's not clear since the backend struct supports + * foreign keys but I don't think the individual + * ResultSetMetadata includes that information.) + * + * @return + */ + default DatabaseMetaData getMetaData() { return null; } } diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWUser.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWUser.java new file mode 100644 index 000000000..a8c453a8e --- /dev/null +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWUser.java @@ -0,0 +1,118 @@ +package org.postgresql.pljava.fdw; + +import java.security.cert.Certificate; + +/** + * Placeholder for user information - it should use the + * existing pl/java class. + * + * The effective and real usernames can be retrieved from + * the OIDs. + * + * For sensitive material the java classes can implement + * additional AuthN and AuthZ functionality on their own. + * In this case it's also common for the AuthN and AuthZ + * to consider both how the user was authenticated and + * their apparent physical location. + * + * Some of this information is available if you have a + * JDBC Connection... but the whole point of this module + * is to hide the fact that this code is used by a database. + * It will be much harder to break the database if the java + * classes have absolutely no access to the connection, or + * even any awareness of its existence. + * + * Kerberos notes: + * + * - The java code should have access to the user's + * Kerberos principal. I know the principal -> username + * mapping is in pg_ident.conf but the java code should + * not rely on it. + * + * - Reminder that Kerberos provides secure authentication + * but it does not provide secure transport. You must explicitly + * add TLS for an encrypted connection. This is often + * overlooked by people unfamiliar with Kerberos and is + * why we need an explicit check for a secure connection. + */ +public interface FDWUser { // also implement Principal ?? + + // is there an actual default user?... + String DEFAULT_USER = "unknown_user"; + + enum AuthenticationMechanism { + TRUST, + REJECT, + MD5, + PASSWORD, + SCRAM_SHA_256, + GSS, // Kerberos + SSPI, + IDENT, + PEER, + PAM, + LDAP, + RADIUS, + CERT, + UNKNOWN + } + + /** + * The user's unique ID. It can be used to maintain a cache. + */ + default Long getOid() { return null; } + + /** + * The real user's unique ID. It can be used to maintain a cache. + */ + default Long getRealOid() { return null; } + + /** + * Get the effective database username. + * @return + */ + default String getUsername() { + return getRealUsername(); + }; + + /** + * Get the real database username. + * @return + */ + default String getRealUsername() { + return DEFAULT_USER; + } + + /** + * Is the connection secure? + * + * This may be superfluous since this information is + * already available via the `DatabaseMetaData` + * connection information. However I can't rule out + * the possibility of a desire to have more details + * about the connection, e.g., the algorithm used, + * the keysize, etc. + */ + default boolean isConnectionSecure() + { + return false; + } + + /** + * Get the authentication mechanism used. + */ + default AuthenticationMechanism getAuthenticationMechanism() + { + return AuthenticationMechanism.UNKNOWN; + } + + /** + * Get the user's location. (named socket, IP address + port) + */ + default Object getLocation() { return null; } + + /** + * Get the user's Certificate, if `cert` authentication was used. + */ + default Certificate getCertificate() { return null; } +} diff --git a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java index ff9a0de42..0aea55966 100644 --- a/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java +++ b/pljava-api/src/main/java/org/postgresql/pljava/fdw/FDWValidator.java @@ -1,9 +1,59 @@ package org.postgresql.pljava.fdw; +import java.util.Collections; +import java.util.Map; + +/** + * The Validator + * + * This class is used to validate the options provided to + * the FDWForeignDataWrapper, FDWForeignServer, and FDWForeignTable + * constructors. + * + * Note: the Foreign Data Wrapper or Foreign Server may already + * exist. If so they will be provided a copy of the appropriate + * options. + */ public interface FDWValidator { - void addOption(int relid, String key, String value); + enum Scope { + FOREIGN_DATA_WRAPPER, + FOREIGN_SERVER, + FOREIGN_TABLE + // plus two others... + }; + + /** + * Add an option + * + * @param scope + * @param key + * @param value + */ + default void addOption(Scope scope, String key, String value) { } - boolean validate(); + /** + * Get options + */ + default Map getOptions(Scope scope) + { + return Collections.emptyMap(); + } - FDWForeignDataWrapper getForeignDataWrapper(); + /** + * Validate all options. + * + * This method should create any missing objects and add + * them to an internal cache. + * + * @TODO - should the return value indicate where the validation + * failed? FDW, SERVER, TABLE, bad property or unable to create + * object? + * + * @param fdwId if for existing ForeignDataWrapper, or null + * @param srvId if for existing Server, or null + * @param ftId if for existing Foreign Table, or null + * + * @return true if successfully validated + */ + default boolean validate(long fdwId, Long srvId, Long ftId) { return true; } } diff --git a/pljava-examples/pom.xml b/pljava-examples/pom.xml index bc245281f..cd338df30 100644 --- a/pljava-examples/pom.xml +++ b/pljava-examples/pom.xml @@ -42,7 +42,13 @@ pljava-api ${project.version} - + + commons-collections + commons-collections + 3.2.2 + compile + + diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java index 89e3f477b..da8cc5fc2 100644 --- a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeForeignTable.java @@ -4,8 +4,12 @@ import org.postgresql.pljava.fdw.FDWPlanState; import org.postgresql.pljava.fdw.FDWScanState; import org.postgresql.pljava.fdw.FDWForeignTable; +import org.postgresql.pljava.fdw.FDWUser; +import java.sql.ResultSetMetaData; +import java.util.ArrayList; import java.util.Collections; +import java.util.List; import java.util.Map; import java.util.logging.Logger; @@ -18,6 +22,8 @@ public class BlackholeForeignTable implements FDWForeignTable { private static final Logger LOG = Logger.getLogger(BlackholeForeignTable.class.getName()); + private final List> dummyTable = new ArrayList<>(); + public BlackholeForeignTable() { this(Collections.emptyMap()); } @@ -27,20 +33,43 @@ public BlackholeForeignTable(Map options) { } @Override - public FDWPlanState newPlanState() { - LOG.info("getPlanState()"); - return new BlackholePlanState(this); + public void getRelSize() { + } + + /* + public void blackholeGetForeignPaths(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid) { } + */ @Override - public FDWScanState newScanState() { - LOG.info("newScanState()"); - return new BlackholeScanState(this); + public FDWPlanState newPlanState(FDWUser user) { + return null; } @Override - public boolean updatable() { - LOG.info("updatable()"); - return false; + public FDWScanState newScanState(FDWUser user, boolean explainOnly) { + return null; } + + /* + @Override + public boolean isUpdatable(FDWUser user); + + @Override + public boolean supportsConcurrency(); + + @Override + public boolean supportsAsyncOperation(); + + @Override + public void analyze(); + + @Override + public void vacuum(); + + @Override + public ResultSetMetaData getMetaData(); + */ } diff --git a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java index 98a2c924b..3a35677cf 100644 --- a/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java +++ b/pljava-examples/src/main/java/org/postgresql/pljava/example/fdw/BlackholeScanState.java @@ -1,7 +1,13 @@ package org.postgresql.pljava.example.fdw; import org.postgresql.pljava.fdw.FDWScanState; +import org.postgresql.pljava.fdw.FDWUser; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; +import java.util.Map; import java.util.logging.Logger; /** @@ -10,30 +16,75 @@ public class BlackholeScanState implements FDWScanState { private static final Logger LOG = Logger.getLogger(BlackholeScanState.class.getName()); - private final BlackholeForeignTable table; + private final FDWUser user; + private final List> data; - public BlackholeScanState(BlackholeForeignTable table) { + private Iterator> iter; + private boolean isOpen = false; + + public BlackholeScanState(FDWUser user, List> data) { LOG.info("constructor()"); - this.table = table; + + this.user = user; + + // create copy + this.data = new ArrayList> (data); + this.iter = null; } @Override - public void open() { + public void open(boolean explainOnly) { LOG.info("open()"); + if (this.isOpen) { + LOG.info("unexpected state!"); + } + else if (!isAuthorizedUser()) + { + LOG.info("unauthorized user"); + } + else if (!explainOnly) + { + // open file, read it, ... + this.iter = this.data.iterator(); + this.isOpen = true; + } } @Override - public void next(Object slot) { + public Map next() { LOG.info("next()"); + if (this.isOpen) { + if (iter.hasNext()) { + return iter.next(); + } else { + return Collections.emptyMap(); + } + } else { + // what about 'explain only?' + LOG.info("unexpected state!"); + } } @Override public void reset() { LOG.info("reset()"); + if (this.isOpen) { + this.iter = this.data.iterator(); + } else { + // what about 'explain only?' + LOG.info("unexpected state!"); + } } @Override public void close() { LOG.info("close()"); + if (!this.isOpen) { + // don't do anything else... + LOG.info("unexpected state!"); + } + + this.iter = null; + this.isOpen = false; } } diff --git a/pljava-so/Dockerfile b/pljava-so/Dockerfile index 1229418d6..788590971 100644 --- a/pljava-so/Dockerfile +++ b/pljava-so/Dockerfile @@ -1,27 +1,27 @@ # TODO: pull various VERS from the environment... -FROM postgres:17.2-bookworm +FROM postgres:16-bookworm LABEL authors="Bear Giles " ENV TARGET=target ENV RESOURCES=src/main/resources # can/should be set as build property... -ENV PG_VERS=17 +ENV PG_VERS=16 ENV LIBDIR=/usr/lib/postgresql/${PG_VERS}/lib ENV EXTDIR=/usr/share/postgresql/${PG_VERS}/extension -ENV SO_NAME=pljava.so +ENV SO_NAME=pljava ENV FDW_NAME=blackhole_fdw -ENV FDW_VERS=1.9.6 +ENV FDW_VERS=1.6.9 # this will install the standard version. It can be updated once the docker image is running in a test environment. RUN apt-get update && apt-get install -y postgresql-${PG_VERS}-pljava postgresql-${PG_VERS}-pljava-dbgsym -COPY ${TARGET}/${SO_NAME}.so ${LIBDIR}/${SO_NAME}.so +COPY ${TARGET}/pljava-pgxs/lib${SO_NAME}-so-2-SNAPSHOT.so ${LIBDIR}/lib${SO_NAME}-so-${FDW_VERS}.so COPY ${RESOURCES}/fdw/${FDW_NAME}.control ${EXTDIR}/ COPY ${RESOURCES}/fdw/sql/${FDW_NAME}*.sql ${EXTDIR}/ -# ENTRYPOINT ["top", "-b"] \ No newline at end of file +# ENTRYPOINT ["top", "-b"] diff --git a/pljava-so/src/main/c/BlackholeFDW.c b/pljava-so/src/main/c/BlackholeFDW.c index 9e1e04051..531040462 100644 --- a/pljava-so/src/main/c/BlackholeFDW.c +++ b/pljava-so/src/main/c/BlackholeFDW.c @@ -11,8 +11,10 @@ * method. */ #include "postgres.h" +#include "postgres_ext.h" #include "access/reloptions.h" +#include "commands/explain.h" #include "foreign/fdwapi.h" #include "foreign/foreign.h" #include "optimizer/pathnode.h" @@ -21,23 +23,62 @@ #include "../include/pljava/FDW.h" -PG_MODULE_MAGIC; +// PG_MODULE_MAGIC; #if (PG_VERSION_NUM < 90500) // fail... #endif +/** + * Sidenotes: + * + * PlanState also provides: + * - Instrumentation *instrument; // Optional runtime stats for this node + * - WorkerInstrumentation *worker_instrument; // per-worker instrumentation + */ + +// this needs to be known BEFORE we execute 'CREATE FOREIGN DATA WRAPPER...' +// static const char* FDW_validator_classname = "org/postgresql/pljava/fdw/BlackholeValidator"; + /* - * SQL functions + * Notes: copied from fdwapi.h + * + * extern Oid GetForeignServerIdByRelId(Oid relid); + * extern bool IsImportableForeignTable(const char *tablename, + * ImportForeignSchemaStmt *stmt); + * extern Path *GetExistingLocalJoinPath(RelOptInfo *joinrel); + * + * extern FdwRoutine *GetFdwRoutine(Oid fdwhandler); + * extern FdwRoutine *GetFdwRoutineByServerId(Oid serverid); + * extern FdwRoutine *GetFdwRoutineByRelId(Oid relid); + * extern FdwRoutine *GetFdwRoutineForRelation(Relation relation, bool makecopy); + * + * And from foreign.h + * + * extern ForeignServer *GetForeignServer(Oid serverid); + * extern ForeignServer *GetForeignServerExtended(Oid serverid, + * bits16 flags); + * extern ForeignServer *GetForeignServerByName(const char *srvname, + * bool missing_ok); + * extern UserMapping *GetUserMapping(Oid userid, Oid serverid); + * extern ForeignDataWrapper *GetForeignDataWrapper(Oid fdwid); + * extern ForeignDataWrapper *GetForeignDataWrapperExtended(Oid fdwid, + * bits16 flags); + * extern ForeignDataWrapper *GetForeignDataWrapperByName(const char *fdwname, + * bool missing_ok); + * extern ForeignTable *GetForeignTable(Oid relid); + * + * extern List *GetForeignColumnOptions(Oid relid, AttrNumber attnum); + * + * extern Oid get_foreign_data_wrapper_oid(const char *fdwname, bool missing_ok); + * extern Oid get_foreign_server_oid(const char *servername, bool missing_ok); */ extern Datum blackhole_fdw_handler(PG_FUNCTION_ARGS); - extern Datum blackhole_fdw_validator(PG_FUNCTION_ARGS); PG_FUNCTION_INFO_V1(blackhole_fdw_handler); PG_FUNCTION_INFO_V1(blackhole_fdw_validator); - /* callback functions */ static void blackholeGetForeignRelSize(PlannerInfo *root, RelOptInfo *baserel, @@ -47,7 +88,13 @@ static void blackholeGetForeignPaths(PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid); -static FdwPlan *blackholePlanForeignScan(Oid foreigntableid, PlannerInfo *root, RelOptInfo *baserel); +static ForeignScan *blackholeGetForeignPlan(PlannerInfo *root, + RelOptInfo *rel, + Oid foreigntableid, + ForeignPath *best_path, + List *tlist, + List *restrictinfo_list, + Plan *outer_plan); static void blackholeBeginForeignScan(ForeignScanState *node, int eflags); @@ -59,60 +106,20 @@ static void blackholeReScanForeignScan(ForeignScanState *node); static void blackholeEndForeignScan(ForeignScanState *node); /* everything below here is optional */ - static int blackholeIsForeignRelUpdatable(Relation rel); +static void blackholeExplainForeignScan(ForeignScanState *node, ExplainState *es); +static List *blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, Oid serverOid); +static bool blackholeIsForeignScanParallelSafe(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); +static bool blackholeIsForeignPathAsyncCapable(ForeignPath *path); -// TODO: locate 'ExplainState' -// static void blackholeExplainForeignScan(ForeignScanState *node, struct ExplainState *es); - -#if (PG_VERSION_NUM >= 120000) -static void blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - TupleTableSlot *slot, - bool *updated); -#else - -static HeapTuple blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - bool *updated); - -#endif - -static List *blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, - Oid serverOid); - -#endif - -static bool blackholeAnalyzeForeignTable(Relation relation, - AcquireSampleRowsFunc *func, - BlockNumber *totalpages); - -#if (PG_VERSION_NUM >= 120000) -static void blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - TupleTableSlot *slot, - bool *updated); -#else - -static HeapTuple blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - bool *updated); - -#endif /* ------------------------------------------------------------ * The POSTGRESQL Functions * -----------------------------------------------------------*/ -// this needs to be known BEFORE we execute 'CREATE FOREIGN DATA WRAPPER...' -static const char FDW_validator_classname = "org/postgresql/pljava/fdw/BlackholeValidator"; -static const char FDW_handler_classname = "org/postgresql/pljava/fdw/BlackholeHandler"; // ??? Datum -blackhole_fdw_handler(PG_FUNCTION_ARGS) { +blackhole_fdw_handler(PG_FUNCTION_ARGS) +{ FdwRoutine *fdwroutine = makeNode(FdwRoutine); elog(DEBUG1, "entering function %s", __func__); @@ -140,6 +147,14 @@ blackhole_fdw_handler(PG_FUNCTION_ARGS) { /* remainder are optional - use NULL if not required */ /* support for insert / update / delete */ fdwroutine->IsForeignRelUpdatable = blackholeIsForeignRelUpdatable; + + /* Support for scanning foreign joins */ + fdwroutine->GetForeignJoinPaths = NULL; + + /* Functions for remote upper-relation (post scan/join) planning */ + fdwroutine->GetForeignUpperPaths = NULL; + + /* Functions for modifying foreign tables */ fdwroutine->AddForeignUpdateTargets = NULL; /* U D */ fdwroutine->PlanForeignModify = NULL; /* I U D */ fdwroutine->BeginForeignModify = NULL; /* I U D */ @@ -148,557 +163,546 @@ blackhole_fdw_handler(PG_FUNCTION_ARGS) { fdwroutine->ExecForeignDelete = NULL; /* D */ fdwroutine->EndForeignModify = NULL; /* I U D */ + /* Next-Generation functions for modifying foreign tables? */ + fdwroutine->PlanDirectModify = NULL; + fdwroutine->BeginDirectModify = NULL; + fdwroutine->IterateDirectModify = NULL; + fdwroutine->EndDirectModify = NULL; + + /* Support for SELECT FOR UPODATE/SHARE row locking */ + fdwroutine->GetForeignRowMarkType = NULL; + fdwroutine->RefetchForeignRow = NULL; + fdwroutine->RecheckForeignScan = NULL; + /* support for EXPLAIN */ - // fdwroutine->ExplainForeignScan = blackholeExplainForeignScan; /* EXPLAIN S U D */ - fdwroutine->ExplainForeignScan = NULL; /* EXPLAIN S U D */ + fdwroutine->ExplainForeignScan = blackholeExplainForeignScan; /* EXPLAIN S U D */ fdwroutine->ExplainForeignModify = NULL; /* EXPLAIN I U D */ + fdwroutine->ExplainDirectModify = NULL; - /* support for ANALYSE */ - fdwroutine->AnalyzeForeignTable = blackholeAnalyzeForeignTable; /* ANALYZE only */ + /* Support functions for ANALYZE */ + fdwroutine->AnalyzeForeignTable = NULL; /* ANALYZE only */ /* Support functions for IMPORT FOREIGN SCHEMA */ fdwroutine->ImportForeignSchema = blackholeImportForeignSchema; - /* Support for scanning foreign joins */ - fdwroutine->GetForeignJoinPaths = NULL; - - /* Support for locking foreign rows */ - fdwroutine->GetForeignRowMarkType = NULL: - fdwroutine->RefetchForeignRow = blackholeRefetchForeignRow; + /* Support functions for TRUNCATE */ +#if (PG_VERSION_NUM >= 14000) + fdwroutine->ExecForeignTruncate = NULL; +#endif - // none of the newer functions are handled yet - they deal with 'direct' access, concurrency, and async. + /* Support functions for parallelism under Gather node */ + fdwroutine->IsForeignScanParallelSafe = blackholeIsForeignScanParallelSafe; + fdwroutine->EstimateDSMForeignScan = NULL; + fdwroutine->InitializeDSMForeignScan = NULL; + fdwroutine->ReInitializeDSMForeignScan = NULL; + fdwroutine->InitializeWorkerForeignScan = NULL; + fdwroutine->ShutdownForeignScan = NULL; + + /* Support functions for path reparameterization. */ + fdwroutine->ReparameterizeForeignPathByChild = NULL; + +#if (PG_VERSION_NUM >= 14000) + /* Support functions for asynchronous execution */ + fdwroutine->IsForeignPathAsyncCapable = blackholeIsForeignPathAsyncCapable; + fdwroutine->ForeignAsyncRequest = NULL; + fdwroutine->ForeignAsyncConfigureWait = NULL; + fdwroutine->ForeignAsyncNotify = NULL; +#endif PG_RETURN_POINTER(fdwroutine); } Datum -blackhole_fdw_validator(PG_FUNCTION_ARGS) { +blackhole_fdw_validator(PG_FUNCTION_ARGS) +{ List *options_list = untransformRelOptions(PG_GETARG_DATUM(0)); elog(DEBUG1, "entering function %s", __func__); - /* make sure the options are valid */ + /* collect options */ - /* no options are supported */ - JNIEnv *env = NULL; - JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname); - - errhint("Blackhole FDW does not support any options"))); + /* validate them */ +#ifdef USE_JAVA + JNI_FDW_Validator newValidator = call constructor (static method) +#endif - PG_RETURN_POINTER(fdwroutine); + if (list_length(options_list) > 0) + ereport(ERROR, + (errcode(ERRCODE_FDW_INVALID_OPTION_NAME), + errmsg("invalid options"), + errhint("Simple FDW does not support any options"))); PG_RETURN_VOID(); } -/* - - - I know from context that the Foreign Data Wrapper, Server, and Foreign Table have OIDs associated with them... - -Datum -blackhole_fdw_server(PG_FUNCTION_ARGS) { - FdwServer *fdwserver = makeNode(FdwServer); - PG_RETURN_POINTER(fdwserver); -} - -Datum -blackhole_fdw_table(PG_FUNCTION_ARGS) { - FdwTable *fdwtable = makeNode(FdwTable); - PG_RETURN_POINTER(fdwtable); -} - */ - -/* ------------------------------------------------------------ - * The JNI headers - * ------------------------------------------------------------*/ - -static typedef struct JNI_FDW_Wrapper JNI_FDW_Wrapper; -static typedef struct JNI_FDW_Server JNI_FDW_Server; -static typedef struct JNI_FDW_Table JNI_FDW_Table; -static typedef struct JNI_FDW_PlanState JNI_FDW_PlanState; -static typedef struct JNI_FDW_ScanState JNI_FDW_ScanState; - -static JNI_FDW_Wrapper *validator_get_wrapper(JNI_FDW_Validator *validator); -static JNI_FDW_Server *wrapper_get_server(JNI_FDW_Wrapper *wrapper); -static JNI_FDW_Table *server_get_table(JNI_FDW_Server *server); -static JNI_FDW_PlanState *table_new_plan(JNI_FDW_Table *table); -static JNI_FDW_ScanState *table_new_scan(JNI_FDW_Table *table - -// not all functions... - -static void validator_add_option(JNI_FDW_Validator *, int relid, String key, String value); -static bool validator_validate(JNI_FDW_Validator *); - -static JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table); -static void plan_open(JNI_FDW_Table *table, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid); -static void plan_close(JNI_FDW_PlanState *plan_state); - -static void scan_open(JNI_FDW_ScanState *scan_state, ForeignScanState *node, int eflag); -static void scan_next(JNI_FDW_ScanState *scan_state, Slot *slot); -static void scan_reset(JNI_FDW_ScanState *scan_state); -static void scan_close(JNI_FDW_ScanState *scan_state); - -static JNI_FDW_ScanState *table_new_scanPlan(JNI_FDW_Table *table); - -// Note: this does not do memory management yet! - -static -jmethodId getMethodId(JNIEnv *env, jclass class, ...) +static void +blackholeShowInfo(Oid foreigntableid, RelOptInfo *rel) { - return env->GetMethodIdw(class, vargargs); + // rel->serverid; + // rel->userid; // may be InvalidOid = current user) + // rel->useriscurrent + // rel->fdwroutine + // rel->fdw_private + + // rel->rows + // rel->relid (only base rel, not joins) + // rel->min_attr (often <0) + // rel->max_attr; + + // ForeignTable *table = GetForeignTableExtended(foreigntableid, flags); // expects relid? + // ForeignServer *server = GetForeignServerExtended(table->serverid, flags); + + // ForeignTable *table = GetForeignTable(rel->relid); // expects relid? + // ForeignServer *server = GetForeignServer(table->serverid); + // ForeignDataWrapper *fdw = GetForeignDataWrapper(server->fdwid); + + // List *ftOptions = table->options; + // List *srvOptions = server->options; + // List *fdwOptions = fdw->options; + // List *userOptions = um->options; + // List *columnOptions = GetForeignColumnOptions(rel->relid, (AttrNumber) 0); + + // macro + // char *username = MappingUserName(userid) + + // Oid serverId = svr->serverid; + // Oid fdwid = svr->fdwid; + // Oid srvOwnerId = svr->ownerid; + // char *servername = svr->servername; + // char *serverType = svr->servertype; // optional + // char *serverVersion = svr->serverversion; // optional + + // Oid fwdid = fdw->fdwid; + // Oid owner = fdw->owner; + // char *fdwname = fdw->fdwname; + // Oid fdwhandler = fdw->fdwhandler; + // Oid fdwvalidator = fdw->fdwvalidator; + + // Oid usermappingoid = um->umid; + // Oid userId = um->userId; + // Oid umServerId = um->serverId; + + // wrapper->options; + // wrapper->owner; } -/** - * Public: fdwvalidator method - */ -static -typedef struct { - void (*addOption)(JNI_FDW_Validator *, int, const char *, const char *) = validator_add_option; - bool (*validate)(JNI_FDW_Validator *) = validator_validate; - JNI_FDW_Wrapper *(*get_wrapper)(JNI_FDW_Validator *) = validator_get_wrapper; - - const JNIEnv *env; - const jclass validatorClass; - const jobject instance; -} JNI_FDW_Validator; +static void +blackholeGetForeignRelSize(PlannerInfo *root, + RelOptInfo *baserel, + Oid foreigntableid) { + /*1 + * Obtain relation size estimates for a foreign table. This is called at + * the beginning of planning for a query that scans a foreign table. root + * is the planner's global information about the query; baserel is the + * planner's information about this table; and foreigntableid is the + * pg_class OID of the foreign table. (foreigntableid could be obtained + * from the planner data structures, but it's passed explicitly to save + * effort.) + * + * This function should update baserel->rows to be the expected number of + * rows returned by the table scan, after accounting for the filtering + * done by the restriction quals. The initial value of baserel->rows is + * just a constant default estimate, which should be replaced if at all + * possible. The function may also choose to update baserel->width if it + * can compute a better estimate of the average result row width. + */ -JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname) { - JNI_FDW_Validator *validator = (JNI_FDW_Validator *) palloc0(sizeof JNI_FDW_Validator); + elog(NOTICE, "entering function %s", __func__); - validator->env = env; - validator->validatorClass = env->FindClass(validator_classname); - validator->instance = env->AllocObject(fdw->validatorClass); +#ifdef USE_JAVA + // JAVA CONSTRUCTOR based on foreigntable id + baserel->fdw_private = JNI_getForeignRelSize(user, root, baserel); - return validator; + // JAVA METHOD + /* initialize required state in plan_state */ + (JNI_FDW_PlanState *baserel->plan_state)->open(plan_state); +#endif + // baserel->rows = plan_state->rows; } -/* FIXME - how to handle 'handler' since the 'create foreign wrapper' requires it but we start with the validator? */ - -/* FIXME - how to we pass wrapper, server, and table options to each? We have them in the Validator function...*/ +static void +blackholeGetForeignPaths(PlannerInfo *root, + RelOptInfo *rel, + Oid foreigntableid) { + /* + * Create possible access paths for a scan on a foreign table. This is + * called during query planning. The parameters are the same as for + * GetForeignRelSize, which has already been called. + * + * This function must generate at least one access path (ForeignPath node) + * for a scan on the foreign table and must call add_path to add each such + * path to rel->pathlist. It's recommended to use + * create_foreignscan_path to build the ForeignPath nodes. The function + * can generate multiple access paths, e.g., a path which has valid + * pathkeys to represent a pre-sorted result. Each access path must + * contain cost estimates, and can contain any FDW-private information + * that is needed to identify the specific scan method intended. + */ -/* FIXME - plus shouldn't the validation already know the associated wrapper, server, and table?... */ + PathTarget *target = NULL; + List *pathkeys = NIL; + Relids required_outer = NULL; + Path *fdw_outerpath = NULL; // extra plan + List *fdw_restrictinfo = NIL; + List *options = NIL; + Cost startup_cost = 0; + Cost total_cost = startup_cost + rel->rows; -/** - * Public - Foreign Data Wrapper - */ -static -typedef struct { - jobject (*newServer)(JNI_FDW_Wrapper *) = wrapper_new_server; + // JNI_FDW_PlanState *plan_state = NULL; - JNIEnv *env; - jclass wrapperClass - jobject *instance; + // elog(NOTICE, "server: %s (%s) type: %s (%d)", server->servername, server->serverversion, server->servertype, server->fdwid); + // elog(NOTICE, "server: %s (%d)", server->servername, server->fdwid); - JNI_FDW_Validator *validator; -} JNI_FDW_Wrapper; + // elog(NOTICE, "table:, rel: %d, serverid: %d", table->relid, table->serverid); + // table->options; -static -JNI_FDW_Wrapper *validator_new_wrapper(JNI_FDW_Validator *validator, const char *handler_classname) -{ - const JNIEnv *env = validator->env; - jmethodId validateMethodId = env->GetMethodID(validator->validatorClass, "validate", "(V)[org.postgresql.pljava.fdw.Wrapper;"); + // List *to_list = table->options; - const JNI_FDW_Wrapper *wrapper = (JNI_FDW_Wrapper *) palloc0(sizeof JNI_FDW_Wrapper); +#ifdef USE_JAVA + // NOTE: the fact that we see the `foreigntableid` parameter means that + // this is probably a null value and we need to call a constructor. + // I remember the documentation referred to a state being available + // but the `fdw_private` ptr was still null. + plan_state = ... - wrapper->env = validator->env; - wrapper->wrapperClass = env->FindClass(validator_classname); - wrapper->instance = env->CallObjectMethod(validator->instance, validator->validateMethodId); - wrapper->validator = validator; + // now update the plan_state. +#endif - return wrapper; + /* Create a ForeignPath node and add it as only possible path */ + add_path(rel, (Path *) create_foreignscan_path( + root, + rel, + target, + rel->rows, // planState->rows +#if (PG_VERSION_NUM >= 180000) + 0, /* no disabled nodes */ +#endif + startup_cost, // planState->startup_cost + total_cost, // planState->total_cost + pathkeys, + required_outer, + fdw_outerpath, +#if (PG_VERSION_NUM >= 170000) + fdw_restrictinfo, +#endif + options)); } -/* - * Public - Server - */ -static -typedef struct { - void* (*newTable)(void) = server_new_table; - - JNIEnv *env; - jclass serverClass; - jobject *instance; - - JNI_FDW_Table *wrqpper; -} JNI_FDW_Server; - -static -JNI_FDW_Server *wrapper_new_server(JNI_FDW_Wrapper *wrapper) +static ForeignScan * +blackholeGetForeignPlan(PlannerInfo *root, + RelOptInfo *rel, + Oid foreigntableid, + ForeignPath *best_path, + List *tlist, + List *restrictinfo_list, + Plan *outer_plan) { - const JNIEnv *env = wrapper->env; - jmethodId newServerMethodId = env->GetMethodID(wrapper->wrapperClass, "newServer", "(V)[org.postgresql.pljava.fdw.Server;"); - - const JNI_FDW_Server *server = (JNI_FDW_Server *) palloc0(sizeof JNI_FDW_Server); - server->env = env; -// server->serverClass = env->FindClass(validator_classname); - server->instance = env->CallObjectMethod(wrapper->instance, newServerMethodId); - server->wrapper = wrapper; - - return server; -} + /* + * Create a ForeignScan plan node from the selected foreign access path. + * This is called at the end of query planning. The parameters are as for + * GetForeignRelSize, plus the selected ForeignPath (previously produced + * by GetForeignPaths), the target list to be emitted by the plan node, + * and the restriction clauses to be enforced by the plan node. + * + * This function must create and return a ForeignScan plan node; it's + * recommended to use make_foreignscan to build the ForeignScan node. + */ -/* - * Public - Foreign Table - */ -static -typedef struct { - void* (*newPlanState)(JNI_FDW_Table *table); - void *(*newScanState)(JNI_FDW_Table *table); - void (*analyze)(JNI_FDW_Table *table); - - JNIEnv *env; - jclass tableClass; - jobject instance; - - JNI_FDW_Table *server; -} JNI_FDW_Table; - -static -JNI_FDW_Table *server_new_table(JNI_FDW_Server *server) { - const JNIEnv *env = server->env; - jmethodId newTableMethodId = env->GetMethodID(server->serverClass, "newTable", - "(V)[org.postgresql.pljava.fdw.Table;"); - - const JNI_FDW_Table *table = (JNI_FDW_Table *) palloc0(sizeof JNI_FDW_Table); - table->env = env; - // table->tableClass = env->FindClass(table_classname); - table->instance = env->CallObjectMethod(wrapper->instance, newTableMethodId); - table->server = server; - - return table; -} + Index scan_relid = rel->relid; -/* - * Private - plan state - */ -static -typedef struct { - void (*open)(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foregntableid) = plan_open; - void (*close)(JNI_FDW_PlanState *planState); + /* + * We have no native ability to evaluate restriction clauses, so we just + * put all the scan_clauses into the plan node's qual list for the + * executor to check. So all we have to do here is strip RestrictInfo + * nodes from the clauses and ignore pseudoconstants (which will be + * handled elsewhere). + */ - JNIEnv *env; - jobject *instance; + bool pseudocontent = false; - JNI_FDW_Table *table; + /* Create the ForeignScan node */ + List *fdw_exprs = NIL; // expressions to evaluate + List *fdw_private = NIL; // private state + List *fdw_scan_tlist = NIL; // custom tlist + List *fdw_recheck_quals = NIL; // remote quals + List *restrictions = extract_actual_clauses(restrictinfo_list, pseudocontent); + ForeignScan *scan = NULL; -} JNI_FDW_PlanState; + // JNI_FDW_ScanState *scan_state; -static -JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table) { - const JNIEnv *env = table->env; + elog(NOTICE, "entering function %s", __func__); - jmethodId openPlanMethodId = env->GetMethodID(table->tableClass, "newPlanState", - "(V)[org.postgresql.pljava.fdw.PlanState;"); +#ifdef USE_JAVA + // the presence of the `foreigntableid` says that we should be calling a constructor. - const JNI_FDW_PlanState *planState = (JNI_FDW_PlanState *) palloc0(sizeof JNI_FDW_PlanState); - planState->env = env; - // table->planStateClass = env->FindClass(planstate_classname); - planState->instance = env->CallObjectMethod(table->instance, newPlanStateMethodId); - planState->table = table; -} + // update the variables mentioned above. +#endif -static -void *plan_open(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid) { - const JNIEnv *env = table->env; + scan = make_foreignscan(tlist, + restrictions, + scan_relid, + fdw_exprs, + fdw_private, + fdw_scan_tlist, + fdw_recheck_quals, + outer_plan); + +#ifdef USE_JAVA + // I'm not sure we do this here... see next method + // scan->fdw_private = scanState; +#endif - // FIXME: for now we don't pass anything through. However we could after a bit of conversions... - const jmethodId openPlanMethodId = env->GetMethodID(planState->planStateClass, "open", "(V)V"); - env->CallObjectMethod(planState->instance, openPlanStateMethodId); + return scan; } -/* - * Private - scan state - */ -static -typedef struct { - void (*open)(JNI_FDW_ScanState *scanState, ForeignScanState *node, int eflags) = open_scan; - void (*next)(JNI_FDW_ScanState *scanState, TableTupleSlot *slot); - void (*reset)(JNI_FDW_ScanState *scanState); - void (*close)(JNI_FDW_ScanState *scanState); - void (*explain)(JNI_FDW_ScanState *scanState); - - JNIEnv *env; - jobject *instance; +static void +blackholeBeginForeignScan(ForeignScanState *node, + int eflags) { + /* + * Begin executing a foreign scan. This is called during executor startup. + * It should perform any initialization needed before the scan can start, + * but not start executing the actual scan (that should be done upon the + * first call to IterateForeignScan). The ForeignScanState node has + * already been created, but its fdw_state field is still NULL. + * Information about the table to scan is accessible through the + * ForeignScanState node (in particular, from the underlying ForeignScan + * plan node, which contains any FDW-private information provided by + * GetForeignPlan). eflags contains flag bits describing the executor's + * operating mode for this plan node. + * + * Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is true, this function + * should not perform any externally-visible actions; it should only do + * the minimum required to make the node state valid for + * ExplainForeignScan and EndForeignScan. + * + */ - JNI_FDW_Table *table; -} JNI_FDW_ScanState; + // ScanState scanState = (ForeignScan *) node->ss + // PlanState planState = (ForeignScan *) node->ss.ps; -static -JNI_FDW_ScanState *table_new_scan_state(JNI_FDW_Table *table, ForeignScanState *node, int eflag) { - const JNIEnv *env = table->env; +// Plan *plan = (ForeignScan *) node->ss.ps.plan; +// EState *state = (ForeignScan *) node->ss.ps.state; +// List *options; + bool explainOnly = eflags & EXEC_FLAG_EXPLAIN_ONLY; - // for now we ignore the extra parameters. - const jmethodId newScanStateMethodId = env->GetMethodID(table->tableClass, "newScanState", - "(V)[org.postgresql.pljava.fdw.ScanState;"); + // this is initially set to NULL as a marker for 'explainOnly' + JNI_FDW_ScanState *scan_state = NULL; - const JNI_FDW_ScanState *scanState = (JNI_FDW_ScanState *) palloc0(sizeof JNI_FDW_ScanState); - scanState->env = env; + elog(NOTICE, "entering function %s", __func__); - // for now we ignore the extra parameters. - scanState->instance = env->CallObjectMethod(table->instance, newScanStateMethodId); - scanState->table = table; + /* Fetch options of foreign table */ +// fileGetOptions(RelationGetRelid(node->ss.ss_currentRelation), +// &filename, &is_program, &options); - return planState; -} + /* Add any options from the plan (currently only convert_selectively) */ +/* + if (plan_state != NULL) + { + options = list_concat(options, plan_state); + } +*/ -static -void scan_open(JNI_FDW_ScanState *scanState, ForeignScanState *node, int eflag) { + /* + * Do nothing in EXPLAIN (no ANALYZE) case. node->fdw_state stays NULL. + */ + if (explainOnly) + { + // this function should not perform any externally-visible actions; + // it should only do the minimum required to make the node state valid for + // ExplainForeignScan and EndForeignScan. -} + return; + } -/* ------------------------------------------------------------ - * Rest of JNI Implementation - does not use memory management yet! - * ------------------------------------------------------------*/ -static void validator_add_option(JNI_FDW_Validator *validator, int relid, String key, String value) -{ - const JNIEnv *env = validator->env; - const jmethodId addOptionMethodId = env->GetMethodID(validator->validatorClass, "addOption", "(int, String, String)V"); - - const jint jrelid = NULL; - const jstring jkey = NULL; - const jstring jvalue = NULL; + /* + * From FileFDW + * Create CopyState from FDW options. We always acquire all columns, so + * as to match the expected ScanTupleSlot signature. + */ + /* + cstate = BeginCopyFrom(NULL, + node->ss.ss_currentRelation, + NULL, + filename, + is_program, + NULL, + NIL, + options); + */ + + // plan_state = (JNI_FDW_PlanState *) plan->fdw_private; +#ifdef USE_JAVA + // 'explain only' is always false here... + scan_state = jni_create_blackhole_fdw_scan_state(node, explainOnly); + + if (scan_state != NULL) { + scan_state->open(scan_state); + } +#endif - env->CallObjectMethod(validator->instance, addOptionMethodId, jrelid, jkey, jvalue); + node->fdw_state = scan_state; } -/* ------------------------------------------------------------ - * The POSTGRESQL implementations - * -----------------------------------------------------------*/ - -/* ------------------------------------------------------------ - * FIXME: How do we get to specific JNI_FDW_Table ?? - * ------------------------------------------------------------*/ +static TupleTableSlot * +blackholeIterateForeignScan(ForeignScanState *node) { + /* + * Fetch one row from the foreign source, returning it in a tuple table + * slot (the node's ScanTupleSlot should be used for this purpose). Return + * NULL if no more rows are available. The tuple table slot infrastructure + * allows either a physical or virtual tuple to be returned; in most cases + * the latter choice is preferable from a performance standpoint. Note + * that this is called in a short-lived memory context that will be reset + * between invocations. Create a memory context in BeginForeignScan if you + * need longer-lived storage, or use the es_query_cxt of the node's + * EState. + * + * The rows returned must match the column signature of the foreign table + * being scanned. If you choose to optimize away fetching columns that are + * not needed, you should insert nulls in those column positions. + * + * Note that PostgreSQL's executor doesn't care whether the rows returned + * violate any NOT NULL constraints that were defined on the foreign table + * columns — but the planner does care, and may optimize queries + * incorrectly if NULL values are present in a column declared not to + * contain them. If a NULL value is encountered when the user has declared + * that none should be present, it may be appropriate to raise an error + * (just as you would need to do in the case of a data type mismatch). + */ -/** - * Called to get an estimated size of the foreign table. - * - * Note: this can be a no-op. - */ -static void -blackholeGetForeignRelSize(PlannerInfo *root, - RelOptInfo *baserel, - Oid foreigntableid) { - - // FIXME - this should be available... somewhere... - const JNI_FDW_Table table = NULL; - JNI_FDW_Plan plan_state; + JNI_FDW_ScanState *scan_state = (JNI_FDW_ScanState *) node->fdw_state; + TupleTableSlot *slot = node->ss.ss_ScanTupleSlot; - elog(DEBUG1, "entering function %s", __func__); + /* get the current slot and clear it */ + ExecClearTuple(slot); - plan_state = table->newPlan(root, baserel, foreigntableid); - baserel->fdw_private = (void *) plan_state; + /* + * Do nothing in EXPLAIN (no ANALYZE) case. node->fdw_state stays NULL. + */ + if (node->fdw_state == NULL) { + return NULL; + } - baserel->rows = plan_state->rows; + elog(NOTICE, "entering function %s", __func__); - /* initialize required state in plan_state */ + if (scan_state != NULL) + { +#ifdef USE_JAVA + /* get the next record, if any, and fill in the slot */ + scan_state->next(scan_state); + populate slot +#endif + } + /* then return the slot */ + return slot; } -/** - * SELECT: Called to find the location of the foreign table's resources. - */ -static void -blackholeGetForeignPaths(PlannerInfo *root, - RelOptInfo *baserel, - Oid foreigntableid) { +static void +blackholeReScanForeignScan(ForeignScanState *node) { /* - * BlackholeFdwPlanState *plan_state = baserel->fdw_private; + * Restart the scan from the beginning. Note that any parameters the scan + * depends on may have changed value, so the new scan does not necessarily + * return exactly the same rows. */ - Cost startup_cost, - total_cost; - - elog(DEBUG1, "entering function %s", __func__); + JNI_FDW_ScanState *scan_state = (JNI_FDW_ScanState *) node->fdw_state; - startup_cost = 0; - total_cost = startup_cost + baserel->rows; + elog(NOTICE, "entering function %s", __func__); - /* Create a ForeignPath node and add it as only possible path */ - add_path(baserel, (Path *) - create_foreignscan_path(root, baserel, - NULL, /* default pathtarget */ - baserel->rows, -#if (PG_VERSION_NUM >= 180000) - 0, /* no disabled nodes */ -#endif - startup_cost, - total_cost, - NIL, /* no pathkeys */ - NULL, /* no outer rel either */ - NULL, /* no extra plan */ -#if (PG_VERSION_NUM >= 170000) - NIL, /* no fdw_restrictinfo list */ + if (scan_state != NULL) + { +#ifdef USE_JAVA + // note: should this include 'user' parameter? + scan_state->reset(scan_state); #endif - NIL)); /* no fdw_private data */ + } } -/** - * SELECT: Called to plan a foreign scan. - */ -static FdwPlan * -blackholePlanForeignScan(Oid foreigntableid, PlannerInfo *root, RelOptInfo *baserel) { - FdwPlan *fdwplan; - fdwplan = makeNode(FdwPlan); - fdwplan->fdw_private = NIL; - fdwplan->startup_cost = 0; - fdwplan->total_cost = 0; - return fdwplan; -} -/** - * SELECT: Called before the first tuple has been retrieved. It allows - * last-second validation of the parameters. - */ static void -blackholeBeginForeignScan(ForeignScanState *node, int eflags) { - // FIXME: how to get JNI_FDW_table? - const JNI_FDW_Table *table = NULL; - const JNI_FDW_ScanState table->newScan(table, node, eflags); - - elog(DEBUG1, "entering function %s", __func__); - - // I'm not sure if this is called before or after test below... - scan_state = table->begin(table, node, eflags); - - if (eflags & EXEC_FLAG_EXPLAIN_ONLY) { - return; - } - - node->fdw_state = scan_state; -} - -/** - * SELECT: Called to retrieve each tuple in the foreign table. - * Note: the external resource must be opened in this function. - */ -static TupleTableSlot * -blackholeIterateForeignScan(ForeignScanState *node) { - const TableTupleType slot = node->ss.ss_ScanTupleSlot(); - const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); +blackholeEndForeignScan(ForeignScanState *node) { + /* + * End the scan and release resources. It is normally not important to + * release palloc'd memory, but for example open files and connections to + * remote servers should be cleaned up. + */ - elog(DEBUG1, "entering function %s", __func__); + JNI_FDW_ScanState *scan_state = (JNI_FDW_ScanState *) node->fdw_state; - // is this EXPLAIN_ONLY ? - if (scan_state == NULL) { + /* + * Do nothing in EXPLAIN (no ANALYZE) case. node->fdw_state stays NULL. + */ + if (node->fdw_state == NULL) { return; } - ExecClearTuple(slot); - scan_state->next(scan_state, slot); - - // additional processing? + elog(NOTICE, "entering function %s", __func__); - return slot; + if (scan_state != NULL) + { +#ifdef USE_JAVA + scan_state->close(scan_state); +#endif + pfree(scan_state); + } } -/** - * SELECT: Called to reset internal state to initial conditions. - */ -static void -blackholeReScanForeignScan(ForeignScanState *node) { - const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); - - elog(DEBUG1, "entering function %s", __func__); - - // is this EXPLAIN_ONLY ? - if (scan_state == null) { - return; - } - - scan_state->reset(scan_state); +static int +blackholeIsForeignRelUpdatable(Relation rel) +{ + // TODO: check FDW_user... + return 0; } /* - * SELECT: Called after the last row has been returned. + * fileExplainForeignScan + * Produce extra output for EXPLAIN */ static void -blackholeEndForeignScan(ForeignScanState *node) { - const JNI_FDW_ScanState *scan_state = (JNI_FDW_Scan *) node->fdw_state(); - - elog(DEBUG1, "entering function %s", __func__); - - scan_state->close(scan_state); - // scan->table.removeScan(scan); ??? - - pfree(scan_state); -} - -/** - * Called when EXPLAIN is executed. This allows us to provide - * Wrapper, Server, and Table options like URLs, etc. - */ -static void -blackholeExplainForeignScan(ForeignScanState *node, - struct ExplainState *es) { +blackholeExplainForeignScan(ForeignScanState *node, ExplainState *es) +{ + // List *options; - elog(DEBUG1, "entering function %s", __func__); +#ifdef NEVER + // this comes from FileFdw. -} + /* Fetch options --- we only need filename and is_program at this point */ + fileGetOptions(RelationGetRelid(node->ss.ss_currentRelation), + &filename, &is_program, &options); -/** - * Called when ANALYZE is executed on a foreign table. - */ -static bool -blackholeAnalyzeForeignTable(Relation relation, - AcquireSampleRowsFunc *func, - BlockNumber *totalpages) { + if (is_program) + ExplainPropertyText("Foreign Program", filename, es); + else + ExplainPropertyText("Foreign File", filename, es); - elog(DEBUG1, "entering function %s", __func__); + /* Suppress file size if we're not showing cost details */ + if (es->costs) + { + struct stat stat_buf; - return false; + if (!is_program && + stat(filename, &stat_buf) == 0) + ExplainPropertyInteger("Foreign File Size", "b", + (int64) stat_buf.st_size, es); + } +#endif } -/** - * Called when two or more foreign tables are on the same foreign server. - */ -static void -blackholeGetForeignJoinPaths(PlannerInfo *root, - RelOptInfo *joinrel, - RelOptInfo *outerrel, - RelOptInfo *innerrel, - JoinType jointype, - JoinPathExtraData *extra) { - - elog(DEBUG1, "entering function %s", __func__); +static List *blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, Oid serverOid) +{ + return NIL; } -/** - * LOCK-AWARE - called to re-fetch a tuple from a foreign table - */ -#if (PG_VERSION_NUM >= 120000) -static void blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - TupleTableSlot *slot, - bool *updated) -#else - -static HeapTuple -blackholeRefetchForeignRow(EState *estate, - ExecRowMark *erm, - Datum rowid, - bool *updated) -#endif +static bool blackholeIsForeignScanParallelSafe(PlannerInfo *root, + RelOptInfo *rel, + RangeTblEntry *rte) { - - elog(DEBUG1, "entering function %s", __func__); - -#if (PG_VERSION_NUM < 120000) - return NULL; -#endif + return false; } - -/* - * Called when IMPORT FOREIGN SCHEMA is executed. - */ -static List * -blackholeImportForeignSchema(ImportForeignSchemaStmt *stmt, - Oid serverOid) { - - elog(DEBUG1, "entering function %s", __func__); - - return NULL; +static bool blackholeIsForeignPathAsyncCapable(ForeignPath *path) { + return false; } diff --git a/pljava-so/src/main/c/BlackholeFDWJNI.c b/pljava-so/src/main/c/BlackholeFDWJNI.c new file mode 100644 index 000000000..cd32cfe81 --- /dev/null +++ b/pljava-so/src/main/c/BlackholeFDWJNI.c @@ -0,0 +1,253 @@ +/** + * Actual implementation of minimal FDW based on the 'blackhole_fdw' + * project. For simplicity all of the comments and unused functions + * have been removed. + * + * The purpose of this file is to demonstrate the ability of C-based + * FDW implementation to successfully interact with a java object + * that implements the FDW interfaces. + * + * The first milestone is simply sending a NOTICE from the java: + * method. + */ +#include "postgres.h" + +#include "access/reloptions.h" +#include "commands/explain.h" +#include "foreign/fdwapi.h" +#include "foreign/foreign.h" +#include "optimizer/pathnode.h" +#include "optimizer/planmain.h" +#include "optimizer/restrictinfo.h" + +#include "pljava/pljava.h" +#include "pljava/FDW.h" + +// PG_MODULE_MAGIC; + +#if (PG_VERSION_NUM < 90500) +// fail... +#endif + +/* ------------------------------------------------------------ */ + +// had been 'JNI_FDW.h' +#ifndef NEVER + +/** + * Wrapper functions + */ + +/* + * Persistent + */ +struct { + // void (*addOption)(JNI_FDW_Validator *, int, const char *, const char *) = validator_add_option; + // bool (*validate)(JNI_FDW_Validator *); + + const JNIEnv *env; + const jclass validatorClass; + const jobject instance; +} JNI_FDW_Validator_; + +/* + * Persistent + */ +struct { + // JNI_FDW_validate_options_for_reuse = fdw_validate_options_for_reuse; + + JNIEnv *env; + jclass wrapperClass; + jobject *instance; +} JNI_FDW_Wrapper_; + +/* + * Persistent + */ +struct { + // JNI_FDW_validate_options_for_reuse = srv_validate_options_for_reuse; + // void* (*getMetadata)(JNI_FDW_Server *server); + + JNIEnv *env; + jclass serverClass; + jobject *instance; +} JNI_FDW_Server_; + +/* + * Persistent + */ +struct { + // JNI_FDW_validate_options_for_reuse = srv_validate_options_for_reuse; + + JNIEnv *env; + jclass serverClass; + jobject *instance; +} JNI_FDW_User_; + +/* + * Persistent + */ +struct { + // JNI_FDW_validate_options_for_reuse = ft_validate_options_for_reuse; + + // void* (*newPlanState)(JNI_FDW_Table *table, JNI_FDW_User *user); + // void *(*newScanState)(JNI_FDW_Table *table, JNI_FDW_User *user); + + // void *(getMetaData)(JNI_FDW_Table *table JNI_FDW_User *user); + + // bool (*updateable)(JNI_FDW_Table *table, JNI_FDW_User *user); + // bool (*supportsConcurrency)(JNI_FDW_Table *table); + // bool (*supportsAsyncOperations)(JNI_FDW_Table *table); + + // void (*analyze)(JNI_FDW_Table *table); + // void (*vacuum)(JNI_FDW_Table *table); + + JNIEnv *env; + jclass tableClass; + jobject instance; +} JNI_FDW_Table_; + +/* + * Temporary + */ +struct { + // void (*open)(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foregntableid); + // void (*open)(JNI_FDW_PlanState *planState); + // void (*close)(JNI_FDW_PlanState *planState); + + JNIEnv *env; + // jclass planStateClass; ? + jobject *instance; + jlong rows; + + // cached values? + jdouble cost; + jdouble startup_cost; + jdouble totalcost; +} JNI_FDW_PlanState_; + +/* + * Temporary + */ +struct { + // void (*open)(JNI_FDW_ScanState *scanState, ForeignScanState *node, int eflags) = open_scan; + // void (*next)(JNI_FDW_ScanState *scanState, TableTupleSlot *slot); + // void (*reset)(JNI_FDW_ScanState *scanState); + // void (*close)(JNI_FDW_ScanState *scanState); + // void (*explain)(JNI_FDW_ScanState *scanState); + + JNIEnv *env; + jobject *instance; +} JNI_FDW_ScanState_; +#endif + +/* ------------------------------------------------------------ */ + +/* + +static JNI_FDW_Wrapper *validator_get_wrapper(JNI_FDW_Validator *validator); +static JNI_FDW_Server *wrapper_get_server(JNI_FDW_Wrapper *wrapper); +static JNI_FDW_Table *server_get_table(JNI_FDW_Server *server); +static JNI_FDW_PlanState *table_new_plan(JNI_FDW_Table *table); +static JNI_FDW_ScanState *table_new_scan(JNI_FDW_Table *table); + +// not all functions... + +static void validator_add_option(JNI_FDW_Validator *, int relid, String key, String value); +static bool validator_validate(JNI_FDW_Validator *); + +static JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table); +static void plan_open(JNI_FDW_Table *table, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid); +static void plan_close(JNI_FDW_PlanState *plan_state); + +static void scan_open(JNI_FDW_ScanState *scan_state, ForeignScanState *node, int eflag); +static void scan_next(JNI_FDW_ScanState *scan_state, Slot *slot); +static void scan_reset(JNI_FDW_ScanState *scan_state); +static void scan_close(JNI_FDW_ScanState *scan_state); + +static JNI_FDW_ScanState *table_new_scanPlan(JNI_FDW_Table *table); + */ + +// Note: this does not do memory management yet! + +/* +static +jmethodID getMethodID(JNIEnv *env, jclass class, ...) +{ + return env->GetMethodID(class, va_arg); +} +*/ + +#ifdef USE_JAVA +JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname) { + JNI_FDW_Validator *validator = (JNI_FDW_Validator *) palloc0(sizeof JNI_FDW_Validator); + + validator->env = env; + validator->validatorClass = env->FindClass(validator_classname); + validator->instance = env->AllocObject(fdw->validatorClass); + + return validator; +} + +static +JNI_FDW_PlanState *table_new_planstate(JNI_FDW_Table *table) { + const JNIEnv *env = table->env; + + jmethodID openPlanMethodId = env->GetMethodID(table->tableClass, "newPlanState", + "(V)[org.postgresql.pljava.fdw.PlanState;"); + + const JNI_FDW_PlanState *planState = (JNI_FDW_PlanState *) palloc0(sizeof JNI_FDW_PlanState); + planState->env = env; + // table->planStateClass = env->FindClass(planstate_classname); + planState->instance = env->CallObjectMethod(table->instance, newPlanStateMethodId); + planState->table = table; +} + +static +void *plan_open(JNI_FDW_PlanState *planState, PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid) { + const ForeignTable foreignTable = GetForeignTable(foreigntableid); + + const JNIEnv *env = table->env; + + // FIXME: for now we don't pass anything through. However we could after a bit of conversions... + const jmethodId openPlanMethodId = env->GetMethodID(planState->planStateClass, "open", "(V)V"); + env->CallObjectMethod(planState->instance, openPlanStateMethodId); +} +#endif + +#ifdef USE_JAVA +JNIEXPORT +JNI_FDW_ScanState * JNICALL table_new_scan_state(JNI_FDW_Table *table, ForeignScanState *node, int eflag) { + const JNIEnv *env = table->env; + + // for now we ignore the extra parameters. + const jmethodID newScanStateMethodId = env->GetMethodID(table->tableClass, "newScanState", + "(V)[org.postgresql.pljava.fdw.ScanState;"); + + const JNI_FDW_ScanState *scanState = (JNI_FDW_ScanState *) palloc0(sizeof JNI_FDW_ScanState); + scanState->env = env; + + // for now we ignore the extra parameters. + scanState->instance = env->CallObjectMethod(table->instance, newScanStateMethodId); + scanState->table = table; + + return planState; +} +#endif + +/* ------------------------------------------------------------ + * Rest of JNI Implementation - does not use memory management yet! + * ------------------------------------------------------------*/ +#ifdef USE_JAVA +static void validator_add_option(JNI_FDW_Validator *validator, int relid, String key, String value) +{ + const JNIEnv *env = validator->env; + const jmethodID addOptionMethodId = env->GetMethodID(validator->validatorClass, "addOption", "(int, String, String)V"); + + const jint jrelid = NULL; + const jstring jkey = NULL; + const jstring jvalue = NULL; + + env->CallObjectMethod(validator->instance, addOptionMethodId, jrelid, jkey, jvalue); +} +#endif \ No newline at end of file diff --git a/pljava-so/src/main/include/pljava/FDW.h b/pljava-so/src/main/include/pljava/FDW.h index f0a6eeeb0..c0ecfdee1 100644 --- a/pljava-so/src/main/include/pljava/FDW.h +++ b/pljava-so/src/main/include/pljava/FDW.h @@ -10,14 +10,29 @@ * The first milestone is simply sending a NOTICE from the java * method. */ -#ifndef PLJAVA_SO_BLACKHOLEFDW_H -#define PLJAVA_SO_BLACKHOLEFDW_H +#ifndef PLJAVA_SO_FDW_H +#define PLJAVA_SO_FDW_H // temporary name... extern Datum blackhole_fdw_handler(PG_FUNCTION_ARGS); extern Datum blackhole_fdw_validator(PG_FUNCTION_ARGS); -typedef struct JNI_FDW_Validator JNI_FDW_Validator; -JNI_FDW_Validator newValidator(JNIEnv *env, const char *validator_classname); +struct JNI_FDW_Wrapper_; +struct JNI_FDW_Server_; +struct JNI_FDW_User_; // or 'UserMapping' ? +struct JNI_FDW_Table_; -#endif //PLJAVA_SO_BLACKHOLEFDW_H +struct JNI_FDW_PlanState_; +struct JNI_FDW_ScanState_; + +// permanent objects (with OID) +typedef struct JNI_FDW_Wrapper_ JNI_FDW_Wrapper; +typedef struct JNI_FDW_Server_ JNI_FDW_Server; +typedef struct JNI_FDW_User_ JNI_FDW_User; // or 'UserMapping' ? +typedef struct JNI_FDW_Table_ JNI_FDW_Table; + +// temporary objects (no OID) +typedef struct JNI_FDW_PlanState_ JNI_FDW_PlanState; +typedef struct JNI_FDW_ScanState_ JNI_FDW_ScanState; + +#endif //PLJAVA_SO_FDW_H