Skip to content

refactor: use curl_global_init_mem with pg allocators #223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

steve-chavez
Copy link
Member

@steve-chavez steve-chavez commented Aug 14, 2025

Uses curl_global_init_mem(CURL_GLOBAL_DEFAULT, net_malloc, net_free, net_realloc, net_strdup, net_calloc) to keep better track of memory allocation by curl.

The aim is to fix #216.

Unfortunately this currently segfaults when reaching 2 iterations:

-- do twice
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,10);
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,10);

-- or once (this will do two iterations since the batch size defaults to 200)
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,300);

The logs:

2025-08-14 17:55:08.571 -05 [643019] DEBUG:  net_realloc(palloc): a((nil)), sz(32)
2025-08-14 17:55:08.571 -05 [643019] DEBUG:  net_malloc: sz(32)
2025-08-14 17:55:08.665 -05 [643013] LOG:  background worker "pg_net 0.19.5 worker" (PID 643019) was terminated by signal 11: Segmentation fault

Using gdb (via convenience sudo xpg gdb) will show that this will always fail when hitting curl functions:

Curl_attach_connection
Curl_llist_count
Curl_conn_is_alive

Which are called when using curl_multi_socket_action in our codebase in

pg_net/src/worker.c

Lines 320 to 336 in a7792bf

for (int i = 0; i < nfds; i++) {
if (is_timer(events[i])) {
EREPORT_MULTI(
curl_multi_socket_action(worker_state->curl_mhandle, CURL_SOCKET_TIMEOUT, 0, &running_handles)
);
} else {
int curl_event = get_curl_event(events[i]);
int sockfd = get_socket_fd(events[i]);
EREPORT_MULTI(
curl_multi_socket_action(
worker_state->curl_mhandle,
sockfd,
curl_event,
&running_handles)
);
}

Unfortunately this is quite hard to debug, it might be a bug in curl itself.

Uses `curl_global_init_mem(CURL_GLOBAL_DEFAULT, net_malloc, net_free, net_realloc, net_strdup, net_calloc)`
to keep better track of memory allocation by curl.

The aim is to fix supabase#216.

Unfortunately this currently segfaults when reaching 2 iterations:

```sql
-- do twice
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,10);
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,10);

-- or once (this will do two iterations since the batch size defaults to 200)
select net.http_get('http://localhost:8080/pathological?status=200') from generate_series(1,300);
```

The logs:

```
2025-08-14 17:55:08.571 -05 [643019] DEBUG:  net_realloc(palloc): a((nil)), sz(32)
2025-08-14 17:55:08.571 -05 [643019] DEBUG:  net_malloc: sz(32)
2025-08-14 17:55:08.665 -05 [643013] LOG:  background worker "pg_net 0.19.5 worker" (PID 643019) was terminated by signal 11: Segmentation fault
```

Using `gdb` (via convenience `sudo xpg gdb`) will show that this will
always fail when hitting curl functions:

```
Curl_attach_connection
Curl_llist_count
Curl_conn_is_alive
```

Which are called when using `curl_multi_socket_action` in our codebase
in https://github.com/supabase/pg_net/blob/a7792bfd913c7859e14025d606624982706c2a7f/src/worker.c#L320-L336

Unfortunately this is quite hard to debug, it might be a bug in curl itself.
@steve-chavez
Copy link
Member Author

Unfortunately this is quite hard to debug, it might be a bug in curl itself.

I think this is because some curl allocations are done in sub memory contexts and the curl_global_init_mem is now causing some frees and mallocs to happen at the top level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

High Memory usage when the net._http_response table is too big
1 participant