Stacks?

[matiasgarciaisaia]

Does the "stack" concept have place in uncloud?

Apps usually have more than a single component (say, an app + a DB), so a stack would provide a namespace for services (so StackA's app can call db to its DB service, and not clash with StackB's db).

Stacks may offer network isolation (ie, deploy services in stack-specific Docker network), and only expose services with x-ports to the Caddy server (using a globally shared network, or creating ad-hoc networks for each Caddy<->exposed service pair).

If network isolation is implemented, there should also be ways to override it and allow connecting services from different stacks (just like in the Caddy example).

[psviderski]

I'm glad you asked this. There is no such isolation as you described but the current design allows it to be introduced at some point and I've been thinking about it for a while.

I didn't like how stacks are implemented in Docker Swarm and Compose, namely, how service names are prefixed with stack/project-name_. You need to include this prefix when you want to work with services directly, e.g. inspect or read logs. I noticed that many people still override this in their compose files by specifying container_name. And Swarm stacks don't really provide any isolation apart from saving you from name collisions.

I was thinking of a namespace/environment/project concept similar to k8s/Render but decided to not complicate things from the very beginning. For now, there is only one namespace. If a service is called service-a in your compose file, it will be created in uncloud as service-a. Super simple, but there is a downside to this approach, a db service in one compose file could accidentally override the deployment of db from another compose file. We briefly discussed what we could do about this with @tonyo, e.g. tag services with a "project name" (name property in compose spec), but haven't agreed on a solution yet.

I believe we can introduce namespaces incrementally if we want to. For example, all the services created before the namespace feature was introduce could be assigned to the default namespace.

Regarding network isolation for stacks/workloads, IMO the Docker model with using multiple networks and assigning services/containers to them is quite coarse and not flexible enough as I'd like to have. So I decided to use the same approach Tailscale, Fly.io, k8s use, namely, one flat network for the cluster and provide the policy-based isolation on top of it. The network itself just provides the connectivity, and the policies (ACLs, security groups, what ever concept we come up with) can define the fine-grained permissions to isolate specific workloads.

We can define reasonable default policies, e.g. to isolate namespaces or stacks from each other if we come up with a way to define stacks. But we don't support policies yet, and similarly to namespaces, they could be introduced in the future.

That's my current thinking.

Do you like the Docker Swarm stack design or can you think of an ideal design for you?

[airtonix]

My use case is that I would:

• deploy stacks, so using docker-compose approach feels useful
• control ingress for particular domains to a particular stack
• have the freedom for a node in a stack to override this ingress and define it's own
• use envvars to control ingress domain name per stack and nodes (so normal things with docker-compose.yml)
  • also stack names. so like you mention here, something that needs to be discovered on how to implement. ( x-stack-name: "${DOMAIN}${DOMAIN:+.}me.com" ? )
  • if stack names can't namespace, then we'll have to fall back to using envvars for all container names in the "stack" and their hostnames (internal or external)
• because i'd want to do preview environments per PR, i'd also want a way to destroy and remove all traces of a stack created for such a PR.

[psviderski]

Thank you so much for describing your use case! Can you please elaborate on what you mean by

have the freedom for a node in a stack to override this ingress and define it's own

[airtonix]

have the freedom for a node in a stack to override this ingress and define it's own

Sure.

I essentially want the functionality of Traefik configuration via labels.

Using traefik as a load balancer on a shared docker network I am able to bring up multiple different stacks where it is the responsibility of one or more of the nodes in that stack to describe if they are routable.

traefik.yml

services:
  lb:
    image: traefik
    networks:
      - proxy
      - project-1

networks:
  proxy:
    name: proxy.bridge
    external: true
  project-1:
    name: project-1.bridge
    external: true

project-1.yml

services:
  db:
    image: postgres
  app:
    image: ghcr.io/me/my-app:v0.1.2-35
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.my-app-container.rule=Host(`my-app.com`)"
      - "traefik.http.routers.whoami.entrypoints=web"
    network:
      - project-1
      - default
  api:
    image: ghcr.io/me/my-api-server:v0.0.1
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.my-api-container.rule=Host(`api.my-app.com`)"
      - "traefik.http.routers.whoami.entrypoints=web"
    network:
      - project-1
      - default


networks:
  project-1:
    name: project-1.bridge
    external: true

you can see that this stack defines that app and api use different routable hostnames. They could even add other custom traefik routing config.

In the world of uncloud this isn't possible (yet).

[airtonix]

so if I have three projects running (dev/stg/prod) that use the same compose file, I'd want to be able to dynamically configure (at deploy time, not build time) what these hostnames and stack names would be:

services:
  db:
    image: postgres
  app:
    image: ghcr.io/me/my-app:${TAGS_APP:-latest}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${DOMAIN}${DOMAIN:+_}my-APP-container.rule=Host(`${DOMAIN}${DOMAIN:+.}my-app.com`)"
      - "traefik.http.routers.${DOMAIN}${DOMAIN:+_}my-APP-container.entrypoints=web"
  api:
    image: ghcr.io/me/my-api-server:${TAGS_API:-latest}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${DOMAIN}${DOMAIN:+_}my-API-container.rule=Host(`api.${DOMAIN}${DOMAIN:+.}my-app.com`)"
      - "traefik.http.routers.${DOMAIN}${DOMAIN:+_}my-API-container.entrypoints=web"

$ TAGS_APP=pr-128 \
  TAGS_API=pr-128 \
  DOMAIN=pr-128.dev \
  uncloud run ./docker-compose.deploy.yml

# or 

$ TAGS_APP=githash_12345678 \
  TAGS_API=githash_12345678 \
  DOMAIN=dev \
  uncloud run ./docker-compose.deploy.yml
  
# or 

$ TAGS_APP=githash_12345678 \
  TAGS_API=githash_12345678 \
  DOMAIN=stg \
  uncloud run ./docker-compose.deploy.yml

# or 

$ TAGS_APP=githash_12345678 \
  TAGS_API=githash_12345678 \
  uncloud run ./docker-compose.deploy.yml

[psviderski]

I see, thank you for clarifying! 🙏

[FoxxMD]

@psviderski

Do you like the Docker Swarm stack design or can you think of an ideal design for you?

I prefer the way Docker uses multiple networks. Leveraging iptables/nft_tables on each host to isolate subnets based on network type feels pragmatic to me and these tools provide a ton of flexibility, I think. I like the idea of policy/ACL but it also feels like re-inventing the wheel when networking is something that uncloud (and docker) still has to deal with anyway.

For instance, namespacing issues you discussed are already a solved problem for overlay/docker since dns/hosts is configured per subnet: Within the stack's own network each service, which needs a unique name, is addressable by that name only. Across attachable/overlay networks they can be addressed by project-service.

For my own use case -- I heavily use overlay networks without using swarm deployments specifically. Birdseye:

• two overlay networks, lan and public
  • public has explicitly defined subnet
• 10 nodes, all joined to a swarm so they can use overlays
• For each stack
  • "stack only" network is internal: true, all services in stack share this
  • one service is joined into a respective overlay, depending on if it's for lan-only or public internet usage
• One traefik instance in each overlay used as reverse proxy only to the services joined to that overlay
• trafficjam deployed on each node and configured to isolate services in the public subnet from each other, except for traefik
  • leverages iptables to add this functionality
• I have additional overlay networks so that stacks unrelated to web traffic can communicate across nodes, isolate from web stuff

The cognitive load is low once this is setup, for me. The networks defined in each compose file determine both what a service can communicate with and also isolates it from everything else not explicitly defined there. I don't need to setup policies, it's implicit.

And since all of this is "just" iptables it's easy for me to extend networking behavior without needing any explicit docker configuration: in addition to trafficjam I have pre-configured rules on each node that

• DROP New traffic from overlay networks -> destination LAN subnet
• ACCEPT Established traffic from overlay networks -> destination LAN subnet

Using uncloud with one, flat network would mean losing all of that implicit isolation, trafficjam, and my established rules that all leverage tried-and-tested tools already available on almost all linux distros. Or having to convert them to policies and hoping uncloud doesn't have any bugs in whatever wheel it reinvents to get feature parity with the existing tools.

[psviderski]

I appreciate such a detailed write-up about your setup 🙏. Everything what you described makes a lot of sense and doesn't really contradict much with what I mentioned above. Let me provide some more context.

When you're saying we're reinventing the wheel, we're consciously creating an alternative system to Docker Swarm, making significantly different decisions about the orchestration part. We have the opportunity to do other things the same way or a different way as well. Regarding networking, we can't reuse the Swarm overlay network to avoid reinventing the wheel because:

• It requires every node to join the swarm, hence breaking the uncloud's no quorum/control plane model
• every node should be reachable from each other for VXLAN to work (uncloud's overlay network allows machines to be behind NAT; for example, all machines can be in a private network + a couple of cloud machines for ingress)

I wanted to weaken these requirements, hence an alternative design for the overlay network based on WireGuard. Once we went down that path, we don't automatically get anything from Swarm (service concept or embedded DNS). We have to implement everything beyond single-host Docker anyway, and we did and this is where we're at.
We already have one overlay network with an embedded DNS so that services can resolve each other by their names. From here I can see two paths we can go:

1. Introduce the concept of isolated networks similar to Swarm and create actual Docker networks connected with a WireGuard mesh. I started writing a blog post about the details of how uncloud overlay works: https://github.com/psviderski/uncloud/blob/da09d22b4708c6e1b0f2516b33e25c603f85d140/docs/blog/wireguard-overlay.md
2. Still use only one overlay network but implement the required isolation using other logical concepts. One of the concept could even be called networks and be configured as a network in a compose file but it will only configure the firewall (iptables/nftables/ebpf) rules on top of the existing overlay network.

Note the current implementation is pretty straightforward and uses regular iptables that you are free to hack if you wish (see the post).

Using uncloud with one, flat network would mean losing all of that implicit isolation, trafficjam, and my established rules that all leverage tried-and-tested tools already available on almost all Linux distros.

Tools like trafficjam won't work with uncloud anyway, so if you want such a functionality, we'd need to create a custom solution for uncloud. Actually, the fact that you need trafficjam indicates the limitations of Swarm networks that we can try to address with more flexible concepts. What trafficjam does is what I meant by policies. So instead of having multiple networks + trafficjam for patching their limitations, we can try to design and implement one concept that addresses them both. I don't know whether there is a simple design for this, but I'm open to experiments and the whole idea of uncloud is to challenge the status quo and explore new ideas.

I hope this clarifies why we can't have the same setup you currently have with Swarm for free and why I'm considering alternatives that don't introduce the solutions that have the same downsides as the Swarm networks.

[FoxxMD]

Thanks for the reply.

I think the point I was trying to get across got lost in my emphasis on Swarm. I realize that Uncloud isn't compatible with Swarm and that my existing solutions (trafficjam) won't work regardless of how Uncloud is implemented. The point I was trying to make was mostly related to what you said here

The network itself just provides the connectivity, and the policies (ACLs, security groups, what ever concept we come up with) can define the fine-grained permissions to isolate specific workloads.

in that I think Swarm's usage of iptables/isolated networks as policy/acl is pragmatic and shouldn't be overlooked to "reinvent the wheel" with some other mechanism specific to uncloud. I don't think uncloud should be Swarm, but I think it should use the same mechanisms of isolation via multiple networks and ip/nat tables as Swarm does. But your reply does seem to confirm that it does (or could!)

wireguard-overlay.md

Thanks, this was an interesting read. Off-topic to this discussion but I've cobbled together something similar you might be interested in.

I've got a netbird (which is also wireguard) overlay configured between my VPC VM and my local home network. Rather than have a WG client on each machine, I have one routing peer that sits in my home network with masquerading off.

I then use a combination of a static route on my router and DHCP option 121 to push a static route to all machines in my network to route all WG traffic through home routing peer's local IP IE 100.110.0.0/16 via 192.168.0.12

The VPC VM is then joined to my home Swarm using its WG IP for --advertiste-addr.

This solution works well for basic swarm capabilities and any traffic over normal docker bridge networks. I'm still working out the kinks for using swarm overlay networks which seems to be tricky due to packet overhead of wireguard + netbird + vxlan. A lower MTU works but I'm still tinkering with whether this needs to be done only at interface (docker ingress or bridge interface) and/or at overlay creation with com.docker.network.driver.mtu.