Using Docker labels to have per-service configuration

[muratcorlu]

Is your feature request related to a problem? Please describe.

I have a big Docker Swarm machine that I host all of my projects as Docker containers. I want to have a central backup solution for all of the Swarm Stacks/Services that can be configured inside the projects' own docker-compose files.

Describe the solution you'd like

I use Traefik for routing incoming traffic to the containers. In Traefik, you don't give a central configuration, instead all of the service configurations are set by Docker labels inside project docker-compose files. This simplify the things a lot.

So I would like to be able to enable backup for a docker service like below:

myservice:
  image: ....
  volumes:
    - myvolume:/data
  deploy:
    labels:
      - "backup.enabled=true"
      - "backup.source=myvolume"
      # and other configurations as well, like a custom schedule, retention etc.

Describe alternatives you've considered

1. Running separate backup instances per stack doesn't seem efficient, since I'll duplicate a lot of configuration.
2. I considered backing up the parent folder of the path that Docker keeps all of the volumes. But that doesn't sound a good idea, maybe is? 🤷🏻
3. Configuring everything in a central place was another consideration but every project has its own Git repository, docker-compose file and CI/CD pipelines and keeping backup logic in another place is always a risk to have hassle.

[m90]

Thanks for this suggestion. I'm not a traefik user myself, but I have seen its configuration approach being very popular in the Swarm/compose world. The trouble is that such an approach is very different from the way the tool is currently sourcing its configuration from, so adding support for this would require some major refactoring (which is probably good). I'll have to think about a bit. If you have a full blown "dream API example" (i.e. a compose service definition) of how you would think this could work, that'd also be helpful.

For the time being, your options are:

• define a backup "sidecar" in each of your compose files, backing up a dedicated service. Configuration could come from global env vars you pass through
• use one backup service with conf files being mounted from a local place https://offen.github.io/docker-volume-backup/how-tos/run-multiple-schedules.html

[pixxon]

I also considered this approach, but one major obstacle would be to figure out how to mount the volumes nicely for backup. I would not want every single volume to be mounted in the manager, so instead some sort of containerized periodic task based approach would be the best.

However, I am not sure how to configure the volume that needs to be backed up. The full name of the volume that needs to be backed up might not be available when the service is being labeled. ( The stack name will be added as a postfix. ) Best way would be to just specify a path in the application and mount the directory from there, but I have no idea if that is possible. ( Something like COPY --from in builds. )

[pixxon]

Maybe a wild take, but instead of adding the labels to services, one could add them to the volumes themselves? Docker allows objects to be labeled, including volumes. Both the volume create and docker compose file allow the specification of them.

To me it is very appealing as it would really bind the service to volumes themselves. The labels to stop containers / services would still be applied to them, but when and what to backup could be defined on the volume itself.

Lastly the where to backup could be defined on the manager itself, somewhat how traefik defines entrypoints. Then these are just referenced in the volume labels to avoid redundancy.

[muratcorlu]

Labelling volumes instead of containers for configuring backup makes so much sense. It would also fix the potential issue of what will happen if same volume mounted to multiple containers.

volumes:
  my_data:
    labels:
      backup.enabled: true
      backup.retention: 7

Looks awesome! 😊

[m90]

I also like the idea of labeling volumes a lot, but I still have a slightly hazy vision of who'd be controlling whom when you can label both volumes and services and containers. I.e. in a setup that runs multiple schedules, who'd tell containers/services when they need to be stopped? Would each labeled volume create a cron schedule? What happens if labels change, who's notfiying the backup container that it needs to create a new cron? What about bind mounts?

I could continue the list forever, but I guess that doesn't lead anywhere. Maybe a good next step would be translating one of the test cases https://github.com/offen/docker-volume-backup/tree/main/test into the desired new configuration style, so we can get an idea of what this would really look like, and how (and if even) such a configuration approach could be compatible with the existing one, or if this would require a hard cut?

[pixxon]

I have tried to make a quick mock example of what I was trying to explain above. This would certainly change the whole approach of the backup software.

The need to backup a volume would be labelled on the volume itself. This would define when to backup and other options related to how long they should be kept, what name should it be stored under.

The connection to where the backups are stored is only defined on the backup container itself, to avoid redundant definitions. ( There could be some issues, for example trying to store in different buckets, then those labels would need to move onto the volumes. )

Thirdly the containers to be interacted with are still defined on the containers themselves. So all the executions and which ones need to be stopped would remain there.

version: '3.9'

volumes:
  postgresql_db:
    labels:
      docker-volume-backup.stop-during-backup: postgres
      docker-volume-backup.filename: postgres-%Y-%m-%dT%H-%M-%S.tar.gz
      docker-volume-backup.pruning-prefix: postgres
      docker-volume-backup.retention-days: 7
      docker-volume-backup.cron-expression: 0 2 * * *

  redis_db:
    labels:
      docker-volume-backup.stop-during-backup: redis
      docker-volume-backup.filename: redis-%Y-%m-%dT%H-%M-%S.tar.gz
      docker-volume-backup.pruning-prefix: redis
      docker-volume-backup.retention-days: 7
      docker-volume-backup.cron-expression: 0 2 * * *

services:
  postgres:
    image: postgres
    volumes:
      - type: volume
        source: postgresql_db
        target: /var/lib/postgresql/data
    labels:
      docker-volume-backup.archive-pre: pg_dumpall -U postgres > /var/lib/postgresql/data/backup.sql
      docker-volume-backup.exec-label: postgres

  redis:
    image: redis
    volumes:
      - type: volume
        source: redis_db
        target: /data

  backup:
    image: offen/docker-volume-backup
    environment:
      AWS_ENDPOINT: minio
      AWS_S3_BUCKET_NAME: backup
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: test
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true

  postgres_user:
    image: testimage:latest
    labels:
      docker-volume-backup.stop-during-backup: postgres

  redis_user:
    image: testimage:latest
    labels:
      docker-volume-backup.stop-during-backup: redis

[pixxon]

how (and if even) such a configuration approach could be compatible with the existing one, or if this would require a hard cut?

It would certainly be possible to keep these two compatible. Not sure if you are familiar with Prometheus. They use something called static configs and service discovery. First would be what is currently available with the backup, second is what this would become.

[pixxon]

What happens if labels change, who's notfiying the backup container that it needs to create a new cron? What about bind mounts?

The backup would need to poll the changes from the docker socket and when there is a label change / new label, it would need to update the configuration. Minor help for this task is that labels cannot be added or changed on docker volumes, they have to be defined when the volume is created. ( At least this is via the cli, I am not sure if possible with some other coding, or from docker plugins. )

who'd tell containers/services when they need to be stopped?

From the above example, the volume would define a label postgres and redis that could be referenced by containers that use the volume ( or have an indirect dependency ) with their own labels.

Would each labeled volume create a cron schedule?

I think yes, not sure if there is a need for one volume to create multiple schedules. ( For something like keeping the daily backups for a week, weekly backups for a year. ) In that case the approach could be similar to how Traefik groups the routers/endpoints using the name of the labels.
So in the above example they would become:

  redis_db:
    labels:
      docker-volume-backup.stop-during-backup: redis
      docker-volume-backup.daily.filename: redis-daily-%Y-%m-%dT%H-%M-%S.tar.gz
      docker-volume-backup.daily.pruning-prefix: redis-daily
      docker-volume-backup.daily.retention-days: 7
      docker-volume-backup.daily.cron-expression: 0 2 * * *
      docker-volume-backup.weekly.filename: redis-weekly-%Y-%m-%dT%H-%M-%S.tar.gz
      docker-volume-backup.weekly.pruning-prefix: redis-weekly
      docker-volume-backup.weekly.retention-days: 365
      docker-volume-backup.weekly.cron-expression: 0 0 * * 0

[m90]

Just to manage expectations: I appreciate all of your feedback, and I like the direction this is going, but it also means the entire tool would need to be rearchitected (i.e. supporting both static config and service discovery), so this is not something I can implement easily in my free time (the way this project is currently run).

I'll move this around in the back of my head for a while, maybe I can come up with a way this could be sliced into several "sub-features" that could be worked on one after the other.

If you have further ideas, please leave them here, I'm happy to learn about them.

[pixxon]

this is not something I can implement easily in my free time (the way this project is currently run).

Since this is something that I would like to use, I could try to help out with the implementation. Disclaimer is that I am not a go developer. ( I mainly use C++. )

I can come up with a way this could be sliced into several "sub-features" that could be worked on one after the other.

Tasks like #268 would lead up to this. I could also help by making a dummy implementation of the above idea that could be adapted into the tool down the line.

[m90]

Thanks for offering your help. I'd be happy if you wanted to work on this.

Still, I won't be able to merge a single PR that basically rewrites the entire tool, so we'd need to plan this out a bit better. I'll still need to understand what's going on as I'll also keep maintaining this.

Right now, my idea would be to maybe proceed something like this:

1. Remove crond usage and instead allow this tool to run as a long running process that schedules work itself. It should still be possible to invoke a backup manually. This will also need to support reading configuration from conf.d. Kind of like Consider using a dedicated cron package #268 as you already mentioned.
2. Come up with a mechanism that can pull configuration from basically anything, i.e. env vars, conf.d or Docker labels. This is probably something abstract, and then we write adapters for each mean of configuration. Maybe this already exists, I'd think it made sense to look at how Traefik does this.
3. Connect the new configuration method with the Docker daemon, polling repeatedly for changes.

I'm not sure if this should be worked on as 1,2,3 or 2,1,3

Let me know what you think.

[pixxon]

I understand the concerns, I was not thinking of a single large PR either.

The 1,2,3 order seems to be easier, especially since 1 is already in the form of an issue. If it's alright, I will start looking into that one.

[m90]

#99 is probably also related to what I wrote, albeit I'm not sure if this tool should start having any sort of persistence, so I'd maybe not offer this feature, even if it'd be possible.

Also, I wanted to mention a change this big would probably warrant a v3, so some minor breaking changes would probably be ok, see #80

[pixxon]

I am not a fan of introducing persistence to a tool that should back that up. ( It would need to create a backup of its own? )

Regarding REST API, the most I could imagine is a read only visualization of the setup. ( What are the storage options, what volumes are configured, which containers/services are labelled. )