Repository navigation
Add scripts to migrate users from WebODM - #11
MJohnson459 wants to merge 7 commits into
Conversation
When forking from WebODM the defaults were changed. This doesn't affect the database but does avoid them being bundled in a later change. I also added a CI check to catch this kind of missing migration.
0008 and 0010 imported ProcessingNode from nodeodm.models, so their queries selected every column the current model has. Adding a field to ProcessingNode in a later migration would make fresh installs fail at 0008 with "column does not exist". Resolve the model through apps.get_model instead, as the Django migrations docs require.
There are two main variants we need to consider. 1. A default user will have the data stored in a docker volume. 2. Expert users may use --db-dir to store the data on the native filesystem. Both cases have extra checks in place to ensure we do not lose any data. The full migration in both cases is carefully constructed to allow WebODM to function afterwards if the migration fails. Data will always be preserved. On first boot of WebUI, we check to see if the WebUI database exists. If it doesn't, we prompt for the migration process. If accepted, we start up both the old and new database and run `pg_dump` and pipe it to the new database. Finally once we know the data is secure, in the case of a native disk, we swap the new and old files. This gives us an updated Postgres database. We don't copy or modify the media files to avoid excessive disk consumption. The next step is to migrate the Django models. This is done automatically when the app is first run. If the user declines to migrate their data, it is possible to still do the migration as long as the new database remains empty. The implementation is split into a python file `importwebodm.py` and a bash file `import-webodm.sh`. This is to aid reusability.
On Windows, communicate() starts a reader thread for pg_dump's stdout even though it was closed after being handed to psql, and that thread raises "read of closed file".
WebODM and WebUI used the same container names (db, webapp, worker, broker), so a WebODM installation stopped with webodm.sh stop blocked WebUI from creating its containers. Service names are unchanged, so the hostnames stored in the database (db, broker, node-odm-1) still resolve. The import checks for a running WebODM database by image, not by name.
Both layouts now copy from a source into a target that this run owns: the docker volume webodm_dbdata into this installation's volume, or the directory given with --webodm-db-dir into the one given with --db-dir (or the volume). The WebODM database is never written to. A start on a WebODM database directory is refused with the import command. Previously the path layout started Postgres 18 inside the WebODM directory and swapped the two clusters afterwards, which needed a temporary directory, its removal on failure, a leftover check and a wait for the WebODM entrypoint's chown to finish. A failed import removes the database volume it created, so the next start offers the import again instead of starting empty. docker run is called with MSYS_NO_PATHCONV=1 where container paths are passed, so Git Bash does not rewrite them into Windows paths.
|
While testing on Windows I found a few more issues. First, Git Bash rewrites arguments that look like POSIX paths before docker sees them, so Second, WebODM and WebUI used the same container names, so a WebODM stopped with Third, the Instead the import now copies into a separate target: I will note that I'm struggling to keep the bash under control. In future we might move these scripts to Python if we can afford adding that as a requirement for the native install. |
We updated both Django and Postgres when forking to WebUI. Both require migrations.
There are two main variants we need to consider.
filesystem.
Both cases have extra checks in place to ensure we do not lose any data. The full migration in both cases is carefully constructed to allow WebODM to function afterwards if the migration fails. Data will always be preserved.
On first boot of WebUI, we check to see if the WebUI database exists. If it doesn't, we prompt for the migration process. If accepted, we start up both the old and new database and run
pg_dumpand pipe it to the new database. Finally once we know the data is secure, in the case of a native disk, we swap the new and old files. This gives us an updated Postgres database. We don't copy or modify the media files to avoid excessive disk consumption.If the user declines to migrate their data, it is possible to still do the migration as long as the new database remains empty.
The implementation is split into a python file
importwebodm.pyand a bash fileimport-webodm.sh. This is to aid reusability.Once the postgres migration is complete, the next step is to migrate the Django models. This is done automatically when the app is first run. eg.
I've been testing this locally with dummy data. We obviously need to be extra careful testing this so I would appreciate it if others could also test to make sure. If you do want to help, please make sure you have a spare copy of the data if its important!