I have a cloud VPS (costs a few $ per month) with Nextcloud instance
- My notebook and phone have folders synced with nextcloud.
- Web service checks for new files every minute in these folders and process them by different logic based on folder names
- Processing is done in a few steps: save original file raw data into mysql table. Then zip and put file into special archive folder.
- Process saved raw data from mysql table into key/values and insert them into another mysql table.
- For a fast queries and processing mysql tables are being mirrored into clickhouse instance
- mysql / clickhouse db being backed up every few hours and zipped backup put into folder. Folder is synced across notebook / pc / phone
- R scripts ran by cron do most job of data processing
This allows to avoid work with file system in data processing scripts, all data is in mysql table which are single place where all raw and processed data can be accessed. This also easily allow to re-process data if needed.
Backups are also stored from different places / types - file system archive folder, mysql backup, clickhouse backup.
Also i have a single table for all api’s i’m using (withings, polar, oura etc) where the api call request & request params, api url and response are stored. This table is also used as single place where all raw data from api calls being stored and can be re-processed if needed, without making api calls again.