I have the initial data from my old database which takes around 6GB. I could "dumpdata" my old database without any problem. But when I attempted to restore them to the new database, I got the MemoryError:
python manage.py loaddata fixtures/initial_data.json
MemoryError: Problem installing fixture 'fixtures/initial_data.json':
Is there any way to make loaddata work with chunks or is it possible to load that big file?
I've wrote this script, which is a fork of django's dumpdata, but dumps data in chunks to avoid MemoryError. And then load these chunks one by one.
Script is available at https://github.com/fastinetserver/django-dumpdata-chunks
Example usage:
1) Dump data into many files:
mkdir some-folder
./manage.py dumpdata_chunks your-app-name
--output-folder=./some-folder --max-records-per-chunk=100000
2) Load data from the folder:
find ./some-folder | egrep -o "([0-9]+_[0-9]+)" | xargs ./manage.py loaddata
PS. I used it to move data from Postgresql to MySQL.
For large database use backup tools for dumping database data instead "django dumpdata".
To load database data use restore tools instead "django loaddata".
Related
There are plenty of resources on how to dump Postgres/Mariadb/MySQL/etc. databases from a volume/container; my question is if I need to do so before backing them up. More explicitly, is it safe to stop my MariaDB container, copy the contents of the volume to another folder, and back that up directly? Are there consequences I should be aware of?
My current export code:
mkdir -p $HOME/backup/mariadb_backup
docker run --rm -v mariadb_volume:/data -v $HOME/backup:/backup ubuntu cp -aruT /data /backup/mariadb_backup
I then run borg on the backup folder.
It is safe to back up the files of a stopped database.
People usually don't want to shut down a database that's providing some service, so they come up with methods how not to do that.
One is run a dump operation that exports the contents of a database while it is serving other requests.
Another is a filesystem snapshot. That is atomically take a snapshot of the files underlying the database so that all files retain their content from a single point in time and then back that up.
The only thing you should not do is back up the files of a running database one by one. You will get an inconsistent copy if you do that.
I'm trying to backup and restore a CouchDB following the official documentation:
https://docs.couchdb.org/en/latest/maintenance/backups.html
"However, you can also copy the actual .couch files from the CouchDB data directory (by default, data/) at any time, without problem. CouchDB’s append-only storage format for both databases and secondary indexes ensures that this will work without issue."
Since the doc seems to not show clearly the steps to restore from files, i copy the entire data folder, build up a local CouchDB docker container and try to paste the files into container opt/couchdb/data folder.
But what i get when i start/restart the container and access localhost:5984 to see the databases, is: "This database failed to load."
What should i do after copy the files? Paste directly should work? What is the right time to paste? Should i create the DBs before?
Thank you all
i've been able to resolve that way:
https://github.com/apache/couchdb/discussions/3436
I think you may need to update the ownership of the backup files on your docker container.
This fixed the issue for me:
# recursively change ownership of data dir to couchdb:couchdb
docker exec <container_id> bash -c 'chown -R couchdb:couchdb /opt/couchdb/data'
Just replace the <container_id> with your docker container id and the destination with the location of couchdb data directory in your container.
So my task is to write an assignment in nodejs and bootstrap with mongodb database.
My next task is to share this with the Assignee in a way so that he/ she can run the project in his/ her local environment.
I can transfer the codes in git and share but how to share the database as well?
You can see MongoDB documentation for that
mongodump -d <database_name> -o <directory_backup>
mongorestore -d <database_name> <directory_backup>
video for mongoexport command
You could use a database-as-a-service like mlab so that the database will be the same on both the machines.
Or if you don't like external database, you could also create a node.js script to init the database the first time the program is run.
Use studio3T, choose your collection. Right click on the collection and export in your desired format. Same way your colleague will import from Studio3T.
I use a Wordpress plugin called 'Shopp'. It stores product images in the database rather than the filesystem as standard, I didn't think anything of this until now.
I have to move server, and so I made a backup, but restoring the backup is proving a horrible task. I need to restore one table called wp_shopp_assets which is 18MB.
Any advice is hugely appreciated.
Thanks,
Henry.
For large operations like this it is better to go to command line. phpMyAdmin gets tricky when lots of data is involved because there are all sorts of timeouts in PHP that can trip it up.
If you can SSH into both servers, then you can do a sequence like the following:
Log in to server1 (your current server) and dump the table to a file using "mysqldump" --- mysqldump --add-drop-table -uSQLUSER -pPASSWORD -h
SQLSERVERDOMAIN DBNAME TABLENAME > BACKUPFILE
Do a secure copy of that file from server1 to server2 using "scp" ---
scp BACKUPFILE USER#SERVER2DOMAIN:FOLDERNAME
Log out of server 1
Log into server 2 (your new server) and import that file into the new DB using "mysql" --- mysql -uSQLUSER -pPASSWORD DBNAME < BACKUPFILE
You will need to replace the UPPERCASE text with your own info. Just ask in the comments if you don't know where to find any of these.
It is worthwhile getting to know some of these command line tricks if you will be doing this sort of admin from time to time.
try HeidiSQL http://www.heidisql.com/
connect to your server and choose the database
go to menu "import > Load sql file" or simply paste the sql file into the sql tab
execute sql (F9)
HeidiSQL is an easy-to-use interface
and a "working-horse" for
web-developers using the popular
MySQL-Database. It allows you to
manage and browse your databases and
tables from an intuitive Windows®
interface.
EDIT: Just to clarify. This is a desktop application, you will connect to your database server remotely. You won't be limited to php script max runtime, or upload size limit.
use bigdupm.
create a folder on your server witch is not easy to guess like "BigDump_D09ssS" or w.e
Download the http://www.ozerov.de/bigdump.php importer file and add them to that directory after reading the instructions and filling out your config information.
FTP The .SQL File to that folder along side the bigdump script and go to your browser and navigate to that folder.
Selecting the file you uploaded will start importing the SQL is split chunks and would be a much faster method!
Or if this is an issue i reccomend the other comment about SSH And mysql -u -p -n -f method!
Even though this is an old post I would like to add that it is recommended to not use database-storage for images when you have more than like 10 product(image)s.
Instead of exporting and importing such a huge file it would be better to transfer the Shopp installation to file-storage for images before transferring.
You can use this free plug-in to help you. Always backup your files and database before performing this action.
What I do is open the file in a code editor, copy and paste into a SQL window within phpmyadmin. Sounds silly, but I swear by it via large files.
It there a tool for django to install some static data (necessary data to have application runing ) to database?
./manage.py syncdb will make the database schema i db, some tool for pushing static data to db ?
Thanks
Fixtures.
EDIT: Better example.
Fill your database with static data you need, then execute command:
./manage.py dumpdata --indent=4 > initial_data.json
After that every ./manage.py syncdb will insert data that you entered the first time into database.
If you are familiar with json format (or xml or yaml) you can edit initial data by hand. For details check out official documentation.