I just started using HeidiSQL to manage my databases at work. I was previously using NaviCat, and I was able to simply drag and drop a database from one server to a database on another server and copy all the tables and data--piece of cake, backed up. Similar functionality isn't obvious in Heidi, but I decided using mysqldump is a standard way of backing up a database and I should get to know it.
I used Heidi to perform the dump--creating databases, creating tables, inserting data, and making one single SQL file. This was performed on one single database that is 801.7 MB (checksum: 1755734665). When I executed the dump file on my local doppelganger database it appeared to work flawlessly, however the database size is 794.0 MB (checksum: 2937674450).
So, by creating a new database from a mysqldump of a database I lost 7.7 MB of something (and the databases have different checksums--not sure what that means though). I searched and found a post saying that performing a mysqldump can optimize a database, but could not find any conclusive information.
Am I being a helicopter parent? Is there any easy way to make sure I didn't lose data/structure somehow?
Thank you,
Kai
Related
[This is a different approach to a similar question I did couple days ago, I recognize the approach of "cloning" was wrong as this would involve too many changes and offline the actual DB]
EXPLANATION:
I have a set of systems-installations (WebApp + SQL Server database), where the databases have two groups of objects, one small "core" and second and large "instance_specific".
Those systems need to be rebuilt base on another system from time to time (like an Excel financial scenario case), a rebuild of a system will be best if "core" can stay, and everything else delete and importing from a source database (the source database can be different from time to time)
SO FAR:
CREATE DATABASE Database1_copy AS COPY OF Database1;
is not an option as will need to rebuild all core and neither Azure manage well DB created from SSMS, neither establish the needed connections, neither maintain the users and roles... (not good idea to change the db)
Same from recovery from backup / Import Data-Tier Application ...
Script seems really not a good option, doesn't include SEQUENCES (probably other objects as well) and it weight too much to be practical (see notes at the end). How should I load a 360mb script on SSMS to execute it on the destination system?
SQLAzureMW v5.15.6 Release Binary for SQL Server 2014, not able really to try as I run out of memory and trow and exception (with 16gb ram). Databases too big, or buggy ?
Solutions based on manual usage of backup sql functions, as:
https://www.mssqltips.com/sqlservertip/1243/auto-generate-sql-server-database-restore-scripts/
are not supported in Azure ...
To get an instance "clear" and ready to receive the backup I really like the idea of the next post, filtering objects out that belong to core:
Drop all the tables, stored procedures, triggers, constraints and all the dependencies in one sql statement
And due the mount of data, it would be great if "data-flows" goes from db to db directly, not back to local, and even better if it could be executed from the destination DB
SO:
In my best dreams I imagine a SP (similar to the one form bellow on stackoverflow) that goes object by object, reconstruct-it, and loads the data within, and even initialize sequences to the last used values.
I try to copy TABLES (with constrains and index, AND DATA), SEQUENCES (with acual value), VIEWS, SPs, FUNCTIONS, and I think that's all (do not need to pass USERS and ROLES)
I come also to Apexsql and RedBeltTools, haven't try yet, but maybe the solution, even I would prefer not to rely in 3rd party software that runs locally.
Am I out of options?
Should I start digging how to build my own SP migration tool?
(I am not really sure how/where to start ...)
Just some numbers of an actual source database:
CHECK_CONSTRAINT 12
DEFAULT_CONSTRAINT 259
FOREIGN_KEY_CONSTRAINT 145
PRIMARY_KEY_CONSTRAINT 162
SEQUENCE_OBJECT 7
SERVICE_QUEUE 3
SQL_INLINE_TABLE_VALUED_FUNCTION 1
SQL_SCALAR_FUNCTION 27
SQL_STORED_PROCEDURE 765
SQL_TABLE_VALUED_FUNCTION 6
UNIQUE_CONSTRAINT 54
USER_TABLE 268
VIEW 42
Other than Users and Roles, I think nothing else is missed here, isn't it?
Script DB delivers a 360 mb (over 500k lines)
and it seems to not include SEQUENCES at least
(SMSS scripting let me just chose: Tables/Views/SPs/UserDefFunctions/Users/DatabaseRole)
Since you have mention you want to replicate 90% of anAzureSQL database to an existent one. I could see this scenario as comparing two databases in schema and data and trying to send selected changes from an origin db to a destiny db.
That said I would recommend using SQL Server Data Tools, it has a complete suite of tools that could aid you in that scenario, and multiple ways to update the destiny db: inline, via script, via dacpac, via bacpac.
Also since your question is so open, I don't know if it will solve your scenario, but try looking at it and its compare functionality.
I need to backup a Drupal database it is huge. So it has over 1500 tables (don't blame me, its a Drupal thing) and is 10GB in size.
I couldn't do it with PHPMyAdmin, I just got an error when it started to build the .sql file.
I want to make sure I wont break anything or take the server down or anything when I try to back it up.
I was going to attempt a mysqldump on my server and then copy the file down locally but realised that this may cause unforeseen problems. So my question to you is, is it safe to use mysqldump on so many tables at once and even if it is safe are there any problems such a huge file could lead to in the future for rebuilding the database?
Thanks for the input guys.
is it safe to use mysqldump on so many tables at once
I run daily backups with mysqldump on servers literally 10x this size: 15000+ tables, 100+ GB.
If you have not examined the contents of a file produced by mysqldump ... you should, because to see its output is to understand why it is an intrinsically safe backup utility:
The backups are human-readable, and consist entirely of the necessary SQL statements to create a database exactly like the one you backed up.
In this form, their content is easily manipulated with ubiquitous tools like sed and grep and perl, which can be used to pluck out just one table from a file for restoration, for example.
If a restoration fails, the error will indicate the line number within the file where the error occurred. This is usually related to buggy behavior in the version of the server where the backup was created (e.g. MySQL Server 5.1 allowed you to create views in some situations where the server itself would not accept the output of its own SHOW CREATE VIEW statement. The create statement was not considered -- by the same server -- to be a valid view definition, but this was not a defect in mysqldump, or in the backup file, per se.)
Restoring from a mysqldump-created backup is not lightning fast, because the server must execute all of those SQL statements, but from the perspective of safety, I would argue that there isn't a safer alternative, since it is the canonical backup tool and any bugs are likely to be found and fixed by virtue of the large user base, if nothing else.
Do not use the --force option, except in emergencies. It will cause the backup to skip over any errors encountered on the server while the backup is running, causing your backup to be incomplete with virtually no warning. Instead, find and fix any errors that occur. Typical errors during backup are related to views that are no longer valid because they reference tables or columns that have been renamed or dropped, or where the user who originally created the view has been removed from the server. Fix these by redefining the view, correctly.
Above all, test your backups by restoring them to a different server. If you haven't done this, you don't really have backups.
The output file can be compressed, usually substantially, with gzip/pigz, bzip2/bpzip2, xz/pixz, or zpaq. These are listed in approximate order by amount of space saved (gzip saves the least, zpaq saves the most) and speed (gzip is the fastest, zpaq is the slowest). pigz, pbzip2, pixz, and zpaq will take advantage of multiple cores, if you have then. The others can only use a single core at a time.
Use mysqlhotcopy it is well working with large databases
Work only MyISAM and ARCHIVE-tables.
Work only on the server where the database is stored.
This utility is deprecated in MySQL 5.6.20 and removed in MySQL 5.7
I have imported about 200 GB of census data into a postgreSQL 9.3 database on a Windows 7 box. The import process involves many files and has been complex and time-consuming. I'm just using the database as a convenient container. The existing data will rarely if ever change, and will be updating it with external data at most once a quarter (though I'll be adding and modifying intermediate result columns on a much more frequent basis. I'll call the data in the database on my desktop the “master.” All queries will come from the same machine, not remote terminals.
I would like to put copies of all that data on three other machines: two laptops, one windows 7 and one Windows 8, and on a Ubuntu virtual machine on my Windows 7 desktop as well. I have installed copies of postgreSQL 9.3 on each of these machines, currently empty of data. I need to be able to do both reads and writes on the copies. It is OK, and indeed I would prefer it, if changes in the daughter databases do not propagate backwards to the primary database on my desktop. I'd want to update the daughters from the master 1 to 4 times a year. If this wiped out intermediate results on the daughter databases this would not bother me.
Most of the replication techniques I have read about seem to be worried about transaction-by-transaction replication of a live and constantly changing server, and perfect history of queries & changes. That is overkill for me. Is there a way to replicate by just copying certain files from one postgreSQL instance to another? (If replication is the name of a specific form of copying, I'm trying to ask the more generic question). Or maybe by restoring each (empty) instance from a backup file of the master? Or of asking postgreSQL to create and export (ideally on an external hard drive) some kind of postgreSQL binary of the data that another instance of postgreSQL can import, without my having to define all the tables and data types and so forth again?
This question is also motivated by my desire to work around a home wifi/lan setup that is very slow – a tenth or less of the speed of file copies to an external hard drive. So if there is a straightforward way to get the imported data from one machine to another by transference of (ideally compressed) binary files, this would work best for my situation.
While you could perhaps copy the data directory directly as mentioned by Nick Barnes in the comments above, I would recommend using a combination of pg_dump and pg_restore, which will dump a self-contained file which can then be dispersed to the other copies.
You can run pg_dump on the master to get a dump of the DB. I would recommend using the options -Fc -j3 to use the custom binary format (instead of dumping in SQL format; this should be much smaller and perhaps faster as well) and will dump 3 tables at once (this can be adjusted up or down depending on the disk throughput capabilities of your machine and the number of cores that it has).
Then you run dropdb on the copies, createdb to recreate an empty DB of the same name, and then run pg_restore on that new empty DB to restore the dump file to the DB. You would want to use the options -d <dbname> -f <dump_file> -j3 (again adjusting the number for -j according to the abilities of the machine).
When you want to refresh the copies with new content from the master DB, simply repeat the above steps
My issue is that I have a SQL database that is the output of a SQL project in an ASP.NET solution. This output consists of 2 datafiles, lets call them MainDB.mdf and DocContent.mdf, and the necessary log files. DocContent.mdf has one table, doc_content, which, you guessed it, stores the bytes for documents. There are multiple dev's working on this project, and we each have a (localdb) instance of the DB to use.
What I need to make happen is to attach MY MainDB.mdf referring to dev B's DocContent.mdf. The data FK's match 1:1, the schema is identical, the only difference is that Dev B populated the varbinary field with uploaded documents.
SQL seems to be doing everything in its power to prevent this. I get:
I realize I could do a Data Compare and pump the data in that way, but unfortunately Dev B and I are not at the same location and the logistics of him sending his full DB output would be harder than finding a work around with what I've got (I hope, anyway).
Surely there is a way to tell SQL to shut up and just use the datafile provided?
The only info I could find on this subject referred to attaching a DB without its LDF. Anyone know how to make this scenario work?
So, for anyone interested, the short answer is 'no'. There are a number of convoluted steps that you you can take to force the database into SUSPECT mode and possible even get it into single user emergency mode, but it was not worth the effort and the end result is that you will have to hack into the sys tables in order to make it limp along this way.
In the end it was easier to get a full transfer of Dev B's data files.
I need to do some structural changes to a database (alter tables, add new columns, change some rows etc) but I need to make sure that if something goes wrong i can rollback to initial state:
All needed changes are inside a SQL script file.
I don't have administrative access to database.
I really need to ensure the backup is done on server side since the BD has more than 30 GB of data.
I need to use sqlplus (under a ssh dedicated session over a vpn)
Its not possible to use "flashback database"! It's off and i can't stop the database.
Am i in really deep $#$%?
Any ideas how to backup the database using sqlplus and leaving the backup on db server?
better than exp/imp, you should use rman. it's built specifically for this purpose, it can do hot backup/restore and if you completely screw up, you're still OK.
One 'gotcha' is that you have to backup the $ORACLE_HOME directory too (in my experience) because you need that locally stored information to recover the control files.
a search of rman on google gives some VERY good information on the first page.
An alternate approach might be to create a new schema that contains your modified structures and data and actually test with that. That presumes you have enough space on your DB server to hold all the test data. You really should have a pretty good idea your changes are going to work before dumping them on a production environment.
I wouldn't use sqlplus to do this. Take a look at export/import. The export utility will grab the definitions and data for your database (can be done in read consistent mode). The import utility will read this file and create the database structures from it. However, access to these utilities does require permissions to be granted, particularly if you need to backup the whole database, not just a schema.
That said, it's somewhat troubling that you're expected to perform the tasks of a DBA (alter tables, backup database, etc) without administrative rights. I think I would be at least asking for the help of a DBA to oversee your approach before you start, if not insisting that the DBA (or someone with appropriate privileges and knowledge) actually perform the modifications to the database and help recover if necessary.
Trying to back up 30GB of data through sqlplus is insane, It will take several hours to do and require 3x to 5x as much disk space, and may not be possible to restore without more testing.
You need to use exp and imp. These are command line tools designed to backup and restore the database. They are command line tools, which if you have access to sqlplus via your ssh, you have access to imp/exp. You don't need administrator access to use them. They will dump the database (with al tables, triggers, views, procedures) for the user(s) you have access to.