I have a 400GB TFS database (tfs_DefaultCollection). I have ran the Attachment cleaner tool that has informed me it has deleted 200GBs of data. After this and querying the largest tables the row counts are the same and sizes haven't changed. The mdf file size remains the same and so does the top four tables. (tbl_FunctionCoverage, tbl_TestResult, tbl_BuildInformation and tbl_Content). I am assuming there is some form of tidy scripts I need to run maybe? I have executed prc_DeleteUnusedContent and prc_DeleteUnUsedFiles but I believe they are more for version control and workspaces as they made no changes.
I will shrink the database and reindex the tables but as the table row counts and sizes haven't changed i can't see it making much difference.
Any advice is appreciated.
So I think I may have solved my problem...
I wrote a small app to enumerate our build defnitions finding all those marked as deleted. For each deleted build I then deleted all associated test runs and lastly destroying the build. This took around 16 hours to run removing 25,000 builds and about 60,000 test runs. This doesn't appear to change a great deal in the database straight away but some of the tables did reduce in row count.
I then left the database a few days (turned into about 10) and background cleanup jobs that ran appeared to clean up a large amount of the data, they did however run for several days in doing this and in a live system I'm not sure on the performance impact.
Shrinking the database after this reclaimed about 160GB's (took around 10-12 hours) also disabling reporting and removing the Warehouse database (70GB) has bought the total down to database size to 175 GBS.
We used the attachment cleaner tool and it worked for us. I do believe we had to shrink the db as well before we saw the db size drop.
Related
I have several SSIS packages jobs running, and some months ago my disk got full because the size of the SSISDB database.
I noticed that the cleanup_server_retention_window was set to 365 days, and I changed it to one day. (It is a development server, and at this point I really dont care about the history).
The (big) problem, obviously, is now that the transaction log grows a lot and fast.
To prevent this, I start performing a full backup every week and a transaction log backup every day, and the size of the database is now controlled.
However, some more experienced guys are telling me that this is not the best aproach to this issue, but I cant see any problem with it..
I would like to know if there is a better solution for this.
I tried just about everything including changing retention window; it was deleted the transactions but not reducing the log size. For me the allocated log file size grew to 75 GB. Nothing seemed to help.
The main issue has to do with the recovery model
of the SSIS DB that was set to 'Full'. Once I set this to 'Simple' and changed the initial log file size, all was fixed!
I have been monitoring this for the last couple of days just to make sure all is well and it looks fine to me so this operation is safe.
The current log file size is 512KBMB as opposed to 75GB!
The (big) problem, obviously, is now that the transaction log grows a lot and fast.
You will not see this everyday..The Cause of transaction log growth was changing cleanup_server_retention_window ..when you changed the value from 365 to 1,internally it has to do a lot of deletes
I start performing a full backup every week and a transaction log backup every day, and the size of the database is now controlled
I don't see an issue with backing up SSISD.In our instance ,we changed the recovery model to simple and do daily full backups
I fixed this in 3 ways:
adding some missing indexes on the SSISDB database; they should be there after installation of CU4
changed the parameter #delete_batch_size from 1000 to 25 in the stored procedure internal.cleanup_server_retention_window
changed the recoevry model from Full to Simple
Now, when running the SSISDB maintenance job, the transaction log no longer fills up beyond repair causing a database 'crash' / rollback
We work for a small company that cannot afford to pay SQL DBA nor consultation.
What started as a small project has now become a full scale system with a lot of data.
I need someone to help me sort out performance improvements. I realise no-one will be able to help directly and nail this issue completely, but I just want to make sure I have covered my tracks.
OK, the problem is basically we are experiencing time-outs with our queries on cached data. I have increased the time-out time with c# code but I can only go so far when it's becoming ridiculous.
The current setup is a database that has data inserted every 5 / 10 seconds, constantly! During this process we populate tables from csv files. Over night we run data caching processes that reduces the overload on the "inserted" tables. Originally we were able to convert 10+ million rows into say 400000 rows, but as users want more filtering we had to include more data rows and of course increases the number of data cached tables from 400000 to 1-3 million rows.
On my SQL Development Server (which does not have data inserted every 5 seconds) it used to take 30 seconds to run queries on data cache table with 5 million rows, now with indexing and some improvements it's now 17 seconds. The live server has standard SQL Server and used to take 57 seconds, now 40 seconds.
We have 15+ instances running with same number of databases.
So far we have outlined the following ways of improving the system:
Indexing on some of the data cached tables - database now bloated and slows down overnight processes.
Increased CommandTimeout
Moved databases to SSD
Recent improvements likely:
Realised we will have to move csv files on another hard disk and not on the same SSD drive SQL Server databases reside.
Possibly use file-groups for indexing or cached tables - not sure if SQL Server standard will cover this.
Enterprise version and partition table data - customer may pay for this but we certainly can't afford this.
As I said I'm looking for rough guidelines and realise no-one may be able to help fix this issue completely. We're are a small team and no-one has extended SQL Server experience. Customer wants answers and we've tried everything we know. Incidentally they had a small scale version in Excel and said they found no issues so why are we?!?!?
Hope someone can help.
Recently I created a new Heroku app for production and populated it's database with a backup that I took from the staging database.
The problem is that the database size, as shown on Heroku's Postgres webpage for the two databases is different!
The first database, where I took the backup from was 360 MBs and the new database that was populated was only 290 MBs.
No errors showed up during the backup/load process. and taking a backup from the two databases results in the same backup file size (around 40 MBs).
The project is working fine, and the two apps look exactly the same, but I'm concerned that I might have lost some data that would cause troubles in the future.
More info: I'm using the same production database plan on both Apps.
Also, the first database is not attached to the first instance (because it was added from the Postgres management page, not from the App's resources page) and the new database is attached to the new App.
Thanks in advance
It is ok for postgresql DB to consume more space when in use.
The reason of this is its MVCC system. Every time you UPDATE any record in a database it creates another "version" of this record instead of rewriting the previous. This "outdated" records will be deleted by VACUUM process, when there will be no need in them.
So, when you restored your db from backup, it didn't have any "dead" records and its size was less.
Details here http://www.postgresql.org/docs/current/static/mvcc.html and http://www.postgresql.org/docs/current/static/sql-vacuum.html.
P.S. You do not need to worry about it. Postgresql will handle VACUUM automatically.
See if this helps: PostgreSQL database size increasing
Also try to measure the size of each table individually and for those tables where you see differences, compare counts of records: postgresql total database size not matching sum of individual table sizes
Note: Not sure if SO has a specific DB site so I'm putting this question here.
SQL Server 2008 Enterprise.
I'm working to resolve many issues with a database that is 450gb. I'm trying to first get the size of the mdf down so that backups can actually take place. For some very large tables I'm moving indexes to a different file group (which hopefully can be moved to a separate drive later on) in an attempt to get the mdf size down.
My question is, let's say that I have all indexes in a separate file group, not part of the mdf and only the mdf is being backed up. If the system went down and a backup had to be restored, what is the worst case scenario if none of the index file groups were backed up?
I'm assuming that things would be ok but SQL would have to rebuild the indexes which would cause massive slow downs for a bit.
Note: Before you blast me on bad practices, I've inherited these issues and I'm playing firefighter right now.
Creating indexes can take a while, so if you have to recreate them all from scratch you'd potentially lose the definitions (unless properly documented) and have to spend a lot of time waiting for them to build one at a time. Depending on the size of the tables, it can take a while for just one to be created.
I would suggest that as part of what you are doing, you look at what the indexes are actually doing and if you can remove/reduce how many you have. A lot of times developers and DBAs build too many indexes, and a few well thought out ones can take the place of several hastily thrown together indexes.
Check out Brent Ozar's answer on ServerFault: https://serverfault.com/questions/1/how-to-exclude-indexes-from-backups-in-sql-server-2008
When disaster strikes, you restore the
primary filegroup by itself. The data
is suddenly online, but the indexes
are not. However, to get back to
normalcy, you'll need to export that
data into a new clean database and add
indexes from there.
Check out the link for more details (and to give him the upvotes). He also has a link to a tutorial on doing filegroup restores.
There is a SqlServer2000 Database we have to update during weekend.
It's size is almost 10G.
The updates range from Schema changes, primary keys updates to some Million Records updated, corrected or Inserted.
The weekend is hardly enough for the job.
We set up a dedicated server for the job,
turned the Database SINGLE_USER
made any optimizations we could think of: drop/recreate indexes, relations etc.
Can you propose anything to speedup the process?
SQL SERVER 2000 is not negatiable (not my decision). Updates are run through custom made program and not BULK INSERT.
EDIT:
Schema updates are done by Query analyzer TSQL scripts (one script per Version update)
Data updates are done by C# .net 3.5 app.
Data come from a bunch of Text files (with many problems) and written to local DB.
The computer is not connected to any Network.
Although dropping excess indexes may help, you need to make sure that you keep those indexes that will enable your upgrade script to easily find those rows that it needs to update.
Otherwise, make sure you have plenty of memory in the server (although SQL Server 2000 Standard is limited to 2 GB), and if need be pre-grow your MDF and LDF files to cope with any growth.
If possible, your custom program should be processing updates as sets instead of row by row.
EDIT:
Ideally, try and identify which operation is causing the poor performance. If it's the schema changes, it could be because you're making a column larger and causing a lot of page splits to occur. However, page splits can also happen when inserting and updating for the same reason - the row won't fit on the page anymore.
If your C# application is the bottleneck, could you run the changes first into a staging table (before your maintenance window), and then perform a single update onto the actual tables? A single update of 1 million rows will be more efficient than an application making 1 million update calls. Admittedly, if you need to do this this weekend, you might not have a lot of time to set this up.
What exactly does this "custom made program" look like? i.e. how is it talking to the data? Minimising the amount of network IO (from a db server to an app) would be a good start... typically this might mean doing a lot of work in TSQL, but even just running the app on the db server might help a bit...
If the app is re-writing large chunks of data, it might still be able to use bulk insert to submit the new table data. Either via command-line (bcp etc), or through code (SqlBulkCopy in .NET). This will typically be quicker than individual inserts etc.
But it really depends on this "custom made program".