I have yet to find a good answer to this online.
The issue:
I have a 2TB database with only one table (archive). Unfortunately, the hard drive will soon be full and I need to somehow split data.
An extension of the hard disk is not possible in the short term. But there is another hard disk where I could store data on it. This one is 1TB in size.
The database has only one primary filegroup and one file to it.
If I add a new file to the filegroup, can I put this file on the 1TB and the database then writes its data there?
If I add a new file to the filegroup, can I put this file on the 1TB and the database then writes its data there?
Yes. SQL Server uses a "proportional fill algorithm" when a filegroup has multiple files.
Filegroups use a proportional fill strategy across all the files
within each filegroup. As data is written to the filegroup, the SQL
Server Database Engine writes an amount proportional to the free space
in the file to each file within the filegroup, instead of writing all
the data to the first file until full. It then writes to the next
file. For example, if file f1 has 100 MB free and file f2 has 200 MB
free, one extent is given from file f1, two extents from file f2, and
so on. In this way, both files become full at about the same time, and
simple striping is achieved.
File and Filegroup Fill Strategy
So if you add a new file, all (or almost all) of the new allocations will be from that file. And so you can easily add usable space by adding a new file to your primary filegroup.
Add a new file to your primary filegroup, data will be distributed in another disk as per your requirement. Even you can add more log files as well.
in your case, if the log file size is high then you can move your log file(.ldf) to another disk.
Related
I have been trying to use CDBFLite to delete records of a DBF file from records 1 to 5 million or so (in order to decrease the filesize). Due to factors beyond my control, this is something I will have to do every day. The filesize exceeds 2 GB.
However, it takes forever to run the delete commands. Is there a faster way to just eliminate the first X records of a DBF (and thus result in a smaller filesize) ?
As noted by Ethan, if a .DBF file, it typically caps at standard 32-bit OS capacity of 2-gig per single file when it comes to .DBFs unless you are dealing with another software engine such as SyBase Database Advantage which can read/write to .DBF files and exceed the 2 gig capacity.
That said, the DBF standard format has a single character on each record as a "flag" that the record is deleted, yet still retains the space. In order to reduce the size, you would need to PACK the file which actually REMOVES the deleted records and thus will reduce the file size back down.
Now Ethan has options via Python, and I via C#.net and using Microsoft Visual Foxpro OleDb Provider and can offer more, but don't know what you have access to.
If you have VFP (or dBASE) directly, then it should be as simple as getting to the command window and doing
USE [YourTable] exclusive
pack
But I would make a backup copy of the file first as simple precaution.
Here's a very rough outline using my dbf package:
import dbf
import shutil
database = r'\some\path\to\database.dbf'
backup = r'\some\backup\path\database.backup.dbf')
# make backup copy
shutil.copy(database, backup)
# open copy
backup = dbf.Table(backup)
# overwrite original
database = backup.new(database)
# copy over the last xxx records
with dbf.Tables(backup, database):
for record in backup[-10000:]:
database.append(record)
I suspect copying over the last however many records you want will be quicker than packing.
I am using MSSQL 2012 and its feature called File Table to store some big amount of files stored in hierarchical directories. I am referencing the entries on the file from other custom tables via the column stream_id, which is unique for every record on the file table. Sometimes I need to move the files on the file table to some other location on the same file table. So far I have noticed that the stream_id does not change if I move the file to another directory. However, now in the production environment the stream_id does change after the move, so my custom table is referencing a not existing entry on the file table.
For moving the Files I am using File.Move(source, target);
Is there something wrong with the deployment of the file table in my production environment or is it just a feature that the stream_id can sometime change if I change the location?
I haven't found any reference in the internet regarding the stream_id and its lifetime.
The stream_id is the pointer to the blob, the path_locator is the PK of the table. The former refers to the file no matter where it is in the system, the latter refers to whatever file is currently occupying the space at that path location.
My understanding is that the stream_id will not change for the life of that file. If you are seeing the "same" file with a different stream_id, consider whether an application (like MS Word), created a temp file then renamed the temp file to the current file when you saved. This would result in the new file having the same path_locator but a different stream_id.
The number of rows in the table is only 601 records. Looking in the database I cannot see any problems: the initial size is set to 4Gb, autogrowth is set by 1Mb (I then set it to 10%) but this did not make any difference.
Could not allocate space for object 'dbo.Fatawa'.'PK_table' in database 'database' because
the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
The error message is telling you that the PRIMARY filegroup (FG) is full. In SQL Server Management Studio (SSMS), right click the database and selection properties.
On the files table, find the primary data file (*.mdf) and write down the location (drive, path, file).
Go to Windows explorer, click the my computer icon, notice the drives. Each drive will have total space and amount available.
Problem:
Is there any space left on the drive? If not, that is your problem.
Solution:
If there is space left in the data file, shrink the file. If not, create a secondary data file. Move some tables to the data file. Shrink the primary data file after move is complete.
Here is a script from microsoft by Roberto Stefanetti. It will move a table and indexes from one FG to another.
http://gallery.technet.microsoft.com/scriptcenter/c1da9334-2885-468c-a374-775da60f256f
i used the stored procedure:
sp_helpdb (DB name)
this showed me the growth size was too small so i changed it like this:
ALTER DATABASE (DB name)
MODIFY FILE
(NAME=(DB name),FILEGROWTH=20MB);
In my case I use SQL Server 2005 EXPRESS version and this have database limitation to 4 GB max size.
I get this error:
Could not allocate space for object 'dbo.SORT temporary run
storage:440737612283904' in database 'LargeDB' because the 'PRIMARY'
filegroup is full Create disk space by deleting unneeded
files,dropping objects in the filegroup, adding additional files to
the filegroup, or setting autogrowth on for existing files in the
filegroup...
To solve this problem you need to upgrade to SQL Server version 2008 R2 Express Database Size Limit Increased to 10GB, or upgrade your license.
my application play videos files after that user they are registered .(files are larger than 100 MB ) .
Is it better to do I store them on the hard drive and Keep file path in database ?
Or
do I store in database as File Stream Type ?
When data is stored in the database, are more secure against manipulation vs with stored in hard ?
How to provide data security against manipulation ?
Thanks .
There's a really good paper by Microsoft Research called To Blob or Not To Blob.
Their conclusion after a large number of performance tests and analysis is this:
if your pictures or document are typically below 256K in size, storing them in a database VARBINARY column is more efficient
if your pictures or document are typically over 1 MB in size, storing them in the filesystem is more efficient (and with SQL Server 2008's FILESTREAM attribute, they're still under transactional control and part of the database)
in between those two, it's a bit of a toss-up depending on your use
If you decide to put your pictures into a SQL Server table, I would strongly recommend using a separate table for storing those pictures - do not store the employee foto in the employee table - keep them in a separate table. That way, the Employee table can stay lean and mean and very efficient, assuming you don't always need to select the employee foto, too, as part of your queries.
For filegroups, check out Files and Filegroup Architecture for an intro. Basically, you would either create your database with a separate filegroup for large data structures right from the beginning, or add an additional filegroup later. Let's call it LARGE_DATA.
Now, whenever you have a new table to create which needs to store VARCHAR(MAX) or VARBINARY(MAX) columns, you can specify this file group for the large data:
CREATE TABLE dbo.YourTable
(....... define the fields here ......)
ON Data -- the basic "Data" filegroup for the regular data
TEXTIMAGE_ON LARGE_DATA -- the filegroup for large chunks of data
Check out the MSDN intro on filegroups, and play around with it!
1 - depends on how you define "better". In general, I prefer to store binary assets in the database so they are backed up alongside the associated data, but cache them on the file system. Streaming the binary data out of SQL Server for a page request is a real performance hog, and it doesn't really scale.
If an attacker can get to your hard drive, your entire system is compromised - storing things in the database will offer no significant additional security.
3 - that's a whole question in its own right. Too wide for Stack Overflow...
I am doing investigation on filestream (asking on stackoverflow while reading whitepapers and google searching), in my current screnario documents are managed in this way:
1) I have a DB table where I keep the document id and the doc path (like \fileserver\DocumentRepository\file000000001.pdf)
2) I have a document folder (\fileserver\DocumentRepository) where I store the documents
Of course I need to change this to a varbinary(max)/filestream storage.
What is the best way to perform this task?
Is it possible to say "\fileserver\DocumentRepository\file000000001.pdf" is assigned to a varbinary(max) field or I have to explicitly insert it? So somehow tell to the varbinary(max) field: "now you are a pointer to the existing document".
You can't assign an existing file to a varbinary(max)/filestream value. You have to explicitly insert it.
That being said, if for some reason this is not an option for you (e.g. you can't afford copying huge amounts of data or would hit a disk space problem while copying) there are some hacks to do the migration with 0-copy. The trick would be to do the following steps:
Switch the DB to simple recovery model.
Insert placeholder filestream files for all the files you're about to migrate. When inserting, use a varbinary value of 0x. While inserting, collect the (document id/file path) => (filestream file name) pairs.
Stop Sql Server.
Overwrite the empty filestream files with the real documents (use move/hard links to avoid copying data).
Start Sql Server, perform some sanity checks (DBCC) and start a new backup chain.
Obviously, this hack is not recommended and prone do database corruption. :)