I'm using visual Studio 2019 community SQL Server localdb on Windows 10 Pro 202H. I've been trying 'restore' the WideWorldImporters database from WideWorldImporters-Full.bak (downloaded from GitHub) to my localdb instances without any success.
This has happened so far:
Qyery: (*** = My Username)
USE master
RESTORE DATABASE WideWorldImporters
FROM disk = 'C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Backup\WideWorldImporters-Full.bak'
WITH MOVE 'WWI_Primary' TO 'C:\Users\***\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\MSSQLLocalDB\WideWorldImporters.mdf',
MOVE 'WWI_UserData' TO 'C:\Users\***\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\MSSQLLocalDB\WideWorldImporters_UserData.ndf',
MOVE 'WWI_Log' TO 'C:\Users\***\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\MSSQLLocalDB\WideWorldImporters.ldf',
MOVE 'WWI_InMemory_Data_1' TO 'C:\Users\***\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\MSSQLLocalDB\WideWorldImporters_InMemory_Data_1',
REPLACE
Message (pane) :
Processed 1464 pages for database 'WideWorldImporters', file 'WWI_Primary' on file 1.
Processed 53096 pages for database 'WideWorldImporters', file 'WWI_UserData' on file 1.
Processed 33 pages for database 'WideWorldImporters', file 'WWI_Log' on file 1.
Processed 3862 pages for database 'WideWorldImporters', file 'WWI_InMemory_Data_1' on file 1.
100% | No issues found
Executing query... (is running on and on)
Output (General) is empty
All the stuff notified in message window can be found in desitination folder:
WideWorldImportes_UserData.ndf (2 097 152 kb)
WideWorldImportes.mdf (1 048 576 kb)
WideWorldImportes.ldf ( 102 400 kb)
\WideWorldImportes_InMemory_Data_1\
filestream.hdr ( 1 kb)
\$FSLOG (empty )
\$HKv2
{1E6DC7E6-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 2 048 kb)
{3E231B6B-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 20 kb)
{4B9D83BE-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 2 048 kb)
{6E82296C-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 16 384 kb)
{6F44D507-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
{07FEB052-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 20 kb)
{7C4940C1-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
{9A77966E-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 4 096 kb)
{28CE0994-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 16 384 kb)
{63F1F945-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 2 048 kb)
{79B6C099-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 4 096 kb)
{122A2C90-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 16 384 kb)
{285FCA71-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 4 kb)
{421C57F0-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 4 096 kb)
{A54BA375-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
{C818BEE6-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 30 836 kb)
{CB6FF974-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
{F6F88B52-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
{F756E9B8-xxxx-xxxx-xxxx-xxxxxxxxxxxx}.hkckp ( 1 024 kb)
Executing query... (still running, no changes in size of files for the last two hours)
This is where I'm now
and I'm getting mad
As you have gone for the option of RESTORE WITH REPLACE, it has overwritten existing WideWorldImporters database. As you have done restore with replace option, there could be uncommitted transactions, which it is trying to clean up. It will come back online after some time.
You have to be careful with REPLACE option. From MSDN
REPLACE Option Impact
REPLACE should be used rarely and only after
careful consideration. Restore normally prevents accidentally
overwriting a database with a different database. If the database
specified in a RESTORE statement already exists on the current server
and the specified database family GUID differs from the database
family GUID recorded in the backup set, the database is not restored.
This is an important safeguard.
The REPLACE option overrides several important safety checks that
restore normally performs. The overridden checks are as follows:
Restoring over an existing database with a backup taken of another
database.
With the REPLACE option, restore allows you to overwrite an existing
database with whatever database is in the backup set, even if the
specified database name differs from the database name recorded in the
backup set. This can result in accidentally overwriting a database by
a different database.
Restoring over a database using the full or bulk-logged recovery model
where a tail-log backup has not been taken and the STOPAT option is
not used.
With the REPLACE option, you can lose committed work, because the log
written most recently has not been backed up.
Overwriting existing files.
For example, a mistake could allow overwriting files of the wrong
type, such as .xls files, or that are being used by another database
that is not online. Arbitrary data loss is possible if existing files
are overwritten, although the restored database is complete.
You can read, how to come out of the restoring mode here: https://www.mssqltips.com/sqlservertip/5460/sql-server-database-stuck-in-restoring-state/
Related
One of my production machines (SQL Server Express 2012) has not been performing too well. I started running the WhoIsActive script and I've been getting a lot of these wait types:
(10871ms)PREEMPTIVE_OS_AUTHZINITIALIZECON
They always occur when calling a function that checks certain user privileges. If I understand this correctly, the function had to wait almost 11 seconds for the Windows function AuthzInitializeContextFromSid (see https://www.sqlskills.com/help/waits/preemptive_os_authzinitializecontextfromsid/).
Am I correct in my assumption? (full output below)
I couldn't find any info online about this wait type going hayrwire. What could be causing this?
Full output:
00 00:00:10.876 75
<?query --
select #RetValue = ([dbo].[Users_IsMember]('some_role_name', #windowsUserName)
| is_srvrolemember('SysAdmin', #windowsUserName))
--?>
<?query --
MyDB.dbo.StoredProcName;1
--?>
DOMAIN\User (10871ms)PREEMPTIVE_OS_AUTHZINITIALIZECON master: 0 (0 kB),tempdb: 0 (0 kB),MyDB: 0 (0 kB) 10,875 0 0 NULL 93 0 0 <ShowPlanXML xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan" Version="1.5" Build="11.0.7001.0"><BatchSequence><Batch><Statements><StmtSimple StatementText="select #RetValue = ([dbo].[Users_IsMember]('some_role_name', #windowsUserName)
| is_srvrolemember('SysAdmin', #windowsUserName))
" StatementId="1" StatementCompId="49" StatementType="ASSIGN WITH QUERY" RetrievedFromCache="true" /></Statements></Batch></BatchSequence></ShowPlanXML> 3 runnable NULL 0 NULL ServerName AppName .Net SqlClient Data Provider 2018-12-17 09:29:35.413 2018-12-17 09:29:35.413 0 2018-12-17 09:29:46.447
In my experience the PREEMPTIVE_OS wait stats are related to an imbalance between how much memory is allocated to SQL Server vs. the Windows OS itself.
In this case, the OS is being starved for memory resources. You might try either adding more total memory to the box, or ensuring that SQL Server is configured to only use 80 percent of the total memory installed on the instance. Or both.
Note - this is not a blanket statement on how to configure memory for SQL server, but rather a good place to start with tuning for PREEMPTIVE_OS related wait types.
I made a database (150GB) to index the Bitcoin blockchain.
table 1 : id, block_height, block_hash : 500 000 lines
table 2 : id, block_height, transaction_hash : 780 millions lines
table 3 : id, transaction_hash, address : 480 millions lines
On a i7#3Ghz, 16GB RAM, Windows 10, SSD SATA3 I tried adding an index on table3.address. The RAM goes to 100% and after 30h there was an I/O error and the index was not created. I tried a select distinct on table3.address, after 86h of my SSD and RAM being at 100% I decided to kill the SQLite process.
What can I do? I'm going toward my custom solution : text files, one text file per address, per transactions, per block. Want to know all the unique address? List the files in the address folder. Want to know what happened in a transaction? Open the file with its hash.txt.
For example:
Given:
An output of the dumpheap WinDbg command containing about 3.5 million rows.
Needed:
Break into columns and dump it into a relational database
The output looks like this:
Opened log file 'c:\Users\MKHARI~1\AppData\Local\Temp\!dumpheap_-mt_00007fff6c9c16b8.log'
0:000> !dumpheap -mt 00007fff6c9c16b8
Address MT Size
0000009347e51018 00007fff6c9c16b8 30
0000009347e51038 00007fff6c9c16b8 46
0000009347e51068 00007fff6c9c16b8 30
0000009347e51088 00007fff6c9c16b8 56
0000009347e510c0 00007fff6c9c16b8 30
0000009347e510e0 00007fff6c9c16b8 94
0000009347e51140 00007fff6c9c16b8 30
...
0000009de2474cd8 00007fff6c9c16b8 52
0000009de2474d10 00007fff6c9c16b8 44
0000009777e71070 00007fff6c9c16b8 130594
Statistics:
MT Count TotalSize Class Name
00007fff6c9c16b8 3645317 28984711048 System.String
Total 3645317 objects
0:000> .logclose
Closing open log file c:\Users\MKHARI~1\AppData\Local\Temp\!dumpheap_-mt_00007fff6c9c16b8.log
As you can see there are a few lines at the beginning and at the end to be ignored, the rest are just three values delimited with spaces and aligned to look nice for a human. This is not a machine friendly format.
Of course, I have a C# code to dump it into a Sql Server database. One has to:
Parse command line arguments (and be able to output the help on them)
Create a new database or reset an existing one with the schema matching our needs. This should include a Table UDT for the records we are going to dump into the database.
Parse the output into three columns and write into the database in batches using TVP
Not a big deal for a programmer, but may be a show stopper for a Support person, who can analyse a core dump, but does not write non trivial code as part of his/her duties.
So, my question is are there any tools that can help out here? The relational database can be the Sql Server or any other free relational database (Sql Server is not free, but we use it already).
We have the application that writes logs in Azure SQL tables. The structure of the table is the following.
CREATE TABLE [dbo].[xyz_event_history]
(
[event_history_id] [uniqueidentifier] NOT NULL,
[event_date_time] [datetime] NOT NULL,
[instance_id] [uniqueidentifier] NOT NULL,
[scheduled_task_id] [int] NOT NULL,
[scheduled_start_time] [datetime] NULL,
[actual_start_time] [datetime] NULL,
[actual_end_time] [datetime] NULL,
[status] [int] NOT NULL,
[log] [nvarchar](max) NULL,
CONSTRAINT [PK__crg_scheduler_event_history] PRIMARY KEY NONCLUSTERED
(
[event_history_id] ASC
)
)
Table stored as clustered index by scheduled_task_id column (non-unique).
CREATE CLUSTERED INDEX [IDX__xyz_event_history__scheduled_task_id] ON [dbo].[xyz_event_history]
(
[scheduled_task_id] ASC
)
The event_history_id generated by the application, it's random (not sequential) GUID. The application either creates, updates and removes old entities from the table. The log column usually holds 2-10 KB of data, but it can grow up to 5-10 MB in some cases. The items are usually accessed by PK (event_history_id) and the most frequent sort order is event_date_time desc.
The problem we see after we lowered performance tier for the Azure SQL to "S3" (100 DTUs) is crossing transaction log rate limits. It can be clearly seen within sys.dm_exec_requests table - there will be records with wait type LOG_RATE_GOVERNOR (msdn).
Occurs when DB is waiting for quota to write to the log.
The operations I've noticed that cause big impact on log rate are deletions from xyz_event_history and updates in log column. The updates made in the following fashion.
UPDATE xyz_event_history
SET [log] = COALESCE([log], '') + #log_to_append
WHERE event_history_id = #id
The recovery model for Azure SQL databases is FULL and can not be changed.
Here is the physical index statistics - there are many pages that crossed 8K per row limit.
TableName AllocUnitTp PgCt AvgPgSpcUsed RcdCt MinRcdSz MaxRcdSz
xyz_event_history IN_ROW_DATA 4145 47.6372868791698 43771 102 7864
xyz_event_history IN_ROW_DATA 59 18.1995058067705 4145 11 19
xyz_event_history IN_ROW_DATA 4 3.75277983691623 59 11 19
xyz_event_history IN_ROW_DATA 1 0.914257474672597 4 11 19
xyz_event_history LOB_DATA 168191 97.592290585619 169479 38 8068
xyz_event_history IN_ROW_DATA 7062 3.65090190264393 43771 38 46
xyz_event_history IN_ROW_DATA 99 22.0080800593032 7062 23 23
xyz_event_history IN_ROW_DATA 1 30.5534964170991 99 23 23
xyz_event_history IN_ROW_DATA 2339 9.15620212503089 43771 16 38
xyz_event_history IN_ROW_DATA 96 8.70488015814184 2339 27 27
xyz_event_history IN_ROW_DATA 1 34.3711391153941 96 27 27
xyz_event_history IN_ROW_DATA 1054 26.5034840622683 43771 28 50
xyz_event_history IN_ROW_DATA 139 3.81632073140598 1054 39 39
xyz_event_history IN_ROW_DATA 1 70.3854707190511 139 39 39
Is there a way to reduce transaction log usage?
How does SQL Server log update transactions as in example above? Is it just "old" plus "new" value? (that would conceivably make adding little pieces of data frequently quite inefficient in terms of transaction log size)
UPDATE (April, 20):
I've made some experiments with suggestions in answers and was impressed by difference that INSERT instead of UPDATE makes.
As per following msdn article about SQL Server Transaction log internals (https://technet.microsoft.com/en-us/library/jj835093(v=sql.110).aspx):
Log records for data modifications record either the logical operation
performed or they record the before and after images of the modified
data. The before image is a copy of the data before the operation is
performed; the after image is a copy of the data after the operation
has been performed.
This automatically makes the scenario with UPDATE ... SET X = X + 'more' highly inefficient in terms of transaction log usage - it requires "before image" capture.
I've created simple test suite to test original way of adding data to "log" column versus the way where we just insert new piece of data into the new table. The results I got quite astonishing (at lest for me, not too experienced with SQL Server guy).
The test is simple: 5'000 times add 1'024 character long parts of log - just 5MB of text as the result (not too bad as one might think).
FULL recovery mode, SQL Server 2014, Windows 10, SSD
UPDATE INSERT
Duration 07:48 (!) 00:02
Data file grow ~8MB ~8MB
Tran. Log grow ~218MB (!) 0MB (why?!)
Just 5000 updates that add 1KB of data can hang out SQL Server for 8 minutes (wow!) - I didn't expect that!
I think original question is resolved at this point, but the following ones raised:
Why transaction log grow looks linear (not quadratic as we can expect when simply capturing "before" and "after" images)? From the diagram we can see that "items per second" grows proportionally to the square root - it's as expected if overhead grows linearly with amount of items inserted.
Why in case with inserts transaction log appears to have the same size as before any inserts at all?
I've took a look on the transaction log (with Dell's Toad) for the case with inserts and looks like only last 297 items are in there - conceivably transaction log got truncated, but why if it's FULL recovery mode?
UPDATE (April, 21).
DBCC LOGINFO output for case with INSERT - before and after. The physical size of the log file matches the output - exactly 1,048,576 bytes on disk.
Why it looks like transaction log remains still?
RecoveryUnitId FileId FileSize StartOffset FSeqNo Status Parity CreateLSN
0 2 253952 8192 131161 0 64 0
0 2 253952 262144 131162 2 64 0
0 2 253952 516096 131159 0 128 0
0 2 278528 770048 131160 0 128 0
RecoveryUnitId FileId FileSize StartOffset FSeqNo Status Parity CreateLSN
0 2 253952 8192 131221 0 128 0
0 2 253952 262144 131222 0 128 0
0 2 253952 516096 131223 2 128 0
0 2 278528 770048 131224 2 128 0
For those who interested I've recorded "sqlserv.exe" activities using Process Monitor - I can see that file being overwritten again and again - looks like SQL Server treats old log items as no longer needed by some reason: https://dl.dropboxusercontent.com/u/1323651/stackoverflow-sql-server-transaction-log.pml.
UPDATE (April, 24). Seems I've finally started to understand what is going on there and want to share with you. The reasoning above is true in general, but has serious caveat that also produced confusion about strange transaction log re-usage with INSERTs.
Database will behave like in SIMPLE recovery mode until first full
backup is taken (even though it's in FULL recovery mode).
We can treat numbers and diagram above as valid for SIMPLE recovery mode, and I have to redo my measurement for real FULL - they are even more astonishing.
UPDATE INSERT
Duration 13:20 (!) 00:02
Data file grow 8MB 11MB
Tran. log grow 55.2GB (!) 14MB
You are violating one of the basic tenants of the normal form with the log field. The log field seams to be holding an appending sequence of info related to the primary. The fix is to stop doing that.
1 Create a table. xyz_event_history_LOG(event_history_id,log_sequence#,log)
2 stop doing updates to the log field in [xyz_event_history], instead do inserts to the xyz_event_history_LOG
The amount of data in your transaction log will decrease GREATLY.
The transaction log contains all the changes to a database in the order they were made, so if you update a row multiple times you will get multiple entries to that row. It does store the entire value, old and new, so you are correct that multiple small updates to a large data type such as nvarchar(max) would be inefficient, you would be better off storing the updates in separate columns if they are only small values.
Does anyone know how I can query the database to find what devices a database uses?
There is the sysdatabases table and the sysdevices table but I don't know how to link them
Anyone know?
The best way is to run sp_helpdb against the database you're interested in:
1> sp_helpdb tempdb2
2> go
... other stuff here...
device_fragments size usage created free kbytes
------------------------------ ------------- -------------------- ------------------------- ----------------
tempdb2data 2048.0 MB data only Dec 17 2008 11:42AM 2086568
tempdb2log 2048.0 MB log only Dec 17 2008 11:42AM not applicable
tempdb2log 2048.0 MB log only Dec 17 2008 11:42AM not applicable
tempdb2data 2048.0 MB data only Dec 17 2008 11:43AM 2088960
tempdb2log 4096.0 MB log only Dec 17 2008 11:44AM not applicable
--------------------------------------------------------------
log only free kbytes = 8355836
1 Just a note re your first question. If you USE database first, you will get even more detail in the report.
2 Do you still need the second question answered, how to link sysdatabase and sysdevices, as in, are you writing queries against the catalogue ? If so, I need your ASE version, the answers are different.