We have an older vendor supplied application that is earmarked for platform upgrades in 2019 but is currently running SQL Server 2008 (SP4). It's about 1.2TB of data. Our internal IT unit has come to the point that we want to create a readable secondary for some reports, but mostly ad-hoc reporting. Usage is about 1500 active sessions and about 25,000 Be/S peak.
Now onto the actual question. The option I forsee are transactional replication, mirroring, and log shipping with a read only standby. One of the developers also put Service Broker with CDC ... any landmines or curveballs with CDC and SB?
Service Broker is a very powerful tool to create and manage ques. CDC reads the log asynchronously to pick up changes to designated tables. They don't interact with each other and are designed to have low impact on an active database. They both work very well even in high volume situations. Like many features in SQL Server they can be used with a minimal learning curve but if you want to really take advantage of these tools some study is required.
Related
I would like to know what are best practices for using SQL Server replication on a SQL Server instance that may have other application databases that may also use replication. That is, our product needs to play well with other users of the instance.
The product currently uses SQL Server replication to create a copy database used for reporting. It is always the sole user of the SQL Server instance. But we now need to document and test (regulatory requirements) how the product can share the instance.
I'm making the assumption here that we still need replication as we do not see another way to isolate reporting load from the application's database.
Has anybody done this successfully?
If we are using instance level replication:
Is there a way we can stop/start/modify replication for our application without affecting others?
Do setting differ greatly? That is, is it realistic to share instance level replication settings across applications?
Non-instance replication just looks hard, do I have the wrong view here?
Our customers use SQL Server 2008 R2 or SQL Server 2012.
At an instance level, replication configures only one distributor. That is, regardless of how many databases you have configured for replication on an instance, they'll all share one distributor. You do have the option to make that distributor local (i.e. on the same instance) or remote. So, if you find that distribution is taking up considerable resources (or anticipate that that's going to be the case), configure remote distribution.
Whatever drive holds your databases' log files will need to have sufficient headroom in their throughput to handle the logreader agent. If you're concerned that your database's activity will be impacting to other databases, isolate.
As for other concerns, replication is a lot like your line of business application. That is, it needs to read data (from the publisher and distributor depending on which phase of replication you're talking about) and write data (from the distributor and subscriber again depending on which phase of replication you're talking about). Provision resources accordingly and you should be just fine.
I just realized that my application was needlessly making 50+ database calls per user request due to some hidden coding -- hidden in the sense that between LINQ, persistence frameworks and events it just so turned out that a huge number of calls were being made without me being aware.
Is there a recommended way to analyze individual transactions going to my SQL 2008 database, preferably with some integration to my Visual Studio 2010 environment? I want to be able to 'spy' on individual transactions being made, but only for certain pieces of my code, and without making serious changes to either the code or database.
I addition to SQL Server Profiler, there are a number of performance counters you can look at to see both a real time evaluation and a historic trend:
Batch Requests/sec: Effectively measures the number of actual calls made to the SQL Server
Transactions/sec: Number of transactions in each database.
Connection resets/sec: number of new connections started from the connection pool by your site.
There are many more performance counters you can monitor, specially if you want to measure performance, but going through is besides the scope here. A good starting point is Monitoring Resource Usage.
You can use the SQL Profiler tool that comes with SQL Server Management Studio.
Microsoft SQL Server Profiler is a graphical user interface to SQL Trace for monitoring an instance of the Database Engine or Analysis Services. You can capture and save data about each event to a file or table to analyze later. For example, you can monitor a production environment to see which stored procedures are affecting performance by executing too slowly.
As mentioned, SQL Profiler is userful at the SQL Server level. It is not available in SQL Server SSMS Express however.
At the .NET level, LINQ to SQL and the Entity Framework both support logging. See Logging every data change with Entity Framework, http://msdn.microsoft.com/en-us/magazine/gg490349.aspx, http://peterkellner.net/2008/12/04/linq-debug-output-vs2008/.
I have 2 databases that have the same structure, one on a local machine and one on the company's server. Every determined amount of time, the data from the local DB should be synchronized to the server DB.
I have a general idea on how to do this - create a script that somehow "merges" the information that is not on the server DB, then make this script run as a scheduled job for the server. However, my problem lies in the fact that I am not very well experienced with this.
Does SQL Server Management Studio provide an easy way to do this (some kind of wizard) and generates this kind of script? Is this something I'll have to build from scratch?
I've done some basic google searches and came across the term 'Replication' but I don't fully understand it. I would rather hear some input from people who have actually done this or who are good with explaining this kind of stuff.
Thanks.
Replication sounds like a good option for this, but there would be some overhead (not technical overhead, but the knowledge need to support it).
Another SQL Server option is SSIS. SSIS provides graphical tools to design what you're trying to do. The SSIS package can also run SQL statements, if appropriate. An SSIS package can be started, and therefore scheduled, from a SQL Server job.
You should consider the complexity of the synchronization rules when choosing your solution. For example, would it be difficult to resolve conflicts, such as a duplicate key, when merging the data. A SQL script may be easy to create if the rules are simple. But, complex conflict rules may be more difficult to implement in a script (or, replication).
SQL Server Management Studio unfortunately doesn't offer much in this way.
You should have a serious look at some of the excellent commercial offerings out there:
Red Gate Software's SQL Compare and SQL Data Compare - excellent tools, highly recommended! You can even compare a live database against a backup from another database and synchronize the data - pretty nifty!
ApexSQL's SQL Diff and SQL Data Diff
They all cost money - but if you're serious about it, and you use them in your daily routine, they're paid for in no time at all - well worth every dime.
The only "free" option you have in SQL Server 2008 would be to create a link between the two servers and then use something like the MERGE statement (new in SQL Server 2008) to transfer the data. That doesn't work for structural changes, and it's limited only to having a live connection between the two servers.
You should definitely read up on transactional replication. It sounds like a good fit for the situation you've described. Here are a few links to get you started.
How Transactional Replication
Works
How do I... Configure
transactional replication between two
SQL Server 2005 systems?
Performance Tuning SQL Server
Transactional Replication
What you want is Peer-to-Peer Transactional Replication, which allows data to be updated at both databases yet keep them in sync through a contiguous merge of changes. This is the closes match to what you want, but is a fairly costly option (requires Enterprise Edition on both sites). Another option is Bidirectional Transactional Replication, but since this requires also two EE licenses, I say that peer-to-peer is easier to deploy for the same money.
A more budget friendly option is Updatable Subscriptions for Transactional Replication, but updatable subscriptions are being deprecated and you'd bet your money on a loosing horse.
Another option is to use Merge Replication. And finally, for the cases when the 'local' database is quite mobile there is Sync Framework.
Note that all these options require some configuration and cooperation from the Company's server DB.
There are some excellent third party tools out there. For me, xSQL Data Compare has always done the trick. And because the comparisons are highly modifiable it is suitable for almost every data compare or data-synchronization scenario. Hope this helps!
We have several systems with Oracle (A) and SQL Server (B) databases on backend. I have to consolidate data from those systems into the new SQL Server database.
Something like that:
(A) =>|---------------|
| some software | => SQL Server
(B) =>|---------------|
where some software is:
transport (A and B systems located in the network)
processing business logic (custom .NET code)
Due to first point, I need some queue software or something similar (like MSMQ, Service Broker or something). In another hand, I can implement a web-service instead of queue.
(A) =>|---------------|-------------|
| queue/service | custom code | => SQL Server
(B) =>|---------------|-------------|
The question is: which queue/transport framework should I use with Oracle and SQL Server databases?
It would be nice, if I can post messages to MSMQ in both Oracle and SQL Server stored procedures (can I?)
It would be nice, if I can call a web-service in both Oracle and SQL Server stored procedures (can I?)
It would be nice, if I can use something similar in both Oracle and SQL Server stored procedures (what exactly?)
What software should I prefer to my requirements?
UPD: some techspec
This would be a regular sync process. Once a day I think.
Latency is not critical (>0.5-1 hour is ok).
Amount of data: 1-50 MB per sync from each system.
Encryption is required while transfer.
I would suggest creating an SSIS package that transfer the new data from the server A,B to the new server when invoked. You would launch the SSIS package on a schedule, say every 30 min, from the new server.
If both A and B would be SQL Server then Service Broker would make sense in order to provide a very low latency. But with one of them being Oracle, and with no real-time requirements, it looses its appeal. As a side note, you can see here an example of using Service Broker for High volume real time contiguous ETL.
Doing the transfer as an SSIS package makes for ease of maintenance (you can modify the package with relative ease), it does not require invasive changes in the existing system, is quite performant and there is a large tonne of SSIS know-how available online.
I would advice against using MSMQ for several reasons:
when transactional reliability is needed you'll have to involve all MSMQ related operations into distributed transaction (DTC between the MSSMQ dequeue and the SQL server insert/update on the new server) which will slow doen the processing throughput significantly
You'll need to come up with quite a few lines of code for the marshaling/unmarshaling and shredding the deltagram messages into the target system (I know codding is fun, but SSIS is simply better at this kind of jobs, and easier to maintain)
MSMQ limitation of 2GB per queue is quite small in real world (fills up quickly if your traffic increases and you have a maintenance downtime)
The real problem I'd be worried about is how to detect changes on A and B: when the SSIS job comes every 30 minutes, how does it know what data is new? Specially, how it detects deletes...
Given a .NET environment with Windows CE, can you persist thousands of records per second in a local database (SQL Server 2008 - standard or CE).
What are the performance issues with persisting realtime instrument data in a database versus a log file?
SQL Server 2008 standard is more than capable of those insertion rates PROVIDED you have hardware capable of supporting it.
The question you really need to be asking is do I require the ability to search the captured data quickly?
This SO answer might be of interest: What does database query and insert speed depend on?
The number (and width) of indexes on a table will obviously have an impact on insertion rate.
If you are considering open-source, then MySQL is often cited as being able to handle high volumes.