I have two SQL Server 2014 DBs with different schemas. These DBs served two distinct web application operating in the same area of interest, hence I have similar tables in these two DBs. What is the easiest way to migrate data between them? I was thinking about a Transact-SQL script. Is there a tool that could solve this task more easily?
If the migration is relatively simple or if you want to reduce the number of tools involved you can stick with a tsql script. If you want to run it on a schedule you can execute it with SQL agent as TSQL or wrap it in a stored procedure and call that from the agent. If there are different servers involved you can create a linked server.
If you like a visual tool or if the process is very complex and you do not want to write tsql scripts then SSIS is a great tool that is specialized in taking data from disparate sources, applying transformations / conversions and importing. Some people also like to use SSIS for simple tasks because of the visual design surface.
Without more details it is hard to say the best route. If I had two DB's that were very similar I would consider merging the designs to accommodate both business lines / customers and add flexibility to allow more businiess lines / customers into the same design in the future
Related
Trying to figure out the best way of managing data stored in an on-premise SQL Server database, for a Power Automate Model-Driven App. For Canvas Apps, typically I use a connector to either access the tables directly, or to access the data via a Power Automate flow.
From my research, it seems like I have two options:
Create Dataverse tables which represent all of my SQL Server tables, and use an ETL tool to keep those tables in sync.
Use a Virtual Table to connect "directly" to the SQL Server tables
Neither of those options really seem great, though. The entire premise of the Power Platform is for a relatively low-code experience, and both of those two options require quite a bit of coding. The ETL option, for example, requires me to write ETL operations for every table that I want to manage, along with dealing with any number of concurrency issues that may arise. The Virtual Table, on the other hand, seems on the surface like it'd be a better approach, but to actually implement a Virtual Table for an SQL Server table, I need to write a custom connector, and I need to leverage the old D365 UI to manage it.
The "best" approach I've seen so far is explored a bit by Juuka Niiranen in this blog post, however, a year later, and the approach he wrote about is still in preview: https://jukkaniiranen.com/2021/06/virtual-dataverse-tables-with-no-code-via-connectors/ and https://learn.microsoft.com/en-us/power-apps/maker/data-platform/create-virtual-tables-using-connectors?tabs=sql
Does anyone have any other suggestions? Is that above connector approach the recommended one, despite still being in preview?
I have a transactional database (SQL Server 2014) with around 60 tables, and there is a requirement to create a separate reporting database for reporting purposes.
This will only need to run every 24 hours - however I will be needing to move the data into a different, more query-friendly schema!
Because of this I would hope I could just create some Views on the Transactional Db and then create a table based on that view in the Reporting db and copy across the data.
I originally thought of writing a scheduled Windows Service that somehow extracts data from the tables and inserts into the new one, but then thought if the schema changes it has to update in two places, and also thought surely an enterprise SQL Server license must have some tricks.
I then looked into 'database mirroring' on specific tables but this looks to soon be deprecated.
'Log shipping' looks like more of a disaster recovery solution!
Is there an industry 'best' approach to this problem?
You will need to devise an ETL process to extract data from your source database, transform it and load it into your reporting database. There are many tools available to you to make this easier. You can use SSIS, Azure Data Factory for Azure SQL, and there are many other options. You can use the SQL Agent to schedule stored procedures to run your ETL process.
Your target database will look much different than your source database. There is really no quick way (quick as in scheduling a backup) to accomplish this. There is a lot of information on data warehouse and ETL design available to you to assist you in deciding how to proceed.
Does anyone have experience building database reports - doesn't matter which database - i just want design ideas - for a system that is made up of many separate, but identical databases?
I cannot "combine" all databases into one. They must be separate.
But the structure is identical across all databases...
I need to build a web interface that will allow a user to get a "global" report that will query all databases and build one combined report.
Do you have any comments on how the model would look like? or anything you think i need to beware of?
Thanks.
I don't have first hand experience with cross database reports, my experience comes from a product the company i work for sells which can create reports from multiple databases, from your description i believe you require something of the "combine" tables kind, in this case i recommend you to detect the tables used in the query, and unify them in a single temporary intermediary database, for example Access, SQL Server CE or SQLite and then run the query against this temporary database or table.
If your databases are Microsoft SQL Server, then using SQL Server Reporting Services seems like a good solution. The software for the report generation / display is bundled along with the database software.
It gives you a web interface, where you can configure 'data sources' from any number of remote databases, and combine data from these sources into reports. It is user friendly and you can do all the report design / configuration through the web interface without having to write any code.
some references :
Building report using SQL Server stored procedure
http://blog.hoegaerden.be/2009/11/10/reporting-on-data-from-stored-procedures-part-1/
We have several SQL Server databases containing measurements from generators that we build. However, this useful data is only accessible to a few engineers since most are unfamiliar with SQL (including me). Are there any tools would allow an engineer to extract chosen subsets of the data in order to analyze it in Excel or another environment? The ideal tool would
protect the database from any accidental changes,
require no SQL knowledge to extract data,
be very easy to use, for example with a GUI to select fields and the chosen time range,
allow export of the data values into a file that could be read by Excel,
require no participation/input from the database manager for the extraction task to run, and
be easy for a newbie database manager to set up.
Thanks for any recommendations or suggestions.
First off, I would never let users run their own queries on a production machine. They could run table scans or some other performance killer all day.
We have a similar situation, and we generally create custom stored procedures for the users to "call", and only allow access to a backup server running "almost live" data.
Our users are familiar with excel, so I create a stored procedure with ample parameters for filtering/customizations and they can easily call it by using something like:
EXEC YourProcedureName '01/01/2010','12/31/2010','Y',null,1234
I document exactly what the parameters do, and they generally are good to go from there.
To set up a excel query you'll need to set up the data sources on the user's PC (control panel - data sources- odbc), which will vary slightly depending on your version of windows.
From in excel, you need to set up the "query", which is just the EXEC command from above. Depending on the version of Excel, it should be something like: menu - data - import external data - new database query. Then chose the data source, connect, skip the table diagram maker and enter the above SQL. Also, don't try to make one procedure do everything, make different ones based on what they do.
Once the data is on the excel sheet, our users pull it to other sheets and manipulate it at will.
Some users are a little advanced and "try" to write their own SQL, but that is a pain. I end up debugging and fixing their incorrect queries. Also, once you do correct the query, they always tinker with it and break it again. using a stored procedure means that they can't change it, and I can put it with our other procedures in the source code repository.
I would recommend you build your own in Excel. Excel can make queries to your SQL Server Database through an ODBC connection. If you do it right, the end user has to do little more than click a "get data" button. Then they have access to all the GUI power of Excel to view the data.
Excel allows to load the output of stored procedures directly into a tab. That IMO is the best way: users need no knowledge of SQL, they just invoke a procedure, and there are no extra moving parts besides Excel and your database.
Depending on your version of SQL server I would be looking at some of the excellent self service BI tools with the later editions such as Report Builder. This is like a stripped down version of visual studio with all the complex bits taken out and just the simple reporting bits left in.
If you setup a shared data source that is logging into the server with quite low access rights then the users can build reports but not edit anything.
I would echo the comments by KM that letting the great unwashed run queries on a production system can lead to some interesting results with either the wrong query being used or massive table scans or cartesian joins etc
I am developing a data driven website and quite a lot of programming logic resides in database stored procedures and database functions. I found myself changing the stored proc/functions quite a lot in order to fix bugs or add new functionality. The data (tables) have remained mostly untouched.
The issue I am having is keeping track of versions of stored proc/functions. Currently I am incrementing version of whole database when I do a set of changes. As data is huge (10 Gb) I get issues having to run development version and release versions of databases in parallel.
I wish to put all the stored procs and functions in one database and keep data in one database, so that I can better manage the changes.
I am sure others would have encountered similar suggest and request suggestions on how to best handle this situation.
I would also recommend using source control keyword expansion in your stored procedures ($Version:$)
That way you can eyeball, grep, search syscomments, etc to see what version you have on your deployed database.
You can version just the schema dumps. In combination with source control keword expansion (as suggested by Rawheiser), you just take a look at what version you have in the database, generate a diff and apply it.
Also, there are several excellent tools to compare databases and their schemas, generate DDL scripts etc.: SQL Workbench, Power Architect, DDLUtils and Redgate SQL Compare, to name a few. SQL Compare is likely to work best with SQL Server, although all the others are FOSS and provide a higher ROI (in terms of time spent learning and what you can do with them) as they are platoform and RDBMS independent.
Finally, I have to say...I understand that the immediate results you get with logic in the DB are tempting, but if you've gone beyond more than a couple of procedures in the database, you're setting your self up for quite a lot of pain, sifting through what easily turns into spaghetti code and locking your application to a single database vendor. You might have your reasons, but I've been there and didn't like it very much. Logic can live very nicely in a different layer.
For source control you have several options:
Use a Visual Studio Database project.
Use SQL Server 2005's built-in support for source control
Use a third part tool such as SQL Compare
IMO Option 1. is preferable.