For my .NET Application I have an Excel file that is used as the data source. This Excel file is updated every hour. My current back-end database technology is MS Access. The Access file has a linked table to the mentioned Excel file. I would like to switch over my back-end to SQL Server.
My question is how can I create a linked table in SQL Server such that I always have the current data in my SQL Server database. Right now I have managed to import the Excel sheet through the import wizard in SQL Server Mgmt. Studio. But obviously this is static (not updated hourly like the excel sheet).
Can I create some sort of procedure that runs every hour, reads the Excel sheet and import to my SQL Server database? How would I go about doing that? Or any other method as long as my table in SQL Server gets updated on an hourly basis with data from the Excel sheet.
One, do not code read "in process" from the excel file. That is a disaster waiting to happen. Some user opens the file, locks it, and now your sql-server starts failing. :) That is a bad scenario.
You should create an ETL package to run (hourly)?
https://learn.microsoft.com/en-us/sql/integration-services/ssis-how-to-create-an-etl-package?view=sql-server-ver15
What is SQL Server Integration Services (SSIS)? MicrosoftSQL Server
Integration Services (SSIS) is a platform for building
high-performance data integration solutions, including extraction,
transformation, and load (ETL) packages for data warehousing. SSIS
includes graphical tools and wizards for building and debugging
packages; tasks for performing workflow functions such as FTP
operations, executing SQL statements, and sending e-mail messages;
data sources and destinations for extracting and loading data;
transformations for cleaning, aggregating, merging, and copying data;
a management database, SSISDB, for administering package execution and
storage; and application programming interfaces (APIs) for programming
the Integration Services object model.
data sources and destinations for extracting and loading data;
Deploy and Schedule:
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
Scheduling the SSIS Package with SQL Server Agent
Manually executing packages is one thing, but normally you will schedule packages so your ETL can run in a specific time windows
(probably at night). The easiest option is SQL Server Agent. You can
right-click on the Jobs node to create a new job:
Related
We have an on-prem SQL Server DB (SQL Server 2017 Comp 140) that is about 1.2 TB. We need to do a repeatable migration of just the data to an on cloud SQL (Paas). The on-prem has procedures and functions that do cross DB queries which eliminates the Data Migration Assistant. Many of the tables that we need to migrate are system versioned tables (just to make this more fun). Ideally we would like to move the data into a different schema of a different DB so we can avoid the use of External tables (worried about performance).
Moving the data is just the first step as we also need to do an ETL job on the data to massage it into the new table structure.
We are looking at using ADF but it has trouble with versioned tables unless we turn them off first.
What are other options that we can look and try to be able to do this quickly and repeatedly? Do we need to change to IaaS or use a third party tool? Did we miss options in ADF to handle this?
If I summarize your requirements, you are not just migrating a database to cloud but a complete architecture of your SQL Server, which includes:
1.2 TB of data,
Continuous data migration afterwards,
Procedures and functions for cross DB queries,
Versioned tables
Point 1, 3, and 4 can be done easily by creating and exporting .bacpac file using SQL Server Management Studio (SSMS) from on premises to Azure Blob storage and then importing that file in Azure SQL Database. The .bacpac file that we create in SSMS allows us to include all version tables which we can import at destination database.
Follow this third-party tutorial by sqlshack to migrate data to Azure SQL Database.
The stored procedures can also be moved using SQL Scripts. Follow the below steps:
Go the server in Management Studio
Select the database, right click on it Go to Task.
Select Generate Scripts option under Task
Once its started select the desired stored procedures you want to copy
and create a file of them and then run script from that file to the Azure SQL DB which you can login in SSMS.
The repeatable migration of data is challenging part. You can try it with Change Data Capture (CDC) but I'm not sure that is what exactly your requirement. You can enable the CDC on database level using below command:
Use <databasename>;
EXEC sys.sp_cdc_enable_db;
Refer to know more - https://www.qlik.com/us/change-data-capture/cdc-change-data-capture#:~:text=Change%20data%20capture%20(CDC)%20refers,a%20downstream%20process%20or%20system.
I want to transfer multiple tables and their data from one SQL Server to another SQL Server on the local network automatically every 1 hour.
There is a tool built-in tool in SSMS to do this.
In SSMS, right click on the database name. Select Data > Import on the destination database. You will be prompted to provide connection information for the source database. This is internally using SSIS integration tool.
Create a SSIS Package USE Type two SCD if you want insert and
updates. you can use staging table from source to
destination is a good practice and it is industry standard. if you
are not having staging environment. You can use temp tables within
ssis package to achieve the same.
Schedule a job and run that ssis package in the job for every half an hour
I'm looking for advice for the following 'issue'.
I need to send Excel files (not CSV) to our customers with SQL SERVER.
After intense searching, I've found out that the .XLSX extension out of SQL server isn't possible (not directly), and I should be using SSIS for this.
I've never used SSIS before, did some research, and found out that I need to create .dtsx packages and schedule them to run in SQL server.
I have up to 80 reports to be sent on a daily basis (each with individual queries, connecting to the same database); this database is a Linked server on the SQL server (Old school firebird DB).
Some of the queries result in multiple mails (e.g. "Send Mail for each record" - the record contains the receiver and it's values for the body of the mail). Others are just simple Xxcel tables.
Just to be sure that I'm on the right track here, these would be the steps:
Create one project in VS 2017 (Integration Services Project) for all the reporting
Create new Package (for each report)
Create ODBC connection to the SQL database (Data Flow => ODBC Source)
And this is where I'm stuck.
So some advice here would be great.
4) Send mail task Specify the attachment as the file you just created.
Personally, I prefer sp_send_dbmail as my method for sending email but either will work and both support attachments:
sp_send_dbmail (Transact-SQL)
A third option is to use SSRS to build the reports as a table/matrix and then define a subscription for the users
What's the correct way of exporting data from Excel 2013 file to SQL Server database? The data from the Excel file should be transferred into SQL when saving excel file to a database.
I know many answers for this are available but my question is bit different: every time the excel data changes, or the user clicks on save button the data in the database also needs to be updated.
The easiest way to do this is with an SSIS package. SSIS (SQL Server Integration Services) is a package built in to SQL which allows transformations between data formats.
You can create a package by right-clicking on your target database in SQL Server Management Studio and selecting Tasks > Import Data. In the wizard that comes up asking for a data source choose "Microsoft Excel" from the top drop-down labelled as Data Source, then follow the wizard through. You'll have the choice of importing the Excel data into a new table or mapping it into an existing table.
If you want do this programmatically, you can save your package at the end of the wizard and then invoke it via code. But that's a different question.
What you want is not possible (as long as I know off). You can use SSIS package to migrate the Excel sheet into SQL Server, but is imposible to determine if someone "click save" or do some changes on the excel file. SSIS package can be programed to run on schedule or by demand. You should investigate SSIS packages. It is not easy to learn, but do what you need.
Maybe you would like to try a tool I have developed? It's an Excel Add-In that exports Excel data to SQL Server.
There is a feature to automatically export the data to SQL Server every time you press the save button in Excel. If you need to update the database every time a cell value change, you will need to add a few lines of VBA-code that will push the data to SQL Server.
To beta testers I currently give away a free license, so if you are interested in testing it out, send me an email:)
www.sqlpreads.com
I have to pull data from a table in Oracle DB and load in to a table in SQL server DB.
I have created a DTS package that imports and load the data by linking two DBs.
Please advice on how to schedule a job to call the DTS package(on monthly basis) which I have created already with the required process flow.
Appreciate your help!
New bee
You really should have googled this... I'm assuming you don't mean DTS (as in SQL 2000 Data Transformation Services), but rather .DTSX files in SSIS.
http://blog.sqlauthority.com/2011/05/23/sql-server-running-ssis-package-in-scheduled-job/