I have a query that generates 1.5M rows that I need to export on a daily basis. I'm trying to automate it and running into some issues.
Using the SSMS Export function works great, but we have SQL Web and apparently that doesn't support saving the SSIS package.
I tried a Powershell script, but that bombs because it's doing something in memory and it just increases to the point that the script fails.
What are my options to do this export as a scheduled task inside of SQL Server?
Related
Using Microsoft products, I have a collection of SQL scripts and a VS project that I use to refresh a DW on a weekly basis. I know the process can be automated, but the documentation for VS is so vast I don't know where to start. The HIGH LEVEL process is outlined below:
Open SSMS and MANUALLY run scripts to truncate tables and drop indexes
Open VS project and in dev mode MANUALLY press START to extract the data from application to the truncated tables in the DW
Open SSMS and MANUALLY run transformation scripts to create analysis cubes end users can access
I am trying to get to a point where I can just schedule this process to run every X period so I don't have to press any buttons.
From what you described it seems like SSIS is something that can cover the tasks you listed. SSDT will need to be installed to use this. Since you're using scripts I'm assuming you have SQL script files saved that you execute. These can definitely be run in SSIS using an Execute SQL Task, with a file connection as the SQL source. With a collection of scripts, I'd suggest looking into using a Foreach Loop to run these via an Execute SQL Task while iterating through the folder(s) that contain the scripts. As far as running the Visual Studio project in development mode, configurations in Visual Studio can used to accomplish this for SSIS. SSIS has both an Analysis Services Processing Task and Analysis Services Execute DDL Task, and sounds like you're looking for the latter. Both XMLA and TMSL commands can be executed from an SSAS Execute DDL Task. Below are some links to get you started. The Data Flow Task may help with what you're doing for your data extraction.
SSIS
SSDT
Execute SQL Task
Analysis Services Processing Task
Analysis Services Execute DDL Task
Configurations
Data Flow Task
Foreach Loop
DTEXEC
I think, you can use jobs. Descriptions is here
I have searched and have not found an answer for my following question:
I am running an SSIS package created from the Export data task for a database, in SSMS jobs. I want to add month and year to the file name when it is created but don't have access to Visual Studio. How would I do this in the Job Step Properties? Or would I need to do it on the SSIS package creation?
I could create a .bat file that could update the file name after the completion of the job but I would rather have it all done in one go.
Thanks.
SQL 2008 R2
--NO Visual Studio--
There is no way to do this using the Export Data Wizard. You would need to edit the SSIS package to do this.
Technically you could do this without Visual Studio, since an SSIS package is just a text file, but you would really have to know what you are doing. I wouldn't want to try it.
You're allowed to run Powershell scripts as job steps. Roughly the same as your .BAT file idea, but you could at least have it run in-line, manage it within the Job Agent, etc.
On one of my server, I have a Pervasive V8 installed. I need to get the data from some tables.
It's possible manually, just Click Right on table, then Export Data.
But, there is a way to automate these Exports ? Sorry, but documentation about Pervasive is very poor.
Ther is no to automate the export through the Pervasive tools. If you want to just export the tables to CSV (what Export Data does), you can use any number of ODBC enabled tools to export to CSV. Most of them can be automated through task scheduler.
I'm trying to desgin a program, that 100 people need to input infornamtion of 300 projects in Relational database(SQL SERVER), and then run SSIS pack to ETL and return calculation results from SSAS. Is it possible that I can pass varible of projet name individually to SSIS pack, and run SSIS pack on demand(not sql server agent scheduled way), so that I can get the SSAS results immediately for certain project? Is there any reference I can learn?
Thanks.
What SQL Server version do you have? If 2012 and newer you can use SSIS Project Deployment Model and Parameters and run packages using stored procedures = on demand
Here you have information about run packages using stored procedures
Here you have information about ssis parameters
I am trying to find a better way to test our SSIS application, which is used to load data from a file into SQL Server and validate the file data.
I have created a SQL script which can be run to insert 'bad' data into the database tables, and ensure that our validations are performing correctly.
The SQL script:
- loads 'bad' data
- executes the SSIS validations
- ensures the errors in the data were detected
- Outputs a PASS or FAIL
- Deletes the TEST data if passed
Is there anyway I can get this script to be run automatically somehow, for example after someone checks in some code? Should I add it as a stored proc?
I had a look at the Default template Build definition but I couldn't see how to run this SQL script.
The wrong way to do this is using a build server. There are lots of parts to a continuous integration and a build server is really designed for compilation and validation that does not require an instance of your environment. There are two good practices that you can adopt:
Create a test harness that allows you to load a singe package from your SSIS scripts and test the inputs and outputs. Essentially unit tests. I did this for a customer a few years ago and it worked well.
Use a release management tool to push out your packages, the data, and everything else that you need. Release Management for Visual Studio can do this easily.
A rule of thumb that I always go buy is if I do not need an instance of anything but in memory objects then all I need is a build server. However if I need an instance of my app then I want to add release management tools to my continuous integration strategy.
It doesn't appear that you can do it as part of a checkin based on my 2 minutes of searching.
How to modify the default Check-in Action in TFS?
http://msdn.microsoft.com/en-us/library/ms243849(VS.80).aspx
Can I modify the default Check-in Action in TFS 2012? (pretty picture)
http://www.codeproject.com/Tips/562994/Perform-a-custom-action-for-Check-in-event-in-Micr
Instead, you'll need to set up a build server to handle the action. Out of the box, MSBuild doesn't have the ability to run a SQL Script so you'll need to snag the MSBuild Extension Pack and leverage SqlExecute
http://msdn.microsoft.com/en-us/library/ms181712.aspx
http://msdn.microsoft.com/en-us/library/ms181710(v=vs.90).aspx
http://msdn.microsoft.com/en-us/library/cc668755(v=vs.90).aspx
http://www.msbuildextensionpack.com/help/4.0.5.0/html/0d864b98-649a-5454-76ea-bd3069fde8bd.htm
You can use SQL Server Agent to embed for example your SQL code and you can than run this job from your application.