Anyone knows how to perform inline loading into SQL server, I have DsGen software provided by TCP council, it generates data files have the extension .dat. Is there any mechanism to load these files directly to SQL database (during the generation period). I have done that using import/export wizard, but that is not inline loading.
There are several ways to accomplish your task, here are two of them.
You may use the BULK INSERT command to import them. Basically, what the import wizzard does is about the same, but it let's you select the various options using a nifty GUI.
You can save the DTS package when running through the wizzard, create a SQL Server Agent job and execute this job using the sp_start_job stored procedure.
I like the BULK INSERT as it it easier to implement. Just play arround with the options until you get what you want.
Related
I need to run some analysis on my queries (specifically finding all the tables which a ssis calls).
Right now I'm opening up every single ssis package, every single step in it and copy and pasting manually the tables from it.
As you can imagine it's very time consuming and mind-numbing.
Is there a way to do export all the queries automatically ?
btw i'm using sql server 2012
Retrieve Queries is not a simple process, you can work in two ways to achieve it:
Analyzing the .dtsx package XML content using Regular Expression
SSIS packages (.dtsx) are XML files, you can read these file as text file and use Regular Expressions to retrieve tables (as example you may search all sentences that starts with SELECT, UPDATE, DELETE, DROP, ... keywords)
There are some questions asking to retrieve some information from .dtsx files that you can refer to to get some ideas:
Reverse engineering SSIS package using C#
Automate Version number Retrieval from .Dtsx files
Using SQL Profiler
You can create and run an SQL Profiler trace on the SQL Server instance and filter on all T-SQL commands executed while executing the ssis package. Some examples can be found in the following posts:
How to capture queries, tables and fields using the SQL Server Profiler
How to monitor just t-sql commands in SQL Profiler?
SSIS OLE DB Source Editor Data Access Mode: “SQL command” vs “Table or view”
Is there a way in SQL profiler to filter by INSERT statements?
Filter Events in a Trace (SQL Server Profiler)
Also you can use Extended Events (has more options than profiler) to monitor the server and collect SQL commands:
Getting Started with Extended Events in SQL Server 2012
Capturing queries run by user on SQL Server using extended events
You could create a schema for this specific project and then have all the SQL stored within views on that schema... Will help keep things tidy and help with issues like this.
Planning to import 1000s of text file to sql server tables. All these files are having different structures and it goes to corresponding new tables in
SQl. Different methods coming in mind is using of Biml /creating ssis packages with number of data flows/using import wizard.
What is the ssis design pattern to achieve this in most time efficient way. This is a onetime load though.
How to handle the failure ? : I am not considering the checkpoint because ,when Control Flow tasks are run in parallel checkpoints act a little erratically.
This might give you a starting point if you want to see what each file has for data, without opening every one of them. The DefaultDir will reside on your SQL Server that you are running from.
SELECT * FROM
OPENROWSET ('MSDASQL', 'Driver={Microsoft Access Text Driver (*.txt, *.csv)};DefaultDir=C:\PathtoFiles',
'select * from FileName.csv');
I have a stored procedure that I am going to run every weekend, it produces a result set that I need to export into an Excel file.
For the above problem I want to automate this process, so I am going to create a SQL Job and I am going to run this stored procedure every weekend so that that generated Excel file is sent to my reporter.
For this I need steps to export the result set data to an Excel file.
And also is it possible to send that Excel file to the specific mail while running the job itself?
So, you might try your luck on https://dba.stackexchange.com/, but in my experience a SQL Agent job running a stored procedure could be coaxed to return CSV or XML - and those could end up in Excel, but there are missing links. I think the missing links would involve programming and potentially 3rd party tools to avoid using Excel's COM API.
I'd strongly recommend your pursuing SQL Server Reporting Services. It is included free with your edition of SQL and includes the ability to
run reports on a schedule (subscriptions),
format the results as an Excel file
distribute the results via email
You'd take your query and use it as the data source for a "report" and use the report wizard to create a very simple table with the results.
Avoid page headers (or footers) that span columns - this will keep the excel output cleaner.
References
Stack Overflow: reporting-services-export-to-excel-with-multiple-worksheets
Technet: Reporting Services
What is the best method to import data from an Excel worksheet? As of now I am use SSMS Express so I don't have access to SQL Import Wizard. I also don't have permissions to execute the BULK INSERT command.
My current workflow is as follow: Clean up the excel file, save as CSV, and import it into a SQLite database. Use an IDE like RazorSQL to generate SQL INSERT statements.
This worked nicely until I hit an Excel file about 75000 rows. SSMS just gives an error saying "query finished with errors" or something like that. No error message is shown. I tried adding GO at the end of each line but I got out of memory error.
What are my options?
To answer your question, the best method to import data from excel, in my past experience has been to read excel data into c#, do any clean up and formatting as necessary since excel likes to mess with the data, then use SqlBulkCopy (you only need select/insert permissions) to insert into SQL Server. See this SO answer if you need help reading excel from C#
Update: Given you're not a dev, try using the bcp utility (you should only need select/insert permission)you may need to save the excel file as CSV first, then import it directly into sql server, see this SO answer
You can use following:
bcp utility (between file system data dump and database),
OPENQUERY (can be used from SSMS, works between external datasource like Excel/csv and database),
BULK INSERT (can be used from SSMS, works between external file with user-defined structure and database),
SSIS (usually as dtsx package, has its own GUI, works with various souces and destinations)
Set of INSERT statements (all of them one after another, eventually sliced with GO or packed with UNION ALL)
Set of records serialized in XML variable (can be used from SSMS only; you have to serialize/deserialize it by your self using FOR XML and XML functions)
There are surely other possibilities, but these are maybe most used ones.
EDIT: It seems to me that you could try with GO after every 5-10K lines in your script.
If that doesn't work, XML serialization/deserialization could be the way to go.
Could you use a linked server to connect to the Excel Document? How to use Excel with SQL Server linked servers and distributed queries
A quick and dirty workaround: pull the rows in batches of 50k.
select * from employee limit 50000
select * from employee limit 50000, 100000
From
http://www.razorsql.com/articles/mysql_limit_query.html
Currently I'm developing the database in sql server. I have nearly 100 tables. I need to dump some sample data to the tables, so that I can test the stored procedures. Any tools is there to do this. Help me please.
The standard tools for bulk importing of data are the bcp, the (almost) equivalent BULK INSERT command and the Import/Export wizard which can be accessed by a database's context menu in Management Studio. The wizard creates an SSSIS package behind the scenes that can be customized to modify data or iterate over a set of source files.
You can use GenerateData.com to generate SQL insert statements.