I need to generate an XML file from a SQL Server database daily and save it on the local machine. What is the way to do it other than using FOR XML clause?
Apart from being unable to output the result to a file, generating the file using this method only works correctly for smaller datasources. I've tried running the script for a datasource that contains 150k+ rows and the generated file just gets cut at some point and SSMS warns me about the file being too big.
I've tried setting the Maximum Characters Retrieved value to unlimited, but it doesn't really make any difference.
What is the best way to solve this problem?
Related
I was trying to solve a smaller integration through logic apps in Azure.
I have a Stored Procedure that selects data from a database and outputs XML as result.
The thing is that the Xml result is about 50k rows and pretty large.
I made an on-premises gateway connection to run the Stored Procedure through logic apps. but when I get the result, not only does it split the big xml, but it also cuts the whole result after about 15k rows.
I know I could use blobs, which means I need to export the sql-xml to files first, which also means that I need to use BCP with something like powershell to export the xml to file in best way. But, I'm trying to scipt most of the on premises steps. I want this solution to be a cloud-based one as much as possible.
Anyone have a solution for this?
Ok, So...
I have boiled it down to two possible outcomes to why this problem occurs.
first one is that I noised I got this error when trying to open the xml im sql server.
'~vs8D51.xml' is too large to open with XML editor. The maximum file size is '10' MB. Please update the registry key 'HKCU\Software\Microsoft\SQL Server Management Studio\13.0_Config\XmlEditor\MaxFileSizeSupportedByLanguageService' to change the maximum size.
Which makes me think that the stored procedure in Azure Logic apps doesn't fetch a result larger than 10mb because of the restrain in sql server.
I have tried to change it in regedit, but every time I restart sql server manager it resets to 10mb.
I have no idea if this is a correct assessment of the problem but its a thought...
Second, a colleague told me he had a similar problem with an file from an FTP.
He said that in some weird way the logic app doesn't get all the data because of some kind of timeout that happens in the background...
He had to fetch the file content in split pieces and somehow stream it through the workflow of the logic app and then recreate the whole thing and save it to a file on the other end of the integration.
That made me think of trying out this: SQL Pagination for bulk data transfer with Logic Apps
It works, but not quite how i want it to work. I can stream the data and save them to blob, but that is as results from the table itself, not as split pieces of the whole XML of that same data...
Anyone know of a way to maybe iterate/paginate though a whole XML result in SQL in a good way, with root tags and all?
In SSMS 18 in order to have the MaxFileSizeSupportedByLanguageService value persist I needed to edit that key's value in the C:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\CommonExtensions\Platform\Shell\Microsoft.XmlEditor.pkgdef file.
I'm building a huge inventory and sales management program on top of Dynamics CRM 2011. I've got a lot done but I'm kinda stuck on one part:
Images are stored in the database encoded as base64 with a MimeType column. I'm wondering how I might extract those images programmatically on a schedule to be sent as part of a data transfer to synchronize another DB.
I have a SQL Server Agent job that exports a view I created. I'm thinking about writing a program that will take that resultant CSV and use it to get a list of products we need to pull images for, and then it queries the DB and saves the files as say productserial-picnum.ext
Is that the best way to do that? Is there an easier way to pull the images out of the DB and into files?
I'm hoping it will be able to only export images that have changed since say a Last Modified column or something.
I don't know C# at all, VB, PHP and JavaScript enough to do some damage though..
you should be able to achieve this in TSQL itself
OPEN cursor with qualifying records (where now>lastmodified etc)
For Each Record
Select Binary Data into "#BinaryData
Convert "#BinaryData to #VarcharData (Something like below will work)
SET #VarcharData = CAST(N'' AS XML).value('xs:base64Binary(xs:hexBinary(sql:variable("#BinaryData")))', 'VARCHAR(MAX)')
Write #VarcharData to file (on server or a network drive if the agent is configured to write out)
Close File
Next Record
I have a complicated query that marshals data to a temporary table, which I then marshal into a further output temporary table before finally selecting on it to display to screen. This gets saved out from Grid view, to text and I get the file I need for processing off site.
What I want to do is have this query be run-able and create that file on the local disk without any need for the operator to change the "Results to" option, or fiddle with anything.
What command or functionality might be available to me to do this?
I can not install any stored procedures or similar to the server involved.
Since you can't do anything on the server I would suggest writing an SSIS package. Create a data flow, and in your source object put your script. Your destination object will then point to the file you want. You have a fair number of options for output.
The SSIS package can then be run by
A SQL Job (assuming you are allowed even that)
A non SQL job running a bat file with a DTEXEC command
The DTEXECUI GUI.
Also you can store your SSIS package in the instance or on any fileshare you choose.
I have a process that backs up my remove SQL Server database to local SDF (SQL CE) database files daily.
What I need is an easy way to search across multiple SDF files. For example, suppose I want to find all occurrences of a name within all of the backups. Instead of opening each individual SDF file -- one by one -- with Enterprise Manager, I'd like to be able to do a search across all the files and show the results in one centralized place. Maybe like a plug-in for Windows Search, etc.
If you're familiar with Notepad++, think about how it's File Search feature works.
Is there any way to accomplish this, or am I just dreaming?
There is no direct way to accomplish this, unless yo write your own search filter for SQL Server Compact. But you could use a tool like my SQL Server Command command line utility http://sqlcecmd.codeplex.com together with some clever batch/Powershell code to query across multiple sdf file and collect the results in a singel file.
You could write a script or program that opens/connects to a set of .SDFs (all files in a folder, from a hardcoded list, from a table in a specialized/master ..SDF), execute a (list of) statement(s) like SELECT ... FROM table WHERE field = 'needle', and display the resultsets to the console, a .HTML file, or push them to still another .SDF.
I have a very annoying task. I have to load >100 CSV-files from a folder to SQL Server database. The files have column names in first row. Data type can be varchar for all columns. The table names in database can just be filename of the CSVs. What I am currently doing is that I use Import/Export Wizard from SSMS, I choose flatfile from dropdown box, choose the file, next->next->next and finish! Any ideas how can I automate such a task in Integration services or with any other practical method?
Note: Files are on my local PC, DB-server is somewhere else, so I cannot use BULK INSERT.
You can use SSIS - Foeach loop container to extract file names - by arranging to particular format.Use a variable to dynamically fill the variable with file name.Then in dataflowtask , use flat file source for source - oledb destination.
Please post some sample file names.so that i can learn and guide you properly.
Thanks
Achudharam