I'm dealing with a three piece datapath: client application, host integration server, db server. Client application (MS Mashup Engine) is generating queries that pass through MS SQL Server to a legacy IBM iSeries DB backend.
I'm running into issues where the client is generating queries like
select * from x where numericValue = 1.46510+003
I'm checking the execution plan for these on the SQL Server and they result in a full data load with the comparison occurring on SQL Server (which is acting as the Host Integration Server).
By comparison, a human generated query
select * from x where numericValue = 1465.1
results in no scan and performance two orders of magnitude faster.
I have tried playing with the client application to force it to generate something like the human generated query, but I've had no luck.
I'm not sure if I can massage the way the query plan is generated in SQL server by playing with column data types. I.e. exposing a view over the backend DB with explicitly defined data types. Or otherwise forcing query plan generation?
Any thoughts?
I'm not sure if I can massage the way the query plan is generated in SQL server by playing with column data types. I.e. exposing a view over the backend DB with explicitly defined data types. Or otherwise forcing query plan generation?
No, the answer is no using all of the following:
casting types prior to delivering to power query/power bi changes nothing
adapter properties set for DB2OLEDB or IBMDASQL ( f.ex. Decimal As Numeric=True; Derive Parameters=True; etc...) have no effect
creating stored procedures over the data requires parametrization in power query which is fine, but does not integrate with the ui. I.e. - using a parameter field instead of the column filters is not clean.
TBH, this has become less and less of a problem as backend performance improves and frontend caching is available.
Hope it helps,
-Alex T.
Related
We are using a MSSQL 15 database for our application. Access to the database is established using OLEDB.
While running a series of 10,000 queries
SELECT * FROM table WHERE primary_key = "some_id"
we experienced a rather large drop in performance (factor of around 10) when we switched from running the application directly on the database server to running it on another machine that connects to the server via TCP/IP.
We then investigated the packets sent to and from the server using wireshark. We found that the server replied to every SELECT with the metadata (fieldname, datatype, etc.) of every field of the result followed by the single row of data we had selected. We do not need the metadata information as we already know the structure of table we run the query on.
Because the metadata block is much larger than the actual data, most of the data sent through the network are (to us) completely redundant.
Is there any way to tell the server not to include the metadata in the query result?
I'm a SSIS Developer. I do lots of SQL stored procedure lookup concepts in SSIS. But when coming to Azure Data Factory I haven't any idea how to perform a lookup using a SQL stored procedure.
Could anyone please guide me on this?
Thanks in advance !
Jay
Azure Data Factory (ADF) is more of an ELT tool rather than ETL, therefore direct lookups are not supported. Instead, this type of operation, along with other transforms is pushed down into the compute you are actually using. For example, if you are moving data to SQL Server, Azure SQL Database or Azure SQL Data Warehouse, you would ensure all data is on the same server and use a Stored Procedure task to execute the lookups using T-SQL and joins. If you are using Azure Data Lake Analytics (ADLA) you would use the U-SQL Activity to run U-SQL or execute ADLA stored procedures, again doing lookups via joins or custom U-SQL code such as Combiner, Applier, Reducer. In fact you can use any of the ADF compute options like SQL, HDInsight (including Hive, Pig, Map Reduce, Streaming and Spark script), Machiine Learning or custom .net activities.
So you need to think about things differently with ADF. Have a look through this article to gain greater understanding of transforming data in ADF:
Transform data in Azure Data Factory
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-data-transformation-activities
As an aside, I would rarely use Lookups in SSIS as performance in early versions used to be poor. Although this has been improved in later versions, generally if you can do it in SQL you probably should. This pattern harnesses the power of SQL Server, rather than dragging data up into the SSIS pipeline, eg for the purposes of lookups (which are essentially joins) and pushing the data back out again. I reserve Data Flow transformations mainly when non-relational data is involved, eg xml or joining your email server with relational data. This is my personal view anyway : )
I have an access 2003 database that holds all of my business data. This access database gets updated every few hours during the day.
We're currently writing a website that will need to use the data from the access database. This website (for the time being) will have only read only capabilities. Meaning there will only need to be one way transfer of data (Access -> SQL).
I'm imaging there's a way to perform this data migration from access to SQL server programatically. Does anyone have any links to something I can read about?
If this practice sounds odd, and you'd like to suggest another way to do this (or a situation where data can go both ways (Access -> SQL, SQL -> Access), that's perfectly fine.
The company is going to continue using Access 2003 for their business functionality. There's no way around that. But I'd like to build the (readonly) website on top of SQL Server.
The strategy you outlined can be very challenging. You could use INSERT queries to copy new Access rows to SQL Server, as described in another answer.
However, if you have changes to existing Access rows, and you also want those changes propagated to SQL Server, it won't be so simple. And it will be more complicated still if you want deleted Access rows deleted from SQL Server, too.
It seems more reasonable to me to use a different approach. Migrate the data to SQL Server once. Then replace the tables in your Access database with ODBC links to the SQL Server tables. Thereafter, changes to the data from within your Access application will not require a separate synchronization step ... they will already be in SQL Server. And you won't need to write any code to synchronize them.
If your concern is that the connections between the web server and SQL Server be read-only, just set them up that way. You can still independently allow read-write permissions for your Access application.
To do the initial data migration and set the SQL Server automatically, I would use the SQL Server Migration Assistant. The only thing you should definitely change that I can think of would be to turn off the Identity property on any columns that have it - to be explained below (MS Access calls Identity autonumber). Once you have your tables loaded, you can set up a dsnless connection to the database (and tables) you just created.
I haven't used the method just linked, but I believe it allows you to use SQL Server authentication to connect to the db. The benefit of using this method is you can easily change which SQL Server instance and/or database your are connecting to for development and testing.
There might be a better, automated way, but you can create several insert queries doing left joins from the primary key of the Access table to the SQL Server table, and putting a WHERE clause that specifies the SQL Server PrimaryKey must be null. This is why you need to turn off the Identity property in the SQL Server tables, so that you can insert the new data.
Finally, put the name of each query in one function, then run the function periodically.
I have used Microsoft's free SQL Server Migration Assistant (SSMA) to migrate Access to SQL Server. The tool is very simple to use. The only problem I have encountered with the tool was overloaded data types when migrating. What I mean by this is a small string will get converted to a NVARCHAR(MAX) in some instances. Otherwise, the tool is very handy and can be reused after setting up a 'profile'.
I am developing an application in C# VS 2010 past 4 months. I used MS Access 2007 to store my nearly 20 tables successfully.
Today I realized that my data base cannot be handled consistently by MS Access 2007. Hence I decided to go for SQL Server 2008 R2 Express with Upsizing wizard and it worked really great!
However, when I tried to run various options of my already well developed application, It kept throwing error each time when a query is fired to SQL Server.
I understood that Many of the stuffs of SQL supported by MS Access are not supported by MS SQL Server
For example: query with date, for representing date format when we use '#', SQL Server 2008 won' t recognize it.
Also, for Bool value, MS Access stores it as True and False where as SQL Server uses 0.
These all queries worked perfect with Access 07
I am sure that there must be some method so that SQL Server can understand MS access queries.
Or will I have to edit my whole application?? It will be as good as digging a mine for earning gold..
I have changed all data access objects such as reader, adapter, command, connection to SQL data objects using System.Data.SqlClient.
So, It is not the problem.
Please help me asap.
Thank you.
You cannot force SQL Server to run the MS Access queries. These queries will need to be rewritten to use T-SQL instead of the query language that MS Access uses.
I feel your pain, I just had to rewrite a large MS Access application (over 1k queries) that needed to be recreated to be used in SQL Server.
There will be some queries that might be able to be ported over directly but as you noticed queries with date, and even some of the aggregate functions (First(), etc) are not used in SQL Server and those queries will need to be changed.
Here is a link with some info on converting Access to SQL
Converting Access Queries to SQL Server
You are right that, most of the time, you cannot just take the SQL of a query from Access and run it within SQL Server. It may work for very simple queries, but usually you need to tweak them.
There are a few steps I would take:
Extract your queries (which I presume are in your code), and re-create them in your Access database. Make sure they work there as normal Access queries.
(you can for instance simply add some code to your app to print all queries to files so you don't have to mess with parameters, then just copy/paste them in your Access DB).
The point is simply to have working queries within Access.
Use SSMA from Microsoft for helping you to move your queries to SQL Server. It does a good job of translating them into T-SQL.
You may still have to convert some troublesome queries by hand, but it shouldn't be that many and usually the conversion is not difficult.
Once converted to T-SQL, just re-inject these working queries into your code, or keep the complex queries in SQL Server as views (which it usually be faster as SQL Server will have already created its execution plan, rather than your application sending raw SQL that the server needs to analyse).
As you pointed out, there could be some issues if your fields use some features that don't cross-over to SQL Server properly.
Look at your tables in Access and do some cleanup before attempting to convert:
For booleans fields:
Make sure you set their default values to 0 or 1 (they should not be empty).
Required fields must be non-null:
Make sure that any fields that you have set as 'Required' does not contain any NULL values in its data.
Unique indexes cannot ignore Null:
Check that your indexes are not set to be both 'Unique' and 'Ignore null'.
All tables must have clean primary keys:
Make sure all your tables have a unique primary key that doesn't have Null values in their data.
I have a backup of an SQL Server DB in .bak format which I've successfully managed to restore to a local instance of SQL Server Express. I now want to export both the structure and data in a format that MySQL will accept. The tools that I use for MySQL management typically allow me to import/export .sql files, but unfortunately Microsoft didn't see fit to make my life this easy!
I can't believe I'm the first to run into this, but Google hasn't been a great deal of help. Has anybody managed this before?
There will be 2 issues:
1) Datatypes. There isn't always a direct analog between an MS SQL type and a MySQL type. For example, MySQL handles timestamps very differently and has the cut-off for when you need to switch between varchar(n) and varchar(max)/text at a different value of n. There are also some small differences in the numeric types.
2) Query syntax. There are a few differences in the query syntax that, again, don't always have a 1:1 analog replacement. The one that comes to the top of my mind is SELECT TOP N * FROM T in MS SQL becomes SELECT * FROM T LIMIT N in MySQL (MySQL makes paging loads easier).