SSIS package Query Optimization - sql-server

I would like to have idea on how to fix a minor bug identified in the program that transfers data between an Oracle Database and a SQL server Database for a web ordering systems that I support.
The issue is, when two orders are place for instance 129 and 130 on the same day, if the subsequent order (130) gets verified first, the previous web order (129) does not get moved over to the Oracle DB. This happens because that program checks for the maximum web order transferred to Oracle DB and tries to move only SQL web order numbers higher than that.
The queries built that support this concept in the SSIS package are the following:
On the Oracle Side
Select nvl(max(web_order_id),0) maxOrderIDParam from web_shipping
On the SQL Server side
SELECT cast(web_order_id as float) web_order_id, web_entry_date, site_num, protocol_num, inv_num, cast(pharm_num as float) pharm_num, status, comments, username, porstatus
FROM Web_Shipping
WHERE web_order_id > ?
AND status = 'V'
ORDER BY web_order_id
On the Oracle side
insert into web_shipping (web_order_id, web_entry_date, site_num, protocol_num, inv_num, pharm_num, status, comments, username, porStatus)
values (:web_order_id, :web_entry_date, :site_num, :protocol_num, :inv_num, :pharm_num, :status, :comments, :username, :porStatus)
On the SQL Server side
select cast(web_order_id as float) web_order_id, line_id, cast(no_of_participants as float) no_of_participants, cast(amt_inventory as float) amt_inventory, cast(NSC_num as float) NSC_num, cast(dose_str as float) dose_str, dose_unit, dose_form, dose_mult, cast(amt_req as float) amt_req
FROM web_ship_detail
WHERE web_order_id = ?
and finally on the Oracle side
insert into web_ship_detail (web_order_id, line_id, no_of_participants, amt_inventory, NSC_num, dose_str, dose_unit, dose_form, dose_mult, amt_req)
values (:web_order_id, :line_id, :no_of_participants, :amt_inventory, :NSC_num, :dose_str, :dose_unit, :dose_form, :dose_mult, :amt_req)
The effort has been to devise a resolution with minimum code change in the whole SSIS package.

I know you are looking for minimal code change not sure if these qualify but will 100% fix the problem. There are 3 options:
Modify the MS SQL table and include a "IsTransferred" bit column. When the verified record is moved to oracle, update the column to a 1/true
Keep track separate table of orders that have been transferred to Oracle. When selecting MS SQL orders exclude those that exist in the new "transferred" table.
Create a Data Flow object, with the Oracle & MS SQL orders tables as sources, use a Merge Join, using left outer. Use the results where Oracle columns are null (there is no matching Oracle records, didn't transfer) and use those records to transfer over to Oracle.
No Idea how many records are on both sides, so there may be performance concerns for some of the options.

Related

How to migrate Access queries to SQL views?

I am using SQL Server Migration Assistant to create linked tables to link from Access to SQL server. During the migration process I also selected Access queries along with the tables to be migrated. The tables are now migrated and linked. There was no option to link queries automatically. However, out of my 15 queries only 5 of them have migrated. SSMA docs says that:
Most SELECT queries are converted to views. Other queries, such as
UPDATE queries, are not migrated.
SELECT queries that take parameters are not converted, nor are
cross-tab queries.
When converting Access Queries to SQL Views, what would be the best course of action as it is not done automatically? I know that views need to be in T-SQL.
I helped to develop a website that will convert an Access SQL query into T-SQL. Paste in your Access SQL and it will convert and format the result as T-SQL. Test the resulting query in SSMS or an Access pass-through query.
https://accessusergroups.org/sql-converter/
The website applies over 80 conversions to the pasted in query. It can handle a lot of syntax that is specific to Access.
SELECT Instr("Please send feedback & share with friends","e") as Last_E
,DATE() AS Today_Date
,NOW() AS Today_Now
,CDate(DateUpdate) AS DateUpdate_CDate
,CCur([flags]) AS Flags_CCur
,NZ([LV], "LV NULL") AS LV_NZ
,CInt([Type]) AS Type_CInt
,CSng([MsysObjects.DateCreate]) AS DateCreate_CSng
,CDbl([MsysObjects].[DateCreate]) AS DateCreate_CDbl
,TRIM(SPACE(20) & NAME & " ") AS Name_Trim
,LEFT(NAME, 4) AS Name_Left4
,RIGHT(NAME, 4) AS Name_Right4
,MID(NAME, 4,5) AS Name_Mid45
,NOT INSTR(1, NAME, "q") <> 0 AS Name_Contains_q
,IIF(Type = 5, "Query", "Other") AS Type_IIF
,IIF(ISNULL(LvProp), "N/A", LvProp) AS LvProp_Handler
,"Amy's code IS righteous." AS SingleQuotes
,"Al's ""new"" cars" AS DoubleQuotes
,"10 + 10 = "& CStr(10 * 2) AS IgnoreMathSymbols
,DCOUNT("Name","MsysObjects","Name like Q*") as RowCount
FROM MsysObjects
WHERE [NAME] LIKE "*e*"
AND RIGHT(DATE(), 4) = 1010 * 2 + 1
The Access query above will convert to T-SQL as the following. Keep in mind that the MsysObjects table doesn't exist in SQL Server.
SELECT CHARINDEX('e', 'Please send feedback & share with friends' ) as Last_E
,CONVERT(date, GETDATE()) AS Today_Date
,GETDATE() AS Today_Now
,CONVERT(date,DateUpdate) AS DateUpdate_CDate
,CONVERT(money,[flags]) AS Flags_CCur
,ISNULL([LV], 'LV NULL') AS LV_NZ
,CONVERT(int,[Type]) AS Type_CInt
,CONVERT(real,[MsysObjects].[DateCreate]) AS DateCreate_CSng
,CONVERT(real,[MsysObjects].[DateCreate]) AS DateCreate_CDbl
,TRIM(SPACE(20) + NAME + ' ') AS Name_Trim
,LEFT(NAME, 4) AS Name_Left4
,RIGHT(NAME, 4) AS Name_Right4
,SUBSTRING(NAME, 4,5) AS Name_Mid45
,NOT CHARINDEX('q', NAME ,1) <> 0 AS Name_Contains_q
,IIF(Type = 5, 'Query', 'Other') AS Type_IIF
,IIF((LvProp IS NULL) , 'N/A', LvProp) AS LvProp_Handler
,'Amy''s code IS righteous.' AS SingleQuotes
,'Al''s "new" cars' AS DoubleQuotes
,'10 + 10 = '+ CStr(10 * 2) AS IgnoreMathSymbols
,
/* DCOUNT beta conversion */ (
SELECT COUNT(Name)
FROM MsysObjects
WHERE Name LIKE Q%
) as RowCount
FROM MsysObjects
WHERE [NAME] LIKE '%e%'
AND RIGHT(CONVERT(date, GETDATE()) , 4) = 1010 * 2 + 1
As a general rule when migrating to sql server, the select queries on the access client side SHOULD NOT be migrated to sql server. As you noted, ONLY select quires can be migrated, and worse is those views will become read only when linked. (Unless you select a PK during the migration process)
After you migrate the data, and link to those tables, then ALL OF you existing Access queries will continue to work as before.
Those Access saved queries will continue to work as before, and they are now working + using the linked tables. Most access quires do not need to be converted, and they will perform rather well without being converted.
The only exception is those complex queries with say group-by, or complex joins should be converted.
And access client quires in most cases will NOT pull the whole sql server table, so once again, little or no benefit occurs by migrating the access client side queries. I would not migrate them.
So if we have
Select * from tblInvoies where InvoiceNum = 1234
The above does NOT need to be converted to t-sql, nor does it need to be converted to a view. Converting such select quires in access to a sql view will likely introduce bugs and issues into your application.
And as noted, sql server cannot “prompt” you like Access queries can.
You only need or want to covert an access query to a sql view for those queries that are running slow.
I would not migrate ANY access queries to sql server. You are far better off to continue to use the saved Access queries client side. Most of them will not see improved performance by converting to t-sql views.
And you potential introduce a large number of bugs, and issues that will break your application.
Migrate the data tables only. Get the client application working ASAP, and then consider tweaking or migrating some of the access queries that don’t perform well. In a application with say 200 client queries, I find that only 5-10 need to be converted to views.
SQL Server definitely supports Pivots; you'll have to re-crate cross-tabs as Pivots. SQL Server definitely supports parameter queries. Just re-write the SQL. Depending on how complex the queries are, you could perhaps just copy the SQL from Access and paste it into SQL Server. You son't really save queries in SQL Server (although you could export them), but you run queries as Stored Procedures. It takes a bit of getting used to, but once you make the switch away from Access, I'm pretty sure you'll never go back.

Copy data from a SQL Server table into a historic table and add a timestamp of the time copied?

I'm trying to work out a specific way to copy all data from a particular table (let's call it opportunities) and copy it into a new table, with a timestamp of the date copied into the new table, for the sole purpose of generating historic data into a database hosted in Azure Data Warehousing.
What's the best way to do this? So far I've gone and created a duplicate table in the data warehouse, with an additional column called datecopied
The query I've started using is:
SELECT OppName, Oppvalue
INTO Hst_Opportunities
FROM dbo.opportunities
I am not really sure where to go from here!
SELECT INTO is not supported in Azure SQL Data Warehouse at this time. You should familiarise yourself with the CREATE TABLE AS or CTAS syntax, which is the equivalent in Azure DW.
If you want to fix the copy date, simply assign it to a variable prior to the CTAS, something like this:
DECLARE #copyDate DATETIME2 = CURRENT_TIMESTAMP
CREATE TABLE dbo.Hst_Opportunities
WITH
(
CLUSTERED COLUMNSTORE INDEX,
DISTRIBUTION = ROUND_ROBIN
)
AS
SELECT OppName, Oppvalue, #copyDate AS copyDate
FROM dbo.opportunities;
I should also mention that the use case for Azure DW is million and billions of rows with terabytes of data. It doesn't tend to do well at low volume, so consider if you need this product, a traditional SQL Server 2016 install, or Azure SQL Database.
You can write insert into select query like below which will work with SQL Server 2008 +, Azure SQL datawarehouse
INSERT INTO Hst_Opportunities
SELECT OppName, Oppvalue, DATEDIFF(SECOND,{d '1970-01-01'},current_timestamp)
FROM dbo.opportunities

Table hint on linked server query

We have a DB2 database in AS400. Added a linked server, all went well, however occasionally the table is locked, even when we doing some select queries. Thinking about table hint in SQL Server, does the linked server query (e.g. select * from ...) support table hint?
Doubtful, but I don't know for sure.
Are you using openquery() , or 4 part names?
A query using 4 part names like so:
select * from LNKSVRNAME.IBMINAME.MYSCHEMA.MYTABLE where somecolumn = '00335';
Pulls back all the rows from MYTABLE and does the WHERE filtering on MS SQL Server.
In contrast, using openquery() like so:
select * from openquery(LNKSVRNAME, 'select * from MYSCHEMA.MYTABLE where somecolumn = ''00335''');
Sends the query to the IBM i, and only the matching rows from MYTABLE are pulled back into MS SQL Server.
If the table is being locked exclusively, there's not much you can do. However, if you're running into row locks. You may want to look at the following DB2 for IBM i clauses
FOR READ ONLY
SKIP LOCKED DATA or USE CURRENTLY COMMITTED or WAIT FOR OUTCOME
So something like this:
select * from openquery(LNKSVRNAME, 'select * from MYSCHEMA.MYTABLE where somecolumn = ''00335'' FOR READ ONLY USE CURRENTLY COMMITTED');
Note If you are actually talking to an AS/400, FOR READ ONLY is all you'll have available. But if you're talking to a relatively recent IBM POWER System running a relatively recent version of IBM i, then the concurrent-access-resolution clauses I've shown should be available.

How do I avoid date type column of MSSQL INTO PIVOTAL HAWQ null at DBMS migration

We are trying to pull data from external source (mssql) to postgres. But when i checked for invoicedate column entries are getting blank at the same time mssql is showing invoicedate values for those entries.
ie
We tried following query on both the DBMS:
When query executed in SQL Server:
select * from tablename where salesorder='168490'
getting 12 rows where invoicedate column is '2015-10-26 00:00:00.000'
But same query is executed on Postgres
select "InvoceDt" from tablename where salesorder='168490'
Getting 12 rows where the column invoicedate is null.
Question is why?
Postgres InvoiceDt column is coming null rather than we can see that SQL Server is showing appropriate data values.
Why is the data different between SQL Server and Postgres for this particular column?
Vicps, you aren't using Postgres and that is why a_horse_with_no_name is having such a hard time trying to understand your question. You are using Pivotal HDB (formally called HAWQ). HAWQ is now associated with the incubator project, "Apache HAWQ" and the commercial version is "Pivotal HDB".
Pivotal HDB is a fork of Pivotal Greenplum database which is a fork of PostgreSQL 8.2. It has many similarities to Postgres but it is most definitely not Postgres.
You are also using Spring-XD to move the data from SQL Server to HDFS which is critical in understanding what the true problem is.
You provided this example:
CREATE TABLE tablename ( "InvoiceDt" timestamp )
LOCATION ('pxf://hostname/path/to/hdfs/?profile=HdfsTextSimple')
FORMAT 'csv' ( delimiter '^' null 'null' quote '~');
Your file only has one column in it? How is this possible? Above, you mention the salesorder column. Secondly, have you tried looking at the file written by Spring-XD?
hdfs dfs -cat hdfs://hostname:8020/path/to/hdfs | grep 168490
I bet you have an extra delimiter, null character, or an escape character in the data which is causing the problem. You also may want to tag your question with spring-xd too.

Operation must use an updatable query Microsoft Access 2010 running on SQL Server 2012

I have an Access 2010 FE , with linked tables on SQL Server 2012. I have several queries which are passed through used for the generation of reports.
After migration and recreation of the queries. When I run the reports it throws up the above error.
I did check the permissions and also tried unchecking "Use simple file sharing (recommended).
I have full access for the account which I'm using.
Here are some pass through queries example that I have in my DB,
1.UPDATE TABLE SET TABLE.COLUM=TABLE.COLUMN WHERE CONDITION
2.INSERT INTO TABLE (COL1......) SELECT * FROM TABLE
3.DELETE FROM TABLE
INSERT INTO TABLE (COL1......)
SELECT * FROM TABLE LEFT JOIN ON CONDITION LEFT JOIN CONTIDION WHERE CONDITION
Could you please let me know what settings or changes I need make to correct the above issue.
Thanks in advance , I have moderate knowledge on SQL and MS Access any help is greatly appreciated.
#GordThompson I took your option 1 , started decoding all the queries that are being called in VBA code , I had all my tables linked from SQL Server with return records set to false.
I was able to correct the error , it was a violation of data type where a column with int and float where updating a column with nvarchar , this happened due to not defining them explicitly in the SQL statement.
Thanks for the tips.

Resources