Im loading 13 cr rows of data from DB2 to SQL server.
currently I have implemented fast load option and set up batch and commit size. thus improved some performance. Please suggest to improve further.
Current version : ssis 2012 and SQL server : 2014
CREATE TABLE tbl(
[ID] [bigint] NULL,
[CTYPE] [bigint] NULL,
[A_ID] [bigint] NULL,
[AC_TYPE] [bigint] NULL,
[ASS_TYPE] [nvarchar](400) NULL,
[t_ID] [bigint] NULL,
[A_T_ID] [bigint] NULL,
[FLAG] [int] NULL,
[REGION] [nvarchar](50) NULL,
[CHANGEDATE] [datetime] NULL,
[FLAG2] [nvarchar](50) NULL
) ON [PRIMARY]
GO
Related
Is there someone who can help me figure out why I cannot query an external table that I created using my SQL Server Mgt Studio. I can see the external table if I expand External Tables but if I Right click and Select Top 1000 Rows I get an error that Invalid object name 'dbo.AuditLogSource'.
I am trying to copy a certain amount of data from an audit log table in DB1.AuditLog into ArchiveDB.AuditLog. I've followed the tutorials on how to use Elastic Queries to archive this simple task but I am now stuck at this point where I should query from the external table created locally in my ArchiveDB. Here's the process I followed maybe I made a mistake somewhere please help me:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '2019MoxvE!';
--DROP MASTER KEY;
CREATE DATABASE SCOPED CREDENTIAL SQL_Credential
WITH IDENTITY = 'myusername',
SECRET = '2019MoxvE!';
--DROP DATABASE SCOPED CREDENTIAL SQL_Credential;
CREATE EXTERNAL DATA SOURCE RemoteReferenceData
WITH
(
TYPE=RDBMS,
LOCATION='ourserver.database.windows.net',
DATABASE_NAME='DB1',
CREDENTIAL= SQL_Credential
);
--DROP EXTERNAL DATA SOURCE RemoteReferenceData;
CREATE EXTERNAL TABLE [dbo].[AuditLogSource]
(
[Id] [int] NOT NULL,
[Userid] [int] NOT NULL,
[ObjectId] [int] NULL,
[CreatedOn] [datetime] NOT NULL,
[ModifiedOn] [datetime] NOT NULL,
[ModifiedBy] [varchar](150) NOT NULL,
[Type] [int] NOT NULL,
[ActionTable] [varchar](50) NOT NULL,
[IsAjaxRequest] [bit] NOT NULL,
[Parameters] [varchar](max) NOT NULL,
[Controller] [varchar](50) NOT NULL,
[Action] [varchar](50) NOT NULL,
[Comments] [varchar](max) NULL,
[BeforeImage] [varchar](max) NULL,
[AfterImage] [varchar](max) NULL,
[Browser] [varchar](max) NULL
)
WITH (DATA_SOURCE = [RemoteReferenceData]);
--DROP EXTERNAL TABLE [dbo].[AuditLogSource];
INSERT INTO [dbo].[AuditLog]
SELECT al.* FROM [dbo].[AuditLogSource] al WHERE al.[CreatedOn] <= '2020/12/31' AND
NOT EXISTS(SELECT 1 FROM [dbo].[AuditLog] al1 WHERE al1.Id=al.Id);
If you see on below screenshot, you can see that there are no errors being highlighted on this query which means that the query window does recognise that the table AuditLogSource does exists but if I run the query it complains that it does not exists. I can also confirm that the user I am logged into the database with is the admin user and own of both DB1 and ArchiveDB What can I do to make this work?
Thanks in advance.
Make sure you're using the correct database also if you create a new SQL Server object, your newly created object does not get updated in the IntelliSense Local Cache and due to this, it shows an Invalid object name: dbo.AuditLogSource.Please follow below reference.
Ex: [DatabaseName].[Schema].[TableName]
Try:
Edit -> IntelliSense -> Refresh Local Cache or Ctrl + shift + R
Reference:
Sql server invalid object name - but tables are listed in SSMS tables list
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql?view=sql-server-ver16&tabs=dedicated
Ok so I will post an answer to this question in case another person comes across the same/similar problem. So I only made 1 mistake in creating the External Table and this is because of the tutorials and other answers I saw on this very platform.
CREATE EXTERNAL TABLE [dbo].[AuditLogSource]
(
[Id] [int] NOT NULL,
[Userid] [int] NOT NULL,
[ObjectId] [int] NULL,
[CreatedOn] [datetime] NOT NULL,
[ModifiedOn] [datetime] NOT NULL,
[ModifiedBy] [varchar](150) NOT NULL,
[Type] [int] NOT NULL,
[ActionTable] [varchar](50) NOT NULL,
[IsAjaxRequest] [bit] NOT NULL,
[Parameters] [varchar](max) NOT NULL,
[Controller] [varchar](50) NOT NULL,
[Action] [varchar](50) NOT NULL,
[Comments] [varchar](max) NULL,
[BeforeImage] [varchar](max) NULL,
[AfterImage] [varchar](max) NULL,
[Browser] [varchar](max) NULL
)
WITH
(
DATA_SOURCE = [RemoteReferenceData],
SCHEMA_NAME = 'dbo', -- I missed this part
OBJECT_NAME = 'AuditLog' -- I missed this part
);
So my problem was that I had omitted the SCHEMA_NAME = 'dbo' and OBJECT_NAME = 'AuditLog' which makes a reference to the AuditLog table in DB1. With my OP, Azure was making a reference to AuditLogSource in DB1 which obviously doesn't exist hence I get the error I was getting. BUT, it would help if the query failed in the first place coz that would've highlighted that there was something wrong somewhere. Anyway, I hope this helps someone.
I recently lost an Azure database (there's no recovering it...) and had to recreate from scratch. This database is supposed to sync to our iOS app, which allows us to keep the data between the DB and Device in sync. Unfortunately, after recreating the database and tables manually (because I'm not sure how else to do it?), my App Service is able to connect to the SQL database, and from the app side it looks like it's syncing, but when I look at the logs, I'm seeing the following error for each table, and I get no data:
The table 'tablename' does not support the 'updatedAt' system
property.
I've searched everywhere for this system property and cannot for the life of me find out where it's at or how to relate it to the database tables.
Here's an example of a schema for a table I'm having issues with:
CREATE TABLE [dbo].[CurrentItems](
[id] [nvarchar](128) NOT NULL,
[siteID] [nvarchar](max) NOT NULL,
[residentUnitNumber] [nvarchar](50) NULL,
[residentID] [nvarchar](max) NULL,
[pinUsedToCheckOut] [nvarchar](max) NULL,
[itemType] [nvarchar](max) NULL,
[itemTrackingNumber] [nvarchar](max) NULL,
[itemNotes] [nvarchar](max) NULL,
[itemID] [nvarchar](max) NULL,
[deviceSerialNumber] [nvarchar](max) NULL,
[dateRecordCreated] [datetime] NOT NULL,
[dateLastModified] [datetime] NULL,
[dateItemCheckedOut] [datetime] NULL,
[dateItemCheckedIn] [datetime] NULL,
[currentStatus] [nvarchar](max) NOT NULL,
[checkedOutEmpID] [nvarchar](max) NULL,
[checkedInEmpID] [nvarchar](max) NOT NULL,
[carrierName] [nvarchar](max) NULL,
[barcodeValue] [nvarchar](max) NULL,
[version] [timestamp] NOT NULL,
[deleted] [bit] NOT NULL,
[createdAt] [datetimeoffset](7) NOT NULL,
[updatedAt] [datetimeoffset](7) NOT NULL,
CONSTRAINT [PK_CurrentItems] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
I really need to get this app back to syncing, even if that means starting from scratch, but I can't seem to find the instructions for that on the Azure website anymore, so I don't know what else to do.
Any help or guidance anyone can provide would be greatly appreciated!
I am having an issue which my table is not able to accept the word 'Quasar' as a string. It will give this error.
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.).
As it's just a NVARCHAR datatype column why is it not accepting 'Quasar' which is not even a keyword nor has any special characters in the word?
Column Name = 'PickName' <- this column having the issue.
I post my table structure and also my insert script below.
CREATE TABLE [dbo].[ConnoteTemplate](
[TemplateNo] [nvarchar](50) NOT NULL,
[CustNo] [nvarchar](50) NOT NULL,
[CtrtNo] [nvarchar](50) NOT NULL,
[CostCentreNo] [nvarchar](50) NOT NULL,
[OperationHubNo] [nvarchar](50) NOT NULL,
[PickName] [nvarchar](50) NULL,
[PickADDR1] [nvarchar](100) NULL,
[PickADDR2] [nvarchar](100) NULL,
[PickADDR3] [nvarchar](100) NULL,
[PickADDR4] [nvarchar](100) NULL,
[PickContact] [nvarchar](50) NULL,
[PickDept] [nvarchar](50) NULL,
[PickTel1] [varchar](20) NULL,
[PickTel2] [varchar](20) NULL,
[PickPostal] [varchar](15) NULL,
[PickDistNo] [nvarchar](50) NULL,
[DlvyName] [nvarchar](50) NULL,
[DlvyADDR1] [nvarchar](100) NULL,
[DlvyADDR2] [nvarchar](100) NULL,
[DlvyADDR3] [nvarchar](100) NULL,
[DlvyADDR4] [nvarchar](100) NULL,
[DlvyContact] [nvarchar](50) NULL,
[DlvyDept] [nvarchar](50) NULL,
[DlvyTel1] [varchar](20) NULL,
[DlvyTel2] [varchar](20) NULL,
[DlvyPostal] [varchar](15) NULL,
[DlvyDistNo] [nvarchar](50) NULL,
[CreateUser] [nvarchar](100) NOT NULL,
[CreateDateTime] [datetime] NOT NULL,
[LastEditUser] [nvarchar](100) NULL,
[LastEditDateTime] [datetime] NULL,
[ComputerName] [nvarchar](100) NOT NULL,
[IPAddress] [nvarchar](100) NOT NULL
) ON [PRIMARY]
GO
INSERT INTO [dbo].[ConnoteTemplate]([TemplateNo],[CustNo],[CtrtNo],[CostCentreNo],[OperationHubNo],[PickName],[PickADDR1],[PickADDR2],[PickADDR3],[PickADDR4],[PickContact],[PickDept],[PickTel1],[PickTel2],[PickPostal],[PickDistNo],[DlvyName],[DlvyADDR1],[DlvyADDR2],[DlvyADDR3],[DlvyADDR4],[DlvyContact],[DlvyDept],[DlvyTel1],[DlvyTel2],[DlvyPostal],[DlvyDistNo],[CreateUser],[CreateDateTime],[LastEditUser],[LastEditDateTime],[ComputerName],[IPAddress])
VALUES
(1,1,1,1,1,'Quasar','','','','','','','','','','','','','','','','','','','','','','','2020-01-22','',null,'','')
GO
Edited:
Long story short.... try running this query below and it will give you the error.
SELECT CAST('Quasar' AS NVARCHAR(50))
Another funny thing is if you try to run this query below, it will post no issue.
SELECT CAST('quasar' AS NVARCHAR(50))
I am using SSMS v 17.9.1/SQL Server 14.0.
https://www.reddit.com/r/SQL/comments/ese9nt/im_not_crazy_who_can_run_this_without_error_on/
One of the comments:
“After discussing with the networking team and reviewing the traffic it appears that for whatever reason the binary string that gets returned from the server is being flagged as part of traffic from a spyware executable and being blocked by the firewall. It only happens on certain subnets because communications to them is routed through the firewall, while communication to the others is not.”
#Irving Lee, I don't see any issue with your CREATE and INSERT statements.
In general, we notice
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
error when you disconnect from internet and try to execute the queries from same SSMS query window. Re-execute will normally execute them fine.
But why did you conclude that its issue with that specific string - "Quasar"?
I have created a table which has the columns
Id, name, CountryId,CreatedOn,CreatedBy,updatedOn,UpdatedBy
but when i see the design, there is no CountryId.
When i see script of the created table, it shows CountryId
the script is as follows --
CREATE TABLE [dbo].[State](
[Id] [int] NOT NULL,
[name] [nvarchar](255) NOT NULL,
[CountryId] [int] NOT NULL,
[CreatedOn] [datetime] NULL,
[CreatedBy] [nvarchar](50) NULL,
[UpdatedOn] [datetime] NULL,
[UpdatedBy] [nvarchar](50) NULL,
);
Also, When I right click and select top 100 rows, i find CountryId there. (The following image)
What can be the reason that the design is not showing a column ??
EDIT:
As suggested by shnugo, i was able to solve it after closing and re-opening SSMS 2014
I have two tables in our data warehouse which need to be audited on several levels. The structure of the first table is as follows:
CREATE TABLE [dbo].[EnctrAPR](
[EncounterNumber] [varchar](50) NOT NULL,
[MedRec] [varchar](50) NULL,
[AdmitDate ] [datetime] NULL,
[DischargeDate ] [datetime] NULL,
[Age ] [varchar](50) NULL,
[Disposition ] [varchar](50) NULL,
[DRG ] [varchar](50) NULL,
[APRDRG ] [varchar](50) NULL,
[APRDRGWeight] [varchar](50) NULL,
[SOI] [varchar](50) NULL,
[ROM] [varchar](50) NULL,
[Age18] [bit] NULL,
[Age18To64] [bit] NULL,
[Age65] [bit] NULL,
[BowelObstruction] [bit] NULL,
[AMI] [bit] NULL,
[CABG] [bit] NULL,
[Valve] [bit] NULL,
[PCI] [bit] NULL,
[Sepsis] [bit] NULL,
[GSP] [bit] NULL,
[HF] [bit] NULL,
[Stroke] [bit] NULL,
[Pneumonia] [bit] NULL,
[DKA] [bit] NULL,
[GIBleed] [bit] NULL,
[Pancreatitis] [bit] NULL,
[RF] [bit] NULL,
[PE] [bit] NULL,
[COPD] [bit] NULL,
CONSTRAINT [PK_EnctrAPR] PRIMARY KEY CLUSTERED
(
[EncounterNumber] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
I need to do several audits against another table in the data warehouse. They are:
Identify records in above table that aren't in the DWH table.
Identify records in the DWH table that aren't in the above table.
Document records where each of the bit fields above is TRUE but are FALSE in the DWH.
Confirm overall record count for a given date range.
Can I use SSIS to accomplish these audits? Or, would a combinaion of stored procedures and SSIS be the best bet? Please provide me with some suggestions on ow to best automate such auditing.
You could use the Lookup Transformation in SSIS to accomplish this, but this component is a little slow. If the databases are on different servers I would be inclined to use SSIS. If they are in the same instance, I would probably use a Stored Procedure without SSIS.
If you do use the SSIS approach, you would have a OLEDB source to read the rows from ODH, a lookup transformation against the table above and then write the results out to a destination.
I see no benefit on combining SSIS and an SP for this specific task.