I have a project that I need to do a massive data load from a CSV file generated by an application. The structure of CSV file is as follows:
indice;hora;puerta;num;nombre;departamento;departamento;id usuario;estado;tarjeta
"0001";"05:51:56";"Parqueadero";"0046";"Rafael Iglesia";"ADMINISTRATIVOS";"Dep2_00";"9229926977";"(M11)Acceso Normal";"04756:22242"
"0002";"05:53:19";"Parqueadero";"0036";"Orlinda Torres";"ADMINISTRATIVOS";"Dep2_00";"4326087729";"(M11)Acceso Normal";"04246:24075"
As you can see, the first row doesn't use quoted identifiers (""), whereas the next row does. As such, I tried to do a bulk insert (using SQL Server 2017) with FIRSTROW=2 and the result is:
0 rows affected()
Conversely, if I use this method without FIRSTROW, SQL obviously can't process the file. Is there a way to process this data while skipping the first row?
This is the table I'm inserting into:
CREATE TABLE [dbo].[TEMPORAL_IV] (
[id_temp_dia] int NULL,
[hora_temp] [nvarchar](20) NULL,
[puerta_temp] [nvarchar](20) NULL,
[id_tar_temp] int NULL,
[usuario_temp] [nvarchar](100) NULL,
[desc_dep_temp] [nvarchar](30) NULL,
[id_dep_temp] [nvarchar](20) NULL,
[doc_usu_temp] [nvarchar](20) NULL,
[mensaje_temp] [nvarchar](200) NULL,
[num_tar_us_temp] [nvarchar](15) NULL
)
And here's the bulk insert command:
BULK INSERT TEMPORAL_IV
FROM '\\argos\informatica$\temp\datos.csv'
WITH (fieldterminator=';',FIRSTROW=2);
Is there an alternative?
Related
Is there someone who can help me figure out why I cannot query an external table that I created using my SQL Server Mgt Studio. I can see the external table if I expand External Tables but if I Right click and Select Top 1000 Rows I get an error that Invalid object name 'dbo.AuditLogSource'.
I am trying to copy a certain amount of data from an audit log table in DB1.AuditLog into ArchiveDB.AuditLog. I've followed the tutorials on how to use Elastic Queries to archive this simple task but I am now stuck at this point where I should query from the external table created locally in my ArchiveDB. Here's the process I followed maybe I made a mistake somewhere please help me:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '2019MoxvE!';
--DROP MASTER KEY;
CREATE DATABASE SCOPED CREDENTIAL SQL_Credential
WITH IDENTITY = 'myusername',
SECRET = '2019MoxvE!';
--DROP DATABASE SCOPED CREDENTIAL SQL_Credential;
CREATE EXTERNAL DATA SOURCE RemoteReferenceData
WITH
(
TYPE=RDBMS,
LOCATION='ourserver.database.windows.net',
DATABASE_NAME='DB1',
CREDENTIAL= SQL_Credential
);
--DROP EXTERNAL DATA SOURCE RemoteReferenceData;
CREATE EXTERNAL TABLE [dbo].[AuditLogSource]
(
[Id] [int] NOT NULL,
[Userid] [int] NOT NULL,
[ObjectId] [int] NULL,
[CreatedOn] [datetime] NOT NULL,
[ModifiedOn] [datetime] NOT NULL,
[ModifiedBy] [varchar](150) NOT NULL,
[Type] [int] NOT NULL,
[ActionTable] [varchar](50) NOT NULL,
[IsAjaxRequest] [bit] NOT NULL,
[Parameters] [varchar](max) NOT NULL,
[Controller] [varchar](50) NOT NULL,
[Action] [varchar](50) NOT NULL,
[Comments] [varchar](max) NULL,
[BeforeImage] [varchar](max) NULL,
[AfterImage] [varchar](max) NULL,
[Browser] [varchar](max) NULL
)
WITH (DATA_SOURCE = [RemoteReferenceData]);
--DROP EXTERNAL TABLE [dbo].[AuditLogSource];
INSERT INTO [dbo].[AuditLog]
SELECT al.* FROM [dbo].[AuditLogSource] al WHERE al.[CreatedOn] <= '2020/12/31' AND
NOT EXISTS(SELECT 1 FROM [dbo].[AuditLog] al1 WHERE al1.Id=al.Id);
If you see on below screenshot, you can see that there are no errors being highlighted on this query which means that the query window does recognise that the table AuditLogSource does exists but if I run the query it complains that it does not exists. I can also confirm that the user I am logged into the database with is the admin user and own of both DB1 and ArchiveDB What can I do to make this work?
Thanks in advance.
Make sure you're using the correct database also if you create a new SQL Server object, your newly created object does not get updated in the IntelliSense Local Cache and due to this, it shows an Invalid object name: dbo.AuditLogSource.Please follow below reference.
Ex: [DatabaseName].[Schema].[TableName]
Try:
Edit -> IntelliSense -> Refresh Local Cache or Ctrl + shift + R
Reference:
Sql server invalid object name - but tables are listed in SSMS tables list
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-external-table-transact-sql?view=sql-server-ver16&tabs=dedicated
Ok so I will post an answer to this question in case another person comes across the same/similar problem. So I only made 1 mistake in creating the External Table and this is because of the tutorials and other answers I saw on this very platform.
CREATE EXTERNAL TABLE [dbo].[AuditLogSource]
(
[Id] [int] NOT NULL,
[Userid] [int] NOT NULL,
[ObjectId] [int] NULL,
[CreatedOn] [datetime] NOT NULL,
[ModifiedOn] [datetime] NOT NULL,
[ModifiedBy] [varchar](150) NOT NULL,
[Type] [int] NOT NULL,
[ActionTable] [varchar](50) NOT NULL,
[IsAjaxRequest] [bit] NOT NULL,
[Parameters] [varchar](max) NOT NULL,
[Controller] [varchar](50) NOT NULL,
[Action] [varchar](50) NOT NULL,
[Comments] [varchar](max) NULL,
[BeforeImage] [varchar](max) NULL,
[AfterImage] [varchar](max) NULL,
[Browser] [varchar](max) NULL
)
WITH
(
DATA_SOURCE = [RemoteReferenceData],
SCHEMA_NAME = 'dbo', -- I missed this part
OBJECT_NAME = 'AuditLog' -- I missed this part
);
So my problem was that I had omitted the SCHEMA_NAME = 'dbo' and OBJECT_NAME = 'AuditLog' which makes a reference to the AuditLog table in DB1. With my OP, Azure was making a reference to AuditLogSource in DB1 which obviously doesn't exist hence I get the error I was getting. BUT, it would help if the query failed in the first place coz that would've highlighted that there was something wrong somewhere. Anyway, I hope this helps someone.
I'm making an SSIS solution to store students' marks in a data warehouse. The OLTP database has different schemas to separate the campuses. Both the OLTP and data warehouse has a "Marks" table, but the data warehouse has an additional CampusID column that is a foreign key that references a table called "Campus" that stores a list of the college's campuses. I'm looking for a way to store a table's schema name in a variable, evaluate it to decide what the CampusID must be, and insert that CampusID in the "Marks" table. E.g if the OLTP table's schema is "AucklandPark", the CampusID is 1 and that gets inserted into the "Marks" table.
This is for a project we have to do. I found this solution SSIS - Using the SQL Server Schema Name as a Variable for Queries and Procedure Calls, but it's not dynamic; the project variable in the solution has a fixed value and I'd have to create fifteen different packages to get the desired result.
The definition for the "Marks" table in the OLTP database looks like this:
CREATE TABLE AucklandPark.Marks(
MarkID INT PRIMARY KEY IDENTITY,
StudentID INT NOT NULL FOREIGN KEY REFERENCES AucklandPark.StudentInfo(StudentID) ON DELETE CASCADE,
FA_1 TINYINT CHECK(FA_1 BETWEEN 0 AND 100),
FA_2 TINYINT CHECK(FA_2 BETWEEN 0 AND 100),
FA_3 TINYINT CHECK(FA_3 BETWEEN 0 AND 100),
SA_1 TINYINT CHECK(SA_1 BETWEEN 0 AND 100),
SA_2 TINYINT CHECK(SA_2 BETWEEN 0 AND 100),
INT_1 TINYINT CHECK(INT_1 BETWEEN 0 AND 100),
INT_2 TINYINT CHECK(INT_2 BETWEEN 0 AND 100),
INT_3 TINYINT CHECK(INT_3 BETWEEN 0 AND 100)
);
GO
And the definition for the data warehouse looks like this (using script generation):
CREATE TABLE [dbo].[Marks](
[DW_MarkID] [int] IDENTITY(1,1) NOT NULL,
[MarkID] [int] NOT NULL,
[DW_StudentID] [int] NOT NULL,
[CourseID] [tinyint] NOT NULL,
[CampusID] [tinyint] NOT NULL,
[DW_FacilitatorID] [int] NOT NULL,
[DateID] [int] NOT NULL,
[FA_1] [tinyint] NULL,
[FA_2] [tinyint] NULL,
[FA_3] [tinyint] NULL,
[SA_1] [tinyint] NULL,
[SA_2] [tinyint] NULL,
[INT_1] [tinyint] NULL,
[INT_2] [tinyint] NULL,
[INT_3] [tinyint] NULL,
(I'm leaving out all the constraints it adds afterwards)
I solved the problem by adding a column for CampusID.
This is a question more about design than about solving a problem.
I created three tables as such
CREATE TABLE [CapInvUser](
[UserId] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](150) NOT NULL,
[AreaId] [int] NULL,
[Account] [varchar](150) NULL,
[mail] [varchar](150) NULL,
[UserLevelId] [int] NOT NULL
)
CREATE TABLE [CapInvUserLevel](
[UserLevelId] [int] IDENTITY(1,1) NOT NULL,
[Level] [varchar](50) NOT NULL
)
CREATE TABLE [CapInvUserRegistry](
[UserRegistryId] [int] IDENTITY(1,1) NOT NULL,
[UserLevelId] int NOT NULL,
[DateRegistry] DATE NOT NULL,
[RegistryStatus] VARCHAR(50) NOT NULL,
)
With a view that shows all the data on the first table with "AreaId" being parsed as the varchar identifier of that table, the UserLevel being parsed as the varchar value of that table, and a join of the registry status of the last one.
Right now when I want to register a new user, I insert into all three tables using separate queries, but I feel like I should have a way to insert into all of them at the same time.
I thought about using a stored procedure to insert, but I still don't know if that would be apropiate.
My question is
"Is there a more apropiate way of doing this?"
"Is there a way to create a view that will let me insert over it? (without passing the int value manually)"
--This are just representations of the tables, not the real ones.
-- I'm still learning how to work with SQL Server properly.
Thank you for your answers and/or guidance.
The most common way of doing this, in my experience, is to write a stored procedure that does all three inserts in the necessary order to create the FK relationships.
This would be my unequivocal recommendation.
I'm developing a SQL SERVER 2012 express and developer solution.
I will receive an xml in an stored procedure. In the stored procedure I will parse the xml and insert its data into a table.
My problem here is that in this xml could contain data that exists on the table, and I need to update the data on the table with the new one.
I don't want to check if each row in xml exists on the table.
I think I can use IGNORE_DUP_KEY but I'm not sure.
How can I update or insert new data without checking it?
This is the table where I want to insert (or update) the new data:
CREATE TABLE [dbo].[CODES]
(
[ID_CODE] [bigint] IDENTITY(1,1) NOT NULL,
[CODE_LEVEL] [tinyint] NOT NULL,
[CODE] [nvarchar](20) NOT NULL,
[COMMISIONING_FLAG] [tinyint] NOT NULL,
[IS_TRANSMITTED] [bit] NOT NULL,
[TIMESPAN] [datetime] NULL,
[USERNAME] [nvarchar](50) NULL,
[SOURCE] [nvarchar](50) NULL,
[REASON] [nvarchar](200) NULL
CONSTRAINT [PK_CODES] PRIMARY KEY CLUSTERED
(
[CODE_LEVEL] ASC,
[CODE] ASC
)
)
The "IGNORE_DUP_KEY" parameter ,is ignore inserting new row, if he is already exists, but it is not dealing with update in case it exists.
the solution to your request is by MERGE or DML operation (INSERT/UPDATE/DELETE) .
BTW,
The parameter "IGNORE_DUP_KEY" is covering existsnce for the index key only (index column).
I have a table that looks as follows below. I don't really want to create a C# application to insert rows into this table, if I can avoid it, because of the VarBinary column. My intent is to store a Crystal report .RPT file in this column. Is there a T-SQL statement I can execute to insert/update rows into this table, and include an .RPT file?
CREATE TABLE [Report].[MesReport](
[MesReportID] [int] IDENTITY(1,1) NOT NULL,
[ParentID] [int] NOT NULL,
[ReportTitle] [nvarchar](80) NOT NULL,
[ReportName] [nvarchar](80) NOT NULL,
[DatabaseServer] [nvarchar](80) NOT NULL,
[DatabaseName] [nvarchar](50) NOT NULL,
[Login] [nvarchar](80) NOT NULL,
[ReportFile] [varbinary](max) NULL,
You can get it into a variable like
DECLARE #VB varbinary(max)
SELECT #VB =BulkColumn FROM OPENROWSET(BULK
N'C:\YourReport.rpt', SINGLE_BLOB) AS Document
that you can then use in an insert statement