I have a lot of mundane update statements I have to push through PLSQL and MSSQL and it requires a lot of editing. I will usually pull the data into excel than reformat it in notepad++ . This is pretty time consuming and I was wondering if there are any solutions for this?
UPDATE ATGSITES SET IPADDRESS = 'xxx' WHERE OWNSITEID = '270789'
UPDATE ATGSITES SET IPADDRESS = '1yyy' WHERE OWNSITEID = '270506'
UPDATE ATGSITES SET IPADDRESS = '158568' WHERE OWNSITEID = '27745'
X(35353) update statements
Perhaps you can create a table of updates and perform a single join
UPDATE ATGSITES SET IPADDRESS = B.IPADDRESS
From ATGSITES A
Join NewTable B on A.OWNSITEID = B.OWNSITEID
Where NewTable has the structure of OWNSITEID and IPADDRESS
I would also add an index on OWNSITEID
In Ms Sql Server
Save the data in a CSV text file delimited by comma like that format
IPADDRESS , OWNSITEID
xxx ,270789
1yyy ,270506
158568 ,27745
......
...........
suppose that the table to be updated is named: users_ip
Run the following script to update table from the text file
-- update table users_ip from csv file delimited wth , and first row is header
-- create temp table for loading the text file
CREATE TABLE #tmp_x (IPADDRESS nvarchar (10),OWNSITEID nvarchar(10))
go
-- import the csv file
BULK INSERT #tmp_x
FROM 'c:\temp\data.txt' --CSV file with header delimited by ,
WITH ( FIELDTERMINATOR =',',rowterminator = '\n',FIRSTROW = 2 )
go
update u
set u.IPADDRESS = t.IPADDRESS
from users_ip u
join #tmp_x t on t.OWNSITEID = u.OWNSITEID
//drop the temp table
drop table #tmp_x
Related
I'm trying to generalize a snowflake script with variables and currently stuck on how to use variables as column names to insert data into the table.
Here's what I'm trying to do.
Source table variables
set schem_name = 'schema';
set table_name = 'destination';
set s_var1 = 'source_col_1';
set s_var2 = 'source_col_2';
destination table variables
set d_var1 = 'dest_col_1';
set d_var2 = 'dest_col_2';
Now trying to insert into destination table
use schema identifier($schema_name);
below select statement works without issue.
select identifier($s_var1), identifier($s_var2)
from identifier($table_name);
But when trying to insert I keep getting errors.
This is what I tried.
insert into destination_table (identifier($d_var1), identifier($d_var2))
select identifier($s_var1), identifier($s_var2)
from identifier($table_name);
I have a .csv file and it gets updated every day. Below is the example of my .csv file
I am pushing this .csv file into SQL Server using Python. My script reads the .csv file and uploads it into a SQL Server database.
This is my Python script:
import pandas as pd
import pyodbc
df = pd.read_csv ("C:/Users/Dhilip/Downloads/test.csv")
print(df)
conn = pyodbc.connect('Driver={SQL Server};'
'Server=DESKTOP-7FCK7FG;'
'Database=test;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
#cursor.execute('CREATE TABLE people_info (Name nvarchar(50), Country nvarchar(50), Age int)')
for row in df.itertuples():
cursor.execute('''
INSERT INTO test.dbo.people_info (Name, Country, Age)
VALUES (?,?,?)
''',
row.Name,
row.Country,
row.Age
)
conn.commit()
The script is working fine. I am trying to automate my Python script using batch file and task scheduler, and it's working fine. However, whenever I add new data in the .csv file and SQL Server gets updated with new data and the same time it prints the old data multiple times.
Example, if I add new record called Israel, the output appears in SQL Server as below
I need output as below,
Can anyone advise me the change I need to do in the above python script?
You can use below query in your python script. if Not exists will check if the record already exists based on the condition in where clause and if record exists then it will go to else statement where you can update or do anything.
checking for existing records in database works faster than checking using python script.
if not exists (select * from Table where Name = '')
begin
insert into Table values('b', 'Japan', 70)
end
else
begin
update Table set Age=54, Country='Korea' where Name = 'A'
end
to find existing duplicate records then use the below query
select Name, count(Name) as dup_count from Table
group by Name having COUNT(Name) > 1
I find duplicates like this
def find_duplicates(table_name):
"""
find duplicates inside table
:param table_name:
:return:
"""
connection = sqlite3.connect("./k_db.db")
cursor = connection.cursor()
findduplicates = """ SELECT a.*
FROM {} a
JOIN (
SELECT shot, seq, lower(user), date_time,written_by, COUNT(*)
FROM {}
GROUP BY shot, seq, lower(user), date_time,written_by
HAVING count(*) > 1 ) b
ON a.shot = b.shot
AND a.seq = b.seq
AND a.date_time = b.date_time
AND a.written_by = b.written_by
ORDER BY a.shot;""".format(
table_name, table_name
)
# print(findduplicates)
cursor.execute(findduplicates)
connection.commit()
records = cursor.fetchall()
cursor.close()
connection.close()
You could rephrase your insert such that it checks for existence of the tuple before inserting:
for row in df.itertuples():
cursor.execute('''
INSERT INTO test.dbo.people_info (Name, Country, Age)
SELECT ?, ?, ?
WHERE NOT EXISTS (SELECT 1 FROM test.dbo.people_info
WHERE Name = ? AND Country = ? AND Age = ?)
''', (row.Name, row.Country, row.Age, row.Name, row.Country, row.Age,))
conn.commit()
An alternative to the above would be to add a unique index on (Name, Country, Age). Then, your duplicate insert attempts would fail and generate an error.
I have generate many xml with the SQL Server command : FOR XML RAW
In that way i have filled a table with the following schema
ACTION as CHAR(1)
TABLE_NAME as NVARCHAR(25)
PAYLOAD as NVARCHAR(MAX)
I
tbl_1
xmlrow_1
I
tbl_1
xmlrow_2
U
tbl_1
xmlrow_3
D
tbl_1
xmlrow_4
D
tbl_2
xmlrow_5
ACTION is a char ( I = insert, U = update, D = delete)
TABLE_NAME is the table on which i have to act (for insert the data, update it or delete it)
PAYLOAD is a XML serialized by SQL Server using the command FOR XML RAW on original table
PAYLOAD Example :
<row COL1="val_col_1" COL2="val_col_2" .. COLN="val_col_n"/>
I am looking for a way (i am writing a stored procedure so i am looking for TSQL) to deserialize it on the "configured" TABLE_NAME possibly for the UPSERT.
If this is not faseable (as i suspect) i will build the SQL script for insert,update or delete dynamically but i still need to deserialize the XML in the PAYLOAD and don't know how to do
I mean, if there is not a better way, how i can do some like that ?
UPDATE [dbo].[tbl_1]
SET [COL1] = CURRENT_ROW.COL1
,[COL2] = CURRENT_ROW.COL2
,[COL3] = CURRENT_ROW.COL3
FROM ( xmlrow_3 --DESERIALIZE ) AS CURRENT_ROW
--EIDTED : added working example in fiddler
https://dbfiddle.uk/?rdbms=sqlserver_2019&fiddle=ac5925a0a2f93791cc7e7c34179137ae
I am trying to load data from azure blob storage.
The data has already been staged.
But, the issue is when I try to run
copy into random_table_name
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv'
Below is the error I encounter:
raise error_class(
snowflake.connector.errors.ProgrammingError: 001757 (42601): SQL compilation error:
Table 'random_table_name' does not exist
Basically, it says table does not exist, which it does not, but the syntax on website is the same as mine.
COPY INTO query on Snowflake returns TABLE does not exist error
In my case the table name is case-sensitive. Snowflake seems to convert everything to upper case. I changed the database/schema/table names to all upper-case and it started working.
First run the below query to fetch the column headers
select $1 FROM #stage_name_i_created/filename.csv limit 1
Assuming below are the header lines from your csv file
id;first_name;last_name;email;age;location
Create a file_format csv
create or replace file format semicolon
type = 'CSV'
field_delimiter = ';'
skip_header=1;
Then you should define the datatype and field name as below
create or replace table <yourtable> as
select $1::varchar as id
,$2::varchar as first_name
,$3::varchar as last_name
,$4::varchar as email
,$5::int as age
,$6::varchar as location
FROM #stage_name_i_created/yourfile.csv
(file_format => semicolon );
The table must exist prior to running a COPY INTO command. In your post, you say that the table does not exist...so that is your issue.
If your table exist, try by forcing the table path like this:
copy into <database>.<schema>.<random_table_name>
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv'
or by steps like this:
use database <database_name>;
use schema <schema_name>;
copy into database.schema.random_table_name
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv';
rbachkaniwala, what do you mean by 'How do I create a table?( according to snowflake syntax it is not possible to create empty tables)'.
You can just do below to create a table
CREATE TABLE random_table_name (FIELD1 VARCHAR, FIELD2 VARCHAR)
The table does need to exist. You should check the documentation for COPY INTO.
Other areas to consider are
do you have the right context set for the database & schema
does the user / role have access to the table or object.
It basically seems like you don't have the table defined yet. You should
ensure the table is created
ensure all columns in the CSV exist as columns in the table
ensure the order of the columns are the same as in the CSV
I'd check data types too.
"COPY INTO" is not a query command, it is the actual data transfer execution from source to destination, which both must exist as others commented here but If you want just to query without loading the files then run the following SQL:
//Display list of files in the stage to verify stage
LIST #stage_name_i_created;
//Create a file format
CREATE OR REPLACE FILE FORMAT RANDOM_FILE_CSV
type = csv
COMPRESSION = 'GZIP' FIELD_DELIMITER = ',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = '\042'
TRIM_SPACE = FALSE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE' ESCAPE_UNENCLOSED_FIELD = 'NONE' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO'
NULL_IF = ('\\N');
//Now select the data in the files
Select $1 as first_col,$2 as second_col //can add as necessary number of columns ...etc
from #stage_name_i_created
(FILE_FORMAT => RANDOM_FILE_CSV)
More information can be found in the documentation link here
https://docs.snowflake.com/en/user-guide/querying-stage.html
I have a table named User in my database. In that there is a xml column named XmlText which contains lots of attributes.
<userDetails>
<MobileVerified>True</MobileVerified>
<EmailVerified>True</EmailVerified>
<IPAddress>122.160.65.232</IPAddress>
<Gender>Male</Gender>
<DateOfBirth>1970-03-22T00:00:00</DateOfBirth>
<DealingInProperties>residential_apartment_flat</DealingInProperties>
<DealingInProperties>residential_villa_bungalow</DealingInProperties>
<DealingInProperties>residential_farm_house</DealingInProperties>
</userDetails>
What is needed to do is i have to merge all the 'residential_villa_bungalow' values to 'residential_apartment_flat' if 'residential_apartment_flat' exists in the XmlText Column else 'residential_apartment_flat' will be left by default. There are approx 700000 record in the database so keep in mid that what technique can be used among normal update vs cursot.
Fire query with following columns "UserID,XmlText"
Probable logic wud be something like this..
if ('residential_villa_bungalow') exists
(
if ('residential_apartment_flat') exists
remove the 'residential_villa_bungalow' node as there must be only one 'residential_apartment_flat' node
else
update 'residential_villa_bungalow' into 'residential_apartment_flat'
)
XML Data Modification Language (XML DML)
-- Delete bungalow where already exist a flat
update YourTable
set XMLText.modify('delete /userDetails/DealingInProperties[. = "residential_villa_bungalow"] ')
where XMLText.exist('/userDetails[DealingInProperties = "residential_apartment_flat"]') = 1 and
XMLText.exist('/userDetails[DealingInProperties = "residential_villa_bungalow"]') = 1
-- Change value from bungalow to flat
update YourTable
set XMLText.modify('replace value of (/userDetails/DealingInProperties[. = "residential_villa_bungalow"]/text())[1]
with "residential_apartment_flat"')
where XMLText.exist('/userDetails[DealingInProperties = "residential_villa_bungalow"]') = 1