I have a pair of databases (one mssql and one oracle), ran by different teams. Some data are now being synchronized regularily by a stored procedure in the mssql table. This stored procedure is calling a very large
MERGE [mssqltable].[Mytable] as s
USING THEORACLETABLE.BLA as t
ON t.[R_ID] = s.[R_ID]
WHEN MATCHED THEN UPDATE SET [Field1] = s.[Field1], ..., [Brokenfield] = s.[BrokenField]
WHEN NOT MATCHED BY TARGET THEN
... another big statement
Field Brokenfield was a numeric one until today, and could take value NULL, 0, 1, .., 24
Now, the oracle team introduced a breaking change today for some reason, changed the type of the column to string and now has values NULL, "", "ALFA", "BRAVO"... in the column. Of course, the sync got broken.
What is the easiest way to fix the sync here? I (Mysql team lead, frontend expert but not so in databases) would usually apply one of our database expert guys here, but all of them are now ill, and the fix must go online today....
I thought of a stored procedure like CONVERT_BROKENFIELD_INT_TO_STRING or so, based on some switch-case, which could be called in that merge statement, but not sure how to do that.
Edit/Clarification:
What I need is a way to make a chunk of SQL code (stored procedure), taking an input of "ALFA" and returning 1, "BRAVO" -> 2, etc. and which can be reused, to avoid writing huge ifs in more then one place.
If you can not simplify the logic for correct values the way #RichardHansell desribed, you can create a crosswalk table for BrokenField to the correct values. Then you can use a common table expression or subquery with a left join to that crosswalk to use in the merge.
create table dbo.BrokenField_Crosswalk (
BrokenField varchar(32) not null primary key
, CorrectedValue int
);
insert into dbo.BrokenField_Crosswalk (BrokenField,CorrectedValue) values
('ALFA', 1)
, ('ALPHA', 1)
, ('BRAVO', 2)
...
go
And your code for the merge would look something like this:
;with cte as (
select o.R_ID
, o.Field1
, BrokenField = cast(isnull(c.CorrectedValue,o.BrokenField) as int)
....
from oracle_table.bla as o
left join dbo.BrokenField_Crosswalk as c
)
merge into [mssqltable].[Mytable] t
using cte as s
on t.[R_ID] = s.[R_ID]
when matched
then update set
[Field1] = s.[Field1]
, ...
, [Brokenfield] = s.[BrokenField]
when not matched by target
then
If they are using names with a letter at the start that goes in a sequence:
A = 1
B = 2
C = 3
etc.
Then you could do something like this:
MERGE [mssqltable].[Mytable] as s
USING THEORACLETABLE.BLA as t
ON t.[R_ID], 1)) - ASCII('A') + 1 = s.[R_ID]
WHEN MATCHED THEN UPDATE SET [Field1] = s.[Field1], ..., [Brokenfield] = s.[BrokenField]
WHEN NOT MATCHED BY TARGET THEN
... another big statement
Edit: but actually I re-read your question and you are talking about [Brokenfield] being the problem column, so my solution wouldn't work.
I don't really understand now, as it seems as though the MERGE statement is updating the oracle table with numbers, so surely you need the mapping to work the other way, i.e. 1 -> ALFA, 2 -> BETA, etc.?
I have a view in my database that has a bunch of fields derived from other information in the database, this is how the view is defined:
create view patient_account_view AS
select patient.p_mrn,
p_fname,
p_lname,
ammount_paid,
quantity*item_cost + repeats*item_cost "ammount_owing",
(quantity*item_cost + repeats*item_cost) - ammount_paid "balance"
from patient_account,
patient,
diagnosis,
prescribed_treatment,
items_used,
item,
perscription
where patient.p_mrn = diagnosis.p_mrn AND
patient_account.p_mrn = patient.p_mrn AND
diagnosis.prescribed_treatment_id = prescribed_treatment.prescribed_treatment_id AND
prescribed_treatment.prescribed_treatment_id = perscription.prescribed_treatment_id AND
items_used.ptreatment_id = prescribed_treatment.prescribed_treatment_id AND
items_used.item_number = item.item_number;
I would like to use pl/sql to access the information in the view to stick it into a form, but I'm getting a 'bad bind variable' error. How do I access this kind of attribute without having to recalculate the information stored there?
Here is the plsql that is problematic:
DECLARE
pmrn patient.p_mrn%TYPE;
var_ptuple patient%ROWTYPE;
var_accttuple patient_account%ROWTYPE;
BEGIN
pmrn := :PATIENT_BLOCK.MRN_FIELD;
SELECT * INTO var_ptuple from patient WHERE patient.p_mrn = pmrn;
SELECT * INTO var_accttuple from patient_account_view WHERE patient_account_view.p_mrn = pmrn;
:PATIENT_BLOCK.FNAME := var_ptuple.p_fname;
:PATIENT_BLOCK.LNAME := var_ptuple.p_lname;
:PATIENT_BLOCK.BALACNCE_OWING := var_accttuple.balance;
END;
The columns of your view patient_account_view do not match exactly the columns of the table patient_account, but in your code you have:
var_accttuple patient_account%ROWTYPE;
which means when you run this:
SELECT * INTO var_accttuple from patient_account_view ...
You haven't specified which columns get mapped to which record attributes, so Oracle requires that the column list matches exactly.
In this case, I'd expect you probably want to change the definition of the variable, e.g.
var_accttuple patient_account_view%ROWTYPE;
Side note
Since you're only using one attribute from the view, you can simplify your code as follows:
SELECT balance INTO :PATIENT_BLOCK.BALACNCE_OWING
from patient_account_view WHERE patient_account_view.p_mrn = pmrn;
and you no longer need var_accttuple.
I am not much strong in SQL, so looking for some help.
First I am looking for suggestion for the best way to implement this logic in SQL and then some sample code to implement.
My portal is going to connect Students and Training Providers.
Students: Select what courses (multiple) they want, type of delivery (online, class room), Industry(domain) to which the course to be targeted more, Location Preference.
Training Providers: Select what courses offering (so one record for each course), offering locations, type of delivery for each course, industries (multiple) it is targeting.
When student login:
I would like to create SP which in turn create view to store the matched records of the Training Providers data which matches that student needs of that StudentID, CourseID passed to SP
I have created the following sp ( but not included create view part as I am not sure how to do this)
set ANSI_NULLS ON
set QUOTED_IDENTIFIER ON
go
ALTER PROCEDURE [dbo].[sp_TPsMatched2StuCourse]
-- Add the parameters for the stored procedure here
#StuID int,
#CourseID int
AS
BEGIN
Select TP.MemID,TP.PastExp,SN.DeliveryType,SN.LocPref,SN.Industry,SC.CourseID from
tbl_TrainingProvider as TP , tbl_StuCourses as SC, tbl_StuNeeds SN
where SN.CourseID = #CourseID and SN.StuID = #StuID and
SN.DeliveryType in (TP.DeliveryMode) and
SN.LocPref IN (TP.LocOffering) and
SN.Industry IN (TP.Industries)
END
--- exec sp_ELsMatched2EntProp 1, 1
Why I need to put the data is as follows:
Assume the data is stored in that dynamic view and that would be bind to datagrid. Student then select interested TPs. Then only contact details would be shared to each other and this cannot be reveresed. So I would put this interested data in another table later. Every time data changes, hence the matches. Student can change some of his/her needs or new TPs join etc so view to be temparory.
when I executed this using above command, I am not getting data though it matches few records. What is wrong I am doing.
Any help would be greatly appreciated.
You are not getting expected results because you filter out too many records in WHERE( I'm talking about this part : SN.DeliveryType in (TP.DeliveryMode) and
SN.LocPref IN (TP.LocOffering) and SN.Industry IN (TP.Industries)). I'd recommend to use JOIN ... ON instead of specifying all tables in FROM and join condition in WHERE. I'm not sure what you want exactly, but I believe you are looking for
FROM tbl_StuNeeds SN
LEFT JOIN tbl_TrainingProvider as TP ON (TP.DeliveryMode = SN.DeliveryType AND
SN.LocPref = TP.LocOffering AND TP.Industries = SN.Industry)
WHERE SN.CourseID = #CourseID and SN.StuID = #StuID
Also, there is no join conditions in your code for tbl_StuCourses as SC which results in cross-join.
Finally, why do you need a stored procedure at all? From what I see in your example, a table-valued function will work better:
CREATE FUNCTION [dbo].getTPsMatched2StuCourse(#StuID INT,#CourseID INT)
RETURNS TABLE AS
RETURN
Select .... ;
I have a table column that uses an enum type. I wish to update that enum type to have an additional possible value. I don't want to delete any existing values, just add the new value. What is the simplest way to do this?
PostgreSQL 9.1 introduces ability to ALTER Enum types:
ALTER TYPE enum_type ADD VALUE 'new_value'; -- appends to list
ALTER TYPE enum_type ADD VALUE 'new_value' BEFORE 'old_value';
ALTER TYPE enum_type ADD VALUE 'new_value' AFTER 'old_value';
NOTE if you're using PostgreSQL 9.1 or later, and you are ok with making changes outside of a transaction, see this answer for a simpler approach.
I had the same problem few days ago and found this post. So my answer can be helpful for someone who is looking for solution :)
If you have only one or two columns which use the enum type you want to change, you can try this. Also you can change the order of values in the new type.
-- 1. rename the enum type you want to change
alter type some_enum_type rename to _some_enum_type;
-- 2. create new type
create type some_enum_type as enum ('old', 'values', 'and', 'new', 'ones');
-- 3. rename column(s) which uses our enum type
alter table some_table rename column some_column to _some_column;
-- 4. add new column of new type
alter table some_table add some_column some_enum_type not null default 'new';
-- 5. copy values to the new column
update some_table set some_column = _some_column::text::some_enum_type;
-- 6. remove old column and type
alter table some_table drop column _some_column;
drop type _some_enum_type;
3-6 should be repeated if there is more than 1 column.
A possible solution is the following; precondition is, that there are not conflicts in the used enum values. (e.g. when removing an enum value, be sure that this value is not used anymore.)
-- rename the old enum
alter type my_enum rename to my_enum__;
-- create the new enum
create type my_enum as enum ('value1', 'value2', 'value3');
-- alter all you enum columns
alter table my_table
alter column my_column type my_enum using my_column::text::my_enum;
-- drop the old enum
drop type my_enum__;
Also in this way the column order will not be changed.
If you are using Postgres 12 (or later) you can just run ALTER TYPE ... ADD VALUE inside of transaction (documentation).
If ALTER TYPE ... ADD VALUE (the form that adds a new value to an enum
type) is executed inside a transaction block, the new value cannot be
used until after the transaction has been committed.
So no hacks needed in migrations.
UPD: here is an example (thanks to Nick for it)
ALTER TYPE enum_type ADD VALUE 'new_value';
If you fall into situation when you should add enum values in transaction, f.e. execute it in flyway migration on ALTER TYPE statement you will be get error ERROR: ALTER TYPE ... ADD cannot run inside a transaction block (see flyway issue #350) you could add such values into pg_enum directly as workaround (type_egais_units is name of target enum):
INSERT INTO pg_enum (enumtypid, enumlabel, enumsortorder)
SELECT 'type_egais_units'::regtype::oid, 'NEW_ENUM_VALUE', ( SELECT MAX(enumsortorder) + 1 FROM pg_enum WHERE enumtypid = 'type_egais_units'::regtype )
Complementing #Dariusz 1
For Rails 4.2.1, there's this doc section:
== Transactional Migrations
If the database adapter supports DDL transactions, all migrations will
automatically be wrapped in a transaction. There are queries that you
can't execute inside a transaction though, and for these situations
you can turn the automatic transactions off.
class ChangeEnum < ActiveRecord::Migration
disable_ddl_transaction!
def up
execute "ALTER TYPE model_size ADD VALUE 'new_value'"
end
end
just in case, if you are using Rails and you have several statements you will need to execute one by one, like:
execute "ALTER TYPE XXX ADD VALUE IF NOT EXISTS 'YYY';"
execute "ALTER TYPE XXX ADD VALUE IF NOT EXISTS 'ZZZ';"
From Postgres 9.1 Documentation:
ALTER TYPE name ADD VALUE new_enum_value [ { BEFORE | AFTER } existing_enum_value ]
Example:
ALTER TYPE user_status ADD VALUE 'PROVISIONAL' AFTER 'NORMAL'
Disclaimer: I haven't tried this solution, so it might not work ;-)
You should be looking at pg_enum. If you only want to change the label of an existing ENUM, a simple UPDATE will do it.
To add a new ENUM values:
First insert the new value into pg_enum. If the new value has to be the last, you're done.
If not (you need to a new ENUM value in between existing ones), you'll have to update each distinct value in your table, going from the uppermost to the lowest...
Then you'll just have to rename them in pg_enum in the opposite order.
Illustration
You have the following set of labels:
ENUM ('enum1', 'enum2', 'enum3')
and you want to obtain:
ENUM ('enum1', 'enum1b', 'enum2', 'enum3')
then:
INSERT INTO pg_enum (OID, 'newenum3');
UPDATE TABLE SET enumvalue TO 'newenum3' WHERE enumvalue='enum3';
UPDATE TABLE SET enumvalue TO 'enum3' WHERE enumvalue='enum2';
then:
UPDATE TABLE pg_enum SET name='enum1b' WHERE name='enum2' AND enumtypid=OID;
And so on...
I can't seem to post a comment, so I'll just say that updating pg_enum works in Postgres 8.4 . For the way our enums are set up, I've added new values to existing enum types via:
INSERT INTO pg_enum (enumtypid, enumlabel)
SELECT typelem, 'NEWENUM' FROM pg_type WHERE
typname = '_ENUMNAME_WITH_LEADING_UNDERSCORE';
It's a little scary, but it makes sense given the way Postgres actually stores its data.
Updating pg_enum works, as does the intermediary column trick highlighted above. One can also use USING magic to change the column's type directly:
CREATE TYPE test AS enum('a', 'b');
CREATE TABLE foo (bar test);
INSERT INTO foo VALUES ('a'), ('b');
ALTER TABLE foo ALTER COLUMN bar TYPE varchar;
DROP TYPE test;
CREATE TYPE test as enum('a', 'b', 'c');
ALTER TABLE foo ALTER COLUMN bar TYPE test
USING CASE
WHEN bar = ANY (enum_range(null::test)::varchar[])
THEN bar::test
WHEN bar = ANY ('{convert, these, values}'::varchar[])
THEN 'c'::test
ELSE NULL
END;
As long as you've no functions that explicitly require or return that enum, you're good. (pgsql will complain when you drop the type if there are.)
Also, note that PG9.1 is introducing an ALTER TYPE statement, which will work on enums:
http://developer.postgresql.org/pgdocs/postgres/release-9-1-alpha.html
Can't add a comment to the appropriate place, but ALTER TABLE foo ALTER COLUMN bar TYPE new_enum_type USING bar::text::new_enum_type with a default on the column failed. I had to:
ALTER table ALTER COLUMN bar DROP DEFAULT;
and then it worked.
Here is a more general but a rather fast-working solution, which apart from changing the type itself updates all columns in the database using it. The method can be applied even if a new version of ENUM is different by more than one label or misses some of the original ones. The code below replaces my_schema.my_type AS ENUM ('a', 'b', 'c') with ENUM ('a', 'b', 'd', 'e'):
CREATE OR REPLACE FUNCTION tmp() RETURNS BOOLEAN AS
$BODY$
DECLARE
item RECORD;
BEGIN
-- 1. create new type in replacement to my_type
CREATE TYPE my_schema.my_type_NEW
AS ENUM ('a', 'b', 'd', 'e');
-- 2. select all columns in the db that have type my_type
FOR item IN
SELECT table_schema, table_name, column_name, udt_schema, udt_name
FROM information_schema.columns
WHERE
udt_schema = 'my_schema'
AND udt_name = 'my_type'
LOOP
-- 3. Change the type of every column using my_type to my_type_NEW
EXECUTE
' ALTER TABLE ' || item.table_schema || '.' || item.table_name
|| ' ALTER COLUMN ' || item.column_name
|| ' TYPE my_schema.my_type_NEW'
|| ' USING ' || item.column_name || '::text::my_schema.my_type_NEW;';
END LOOP;
-- 4. Delete an old version of the type
DROP TYPE my_schema.my_type;
-- 5. Remove _NEW suffix from the new type
ALTER TYPE my_schema.my_type_NEW
RENAME TO my_type;
RETURN true;
END
$BODY$
LANGUAGE 'plpgsql';
SELECT * FROM tmp();
DROP FUNCTION tmp();
The whole process will run fairly quickly, because if the order of labels persists, no actual change of data will happen. I applied the method on 5 tables using my_type and having 50,000−70,000 rows in each, and the whole process took just 10 seconds.
Of course, the function will return an exception in case if labels that are missing in the new version of the ENUM are used somewhere in the data, but in such situation something should be done beforehand anyway.
For those looking for an in-transaction solution, the following seems to work.
Instead of an ENUM, a DOMAIN shall be used on type TEXT with a constraint checking that the value is within the specified list of allowed values (as suggested by some comments). The only problem is that no constraint can be added (and thus neither modified) to a domain if it is used by any composite type (the docs merely says this "should eventually be improved"). Such a restriction may be worked around, however, using a constraint calling a function, as follows.
START TRANSACTION;
CREATE FUNCTION test_is_allowed_label(lbl TEXT) RETURNS BOOL AS $function$
SELECT lbl IN ('one', 'two', 'three');
$function$ LANGUAGE SQL IMMUTABLE;
CREATE DOMAIN test_domain AS TEXT CONSTRAINT val_check CHECK (test_is_allowed_label(value));
CREATE TYPE test_composite AS (num INT, word test_domain);
CREATE TABLE test_table (val test_composite);
INSERT INTO test_table (val) VALUES ((1, 'one')::test_composite), ((3, 'three')::test_composite);
-- INSERT INTO test_table (val) VALUES ((4, 'four')::test_composite); -- restricted by the CHECK constraint
CREATE VIEW test_view AS SELECT * FROM test_table; -- just to show that the views using the type work as expected
CREATE OR REPLACE FUNCTION test_is_allowed_label(lbl TEXT) RETURNS BOOL AS $function$
SELECT lbl IN ('one', 'two', 'three', 'four');
$function$ LANGUAGE SQL IMMUTABLE;
INSERT INTO test_table (val) VALUES ((4, 'four')::test_composite); -- allowed by the new effective definition of the constraint
SELECT * FROM test_view;
CREATE OR REPLACE FUNCTION test_is_allowed_label(lbl TEXT) RETURNS BOOL AS $function$
SELECT lbl IN ('one', 'two', 'three');
$function$ LANGUAGE SQL IMMUTABLE;
-- INSERT INTO test_table (val) VALUES ((4, 'four')::test_composite); -- restricted by the CHECK constraint, again
SELECT * FROM test_view; -- note the view lists the restricted value 'four' as no checks are made on existing data
DROP VIEW test_view;
DROP TABLE test_table;
DROP TYPE test_composite;
DROP DOMAIN test_domain;
DROP FUNCTION test_is_allowed_label(TEXT);
COMMIT;
Previously, I used a solution similar to the accepted answer, but it is far from being good once views or functions or composite types (and especially views using other views using the modified ENUMs...) are considered. The solution proposed in this answer seems to work under any conditions.
The only disadvantage is that no checks are performed on existing data when some allowed values are removed (which might be acceptable, especially for this question). (A call to ALTER DOMAIN test_domain VALIDATE CONSTRAINT val_check ends up with the same error as adding a new constraint to the domain used by a composite type, unfortunately.)
Note that a slight modification such as CHECK (value = ANY(get_allowed_values())), where get_allowed_values() function returned the list of allowed values, would not work - which is quite strange, so I hope the solution proposed above works reliably (it does for me, so far...). (it works, actually - it was my error)
As discussed above, ALTER command cannot be written inside a transaction. The suggested way is to insert into the pg_enum table directly, by retrieving the typelem from pg_type table and calculating the next enumsortorder number;
Following is the code that I use. (Checks if duplicate value exists before inserting (constraint between enumtypid and enumlabel name)
INSERT INTO pg_enum (enumtypid, enumlabel, enumsortorder)
SELECT typelem,
'NEW_ENUM_VALUE',
(SELECT MAX(enumsortorder) + 1
FROM pg_enum e
JOIN pg_type p
ON p.typelem = e.enumtypid
WHERE p.typname = '_mytypename'
)
FROM pg_type p
WHERE p.typname = '_mytypename'
AND NOT EXISTS (
SELECT * FROM
pg_enum e
JOIN pg_type p
ON p.typelem = e.enumtypid
WHERE e.enumlabel = 'NEW_ENUM_VALUE'
AND p.typname = '_mytypename'
)
Note that your type name is prepended with an underscore in the pg_type table. Also, the typname needs to be all lowercase in the where clause.
Now this can be written safely into your db migrate script.
DB::statement("ALTER TABLE users DROP CONSTRAINT users_user_type_check");
$types = ['old_type1', 'old_type1', 'new_type3'];
$result = join( ', ', array_map(function ($value){
return sprintf("'%s'::character varying", $value);
}, $types));
DB::statement("ALTER TABLE users ADD CONSTRAINT users_user_type_check CHECK (user_type::text = ANY (ARRAY[$result]::text[]))");
When using Navicat you can go to types (under view -> others -> types) - get the design view of the type - and click the "add label" button.
I don't know if have other option but we can drop the value using:
select oid from pg_type where typname = 'fase';'
select * from pg_enum where enumtypid = 24773;'
select * from pg_enum where enumtypid = 24773 and enumsortorder = 6;
delete from pg_enum where enumtypid = 24773 and enumsortorder = 6;
Simplest: get rid of enums. They are not easily modifiable, and thus should very rarely be used.