I'm trying use SSIS/SSDT to sync a PostgreSQL source view into a SQL Server target table. I have the initial sync working after a lot of trail n error.
The issue I'm having is I want to use an out loop to limit the scope of the Data Flow to sync on day at the a time.
I have the loop increasing the date by 1 fine.
The expression I'm using for the ODBC Source (PostgreSQL) is as follows
"SELECT * from usage r
where
start_time >= '" + ((DT_STR, 4, 1252) DATEPART("yyyy", #[User::PeriodStart]) + "-" + (DT_STR, 2, 1252) DATEPART("mm", #[User::PeriodStart]) + "-" + (DT_STR, 2, 1252) DATEPART("dd", #[User::PeriodStart]))
+ "'
and start_time < ('" + ((DT_STR, 4, 1252) DATEPART("yyyy", #[User::PeriodStart]) + "-" + (DT_STR, 2, 1252) DATEPART("mm", #[User::PeriodStart]) + "-" + (DT_STR, 2, 1252) DATEPART("dd", #[User::PeriodStart]))
+ "'::date + INTERVAL '1 day')
and r.chargeable=true
AND r.start_time > '2000-01-01T12:00:00'
order by r.start_time asc"
The #[User::PeriodStart] is only evaluated once and isn't updated each time its loops.
I'm sure this is simple but my google skills with SSIS is failing
Any help would be much appreciated
It fixed it self. not sure what I did.
Related
I was doing some data-check for 1-2 months old data in my sql server and a question rised in my mind.
I made a string 926084711140 and 20210926084711140 which I am considering to be a date-time stamp.
I thought of converting into actual datetime value. So I tried this:
'2021-' + SUBSTRING(CAST(th.edit_timestamp as varchar(50)),1,1) + '-' + SUBSTRING(CAST(th.edit_timestamp as varchar(50)),2,2)
And this returned a value 2021-09-.2 I am unable to understand this output.
I thought of doing substring but so far it failed.
Can anyone help regarding this? Any help would be appreciated.
Thanks!
As others have stated - you should fix your data model, but here is an alternative method:
Select cast(stuff(stuff(stuff(stuff(#date, 15, 0, '.'), 13, 0, ':'), 11, 0, ':'), 9, 0, ' ') as datetime)
And if you really have the first date format without the year you can do this:
Select cast(stuff(stuff(stuff(stuff(concat('2021', right(concat('0', #date), 13)), 15, 0, '.'), 13, 0, ':'), 11, 0, ':'), 9, 0, ' ') as datetime)
The first thing I'm going to tell you is: don't use this format. You have to fix the data input, don't try to fix the data output
But if there is no other way ...
I'm going to assume that your format is
aaaa-mm-ddThh:mi:ss.mmm
We can do something like this
select convert(datetime,'2021-09-26T08:47:11.140',126)
OUTPUT: 2021-09-26 08:47:11.140
declare #date varchar(50) = '20210926084711140'
select convert(datetime,substring(#date,1,4) + '-' + substring(#date,5,2) + '-'
+ substring(#date,7,2) + 'T' + substring(#date,9,2) + ':' +
substring(#date,11,2) + ':' + substring(#date,13,2) + '.' +
substring(#date,15,3),126)
one more time ... this is horrible.
try to fix your data source and not work with this.
Good luck
I have a java application that for some installations acceses a PostgreSQL database, while in others it acceses essentially the same database in Derby.
I have a SQL query that returns an examination record from the examination table. There is an exam_procedure table that relates to the examination table in a one (examination) to many fashion. I need to concatenate the potentially multiple string records in the exam_procedure table so that I can add a single string value to the query return that represents all the related exam_procedure records. For a variety of reasons (eg, joins return too many records, especially when multiple subqueries are needed for other related one to many tables), I need to do this via a subquery in the SELECT section of the main query. The following SQL works just fine for PostgreSQL, but my understanding is that array_agg is not available in Derby. What Derby subquery can I substitute for the PostgreSQL subquery?
Many thanks.
// part of the query
"SELECT "
+ "patient_id, "
+ "examination_date, "
+ "examination_number, "
+ "operating_physician_id, "
+ "referring_physician_id, "
+ "patient.last_name AS pt_last_name, "
+ "patient.first_name AS pt_first_name, "
+ "patient.middle_name AS pt_middle_name, "
+ "("
+ "SELECT "
+ "array_agg(prose) "
+ "FROM "
+ "exam_procedure "
+ "WHERE examination_id = " + examId
+ " GROUP BY examination_id"
+ ") AS agg_procedures, "
+ "FROM "
+ "examination "
+ "JOIN patient ON patient.id = examination.patient_id "
+ "WHERE "
+ "examination.id = ?"
;
I'm receiving dates in a MMddyyyy format but want to convert it to the date format in SQL/SSIS yyyy-MM-dd, how can I do so using an SSIS expression.
An example of dates I'm receiving is 03051978 but want it to appear in 1978-03-05.
Thank you
You can use the following expression:
(DT_DATE)(RIGHT("0000" + RIGHT("03051978",4),4) + "-" + RIGHT("00" + LEFT("03051978",2),2) + "-" + RIGHT("00" + SUBSTRING("03051978",3,2),2))
Just replace "03051978" with the column name, as an example:
(DT_DATE)(RIGHT("0000" + RIGHT([DateColumn],4),4) + "-" + RIGHT("00" + LEFT([DateColumn],2),2) + "-" + RIGHT("00" + SUBSTRING([DateColumn],3,2),2))
Screenshot from SSIS expression tester tool
Here is your SSIS expression. Obviously, the assumption here is that day and month parts are always 2 digits:
(DT_DBDATE)(SUBSTRING(col,5,4) + "-" + SUBSTRING(col,1,2) + "-" + SUBSTRING(col,3,2))
I'd use a script component.
DateTime.ParseExact([datecol],"MMddyyyy", System.Globalization.CultureInfo.InvariantCulture);
I need to get the latest "version" of a Task object for a given objectUuid. The Task is identified by its objectUuid, taskName and createdTimestamp attributes.
I had the following HQL query:
select new list(te) from " + TaskEntity.class.getName() + " te
where
te.objectUuid = '" + domainObjectId + "' and
te.createdTimestamp = (
select max(te.createdTimestamp) from " + TaskEntity.class.getName() + " teSub
where teSub.objectUuid = te.objectUuid and teSub.taskName = te.taskName
)
which ran and produced the correct results on H2 (embedded) and MySQL.
However after installing this code in production to MS SQL Server I get the following error:
An aggregate may not appear in the WHERE clause unless it is in a
subquery contained in a HAVING clause or a select list, and the column
being aggregated is an outer reference.
I tried to rewrite the query but HQL doesn't seem to support subqueries properly. My latest attempt is something like:
select new list(te) from " + TaskEntity.class.getName() + " te
inner join (
select te2.objectUuid, te2.taskName, max(te2.createdTimestamp)
from " + TaskEntity.class.getName() + " te2
group by te2.objectUuid, te2.taskName
) teSub on
teSub.objectUuid = te.objectUuid and teSub.taskName = te.taskName
where
te.objectUuid = '" + domainObjectId + "'
but of course it fails at the "(" after the join statement.
Since this is a very frequent type of query I cannot believe there is no solution that works with HQL+MSSQL.
Uh-oh. Can this be a typo?
max(teSub.createdTimestamp)
instead of
max(te.createdTimestamp)
in the subquery.
Have a date field that I need to convert from mddyy or mmddyy (numeric format) when the month is >=10 to a mmddyy date format. Have yet to find any answers on the web.
You can use FORMAT:-
FORMAT ( table.myDateColumn, 'd', 'en-US' )
Edit:-
Try this:-
SELECT LTRIM(STR(MONTH(DateColumn)))+'/'+LTRIM(STR(DAY(DateColumn)))+'/'+STR(YEAR(DateColumn),4)
This gets it in the genearal appearance I sought:
select
CASE WHEN LEN(CONVERT(varchar(6), datefield)) < 6
THEN ('0' + LEFT(CONVERT(varchar(6), datefield),1) + '-' +
SUBSTRING(CONVERT(varchar(6), datefield),2,2) + '-' +
LEFT(CONVERT(varchar(7), datefield),4))
ELSE (LEFT(CONVERT(varchar(6), datefield),2) + '-' +
SUBSTRING(CONVERT(varchar(6), datefield),3,2) + '-' +
LEFT(CONVERT(varchar(7), datefield),4))
END as DesiredDate
My above solution technically does not work nor is consistent with SQL's system-wide setting in SQL Server - the "two-digit year cutoff". Typically SQL Server considers any date with a 2 digit year or value of 49 as 20xx year or 2049. Dates with a 2 digit year after that, say 70 get interpreted as 19xx (70 -> 1970). Thanks to marc_s for his post on the topic. My solution does not require a custom function or SQL Server setting change, and it does account for both one and two digit months. It is:
Select
Convert (datetime, --varchar depending on desired format--
CASE
WHEN LEN(CONVERT(varchar(6), dt.datefield)) < 6 and
SUBSTRING(CONVERT(varchar(6), dt.datefield),4,2) < 50
THEN ('0' + LEFT(CONVERT(varchar(6), dt.datefield),1)) + '-' +
SUBSTRING(CONVERT(varchar(6), dt.datefield),2,2) + '-' +
('20'+RIGHT(CONVERT(varchar(6), dt.datefield),2))
WHEN LEN(CONVERT(varchar(6), dt.datefield)) = 6 and
SUBSTRING(CONVERT(varchar(6), dt.datefield),5,2) < 50
THEN (LEFT(CONVERT(varchar(6), dt.datefield),2)) + '-' +
SUBSTRING(CONVERT(varchar(6), dt.datefield),2,2) + '-' +
('20'+RIGHT(CONVERT(varchar(6), dt.datefield),2))
WHEN LEN(CONVERT(varchar(6), dt.datefield)) < 6 and
SUBSTRING(CONVERT(varchar(6), dt.datefield),4,2) >= 50
THEN ('0' + LEFT(CONVERT(varchar(6), dt.datefield),1)) + '-' +
SUBSTRING(CONVERT(varchar(6), dt.datefield),5,2) + '-' +
('19'+RIGHT(CONVERT(varchar(6), dt.datefield),2))
WHEN LEN(CONVERT(varchar(6), dt.datefield)) = 6 and
SUBSTRING(CONVERT(varchar(6), dt.datefield),5,2) >= 50
THEN (LEFT(CONVERT(varchar(6), dt.datefield),2)) + '-' +
SUBSTRING(CONVERT(varchar(6), dt.datefield),2,2) + '-' +
('19'+(LEFT(CONVERT(varchar(7), dt.datefield),2)))
End) as MaintDate
FROM
dbo.date_table dt