I have model I created on the fly for peewee. Something like this:
class TestTable(PeeweeBaseModel):
whencreated_dt = DateTimeField(null=True)
whenchanged = CharField(max_length=50, null=True)
I load data from a text file to a table using peewee, the column "whenchanged" contains all dates in a format of '%Y-%m-%d %H:%M:%S' as varchar column. Now I want to convert the text field "whenchanged" into a datetime format in "whencreated_dt".
I tried several things... I ended up with this:
# Initialize table to TestTable
to_execute = "table.update({table.%s : datetime.strptime(table.%s, '%%Y-%%m-%%d %%H:%%M:%%S')}).execute()" % ('whencreated_dt', 'whencreated')
which fails with a "TypeError: strptime() argument 1 must be str, not CharField": I'm trying to convert "whencreated" to datetime and then assign it to "whencreated_dt".
I tried a variation... following e.g. works without a hitch:
# Initialize table to TestTable
to_execute = "table.update({table.%s : datetime.now()}).execute()" % (self.name)
exec(to_execute)
But this is of course just the current datetime, and not another field.
Anyone knows a solution to this?
Edit... I did find a workaround eventually... but I'm still looking for a better solution... The workaround:
all_objects = table.select()
for o in all_objects:
datetime_str = getattr( o, 'whencreated' )
setattr(o, 'whencreated_dt', datetime.strptime(datetime_str, '%Y-%m-%d %H:%M:%S'))
o.save()
Loop over all rows in the table, get the "whencreated". Convert "whencreated" to a datetime, put it in "whencreated_dt", and save each row.
Regards,
Sven
Your example:
to_execute = "table.update({table.%s : datetime.strptime(table.%s, '%%Y-%%m-%%d %%H:%%M:%%S')}).execute()" % ('whencreated_dt', 'whencreated')
Will not work. Why? Because datetime.strptime is a Python function and operates in Python. An UPDATE query works in database-land. How the hell is the database going to magically pass row values into "datetime.strptime"? How would the db even know how to call such a function?
Instead you need to use a SQL function -- a function that is executed by the database. For example, Postgres:
TestTable.update(whencreated_dt=whenchanged.cast('timestamp')).execute()
This is the equivalent SQL:
UPDATE test_table SET whencreated_dt = CAST(whenchanged AS timestamp);
That should populate the column for you using the correct data type. For other databases, consult their manuals. Note that SQLite does not have a dedicated date/time data type, and the datetime functionality uses strings in the Y-m-d H:M:S format.
Related
I want to be able to add a timestamp the filename I'm writing to s3. So far I've been able to write files to AWS S3 using example below. Can someone guide me as to how do I go about putting datetime stamp in the file name?
copy into #s3bucket/something.csv.gz
from (select * from mytable)
file_format = (type=csv FIELD_OPTIONALLY_ENCLOSED_BY = '"' compression='gzip' )
single=true
header=TRUE;
Thanks in advance.
The syntax for defining a path inside of a stage or location portion of the COPY INTO statement does not allow for functions to dynamically define it in SQL.
However, you can use a stored procedure to accomplish building dynamic queries, using JavaScript Date APIs and some string formatting.
Here's a very trivial example for your use-case, with some code adapted from another question:
CREATE OR REPLACE PROCEDURE COPY_INTO_PROCEDURE_EXAMPLE()
RETURNS VARIANT
LANGUAGE JAVASCRIPT
EXECUTE AS CALLER
AS
$$
var rows = [];
var n = new Date();
// May need refinement to zero-pad some values or achieve a specific format
var datetime = `${n.getFullYear()}-${n.getMonth() + 1}-${n.getDate()}-${n.getHours()}-${n.getMinutes()}-${n.getSeconds()}`;
var st = snowflake.createStatement({
sqlText: `COPY INTO '#s3bucket/${datetime}_something.csv.gz' FROM (SELECT * FROM mytable) FILE_FORMAT=(TYPE=CSV FIELD_OPTIONALLY_ENCLOSED_BY='"' COMPRESSION='gzip') SINGLE=TRUE HEADER=TRUE;`
});
var result = st.execute();
result.next();
rows.push(result.getColumnValue(1))
return rows;
$$
To execute, run:
CALL COPY_INTO_PROCEDURE_EXAMPLE();
The above is missing perfected date format handling (zero padding months, days, hours, minutes, seconds), error handling (if the COPY INTO fails), parameterisation of input query, etc. but it should give a general idea on how to achieve this.
As Sharvan Kumar suggests above, Snowflake now support this:
-- Partition the unloaded data by date and hour. Set ``32000000`` (32 MB) as the upper size limit of each file to be generated in parallel per thread.
copy into #%t1
from t1
partition by ('date=' || to_varchar(dt, 'YYYY-MM-DD') || '/hour=' || to_varchar(date_part(hour, ts))) -- Concatenate labels and column values to output meaningful filenames
file_format = (type=parquet)
max_file_size = 32000000
header=true;
list #%t1
This features is not supported yet in snowflake, however will be coming soon.
I have a pair of databases (one mssql and one oracle), ran by different teams. Some data are now being synchronized regularily by a stored procedure in the mssql table. This stored procedure is calling a very large
MERGE [mssqltable].[Mytable] as s
USING THEORACLETABLE.BLA as t
ON t.[R_ID] = s.[R_ID]
WHEN MATCHED THEN UPDATE SET [Field1] = s.[Field1], ..., [Brokenfield] = s.[BrokenField]
WHEN NOT MATCHED BY TARGET THEN
... another big statement
Field Brokenfield was a numeric one until today, and could take value NULL, 0, 1, .., 24
Now, the oracle team introduced a breaking change today for some reason, changed the type of the column to string and now has values NULL, "", "ALFA", "BRAVO"... in the column. Of course, the sync got broken.
What is the easiest way to fix the sync here? I (Mysql team lead, frontend expert but not so in databases) would usually apply one of our database expert guys here, but all of them are now ill, and the fix must go online today....
I thought of a stored procedure like CONVERT_BROKENFIELD_INT_TO_STRING or so, based on some switch-case, which could be called in that merge statement, but not sure how to do that.
Edit/Clarification:
What I need is a way to make a chunk of SQL code (stored procedure), taking an input of "ALFA" and returning 1, "BRAVO" -> 2, etc. and which can be reused, to avoid writing huge ifs in more then one place.
If you can not simplify the logic for correct values the way #RichardHansell desribed, you can create a crosswalk table for BrokenField to the correct values. Then you can use a common table expression or subquery with a left join to that crosswalk to use in the merge.
create table dbo.BrokenField_Crosswalk (
BrokenField varchar(32) not null primary key
, CorrectedValue int
);
insert into dbo.BrokenField_Crosswalk (BrokenField,CorrectedValue) values
('ALFA', 1)
, ('ALPHA', 1)
, ('BRAVO', 2)
...
go
And your code for the merge would look something like this:
;with cte as (
select o.R_ID
, o.Field1
, BrokenField = cast(isnull(c.CorrectedValue,o.BrokenField) as int)
....
from oracle_table.bla as o
left join dbo.BrokenField_Crosswalk as c
)
merge into [mssqltable].[Mytable] t
using cte as s
on t.[R_ID] = s.[R_ID]
when matched
then update set
[Field1] = s.[Field1]
, ...
, [Brokenfield] = s.[BrokenField]
when not matched by target
then
If they are using names with a letter at the start that goes in a sequence:
A = 1
B = 2
C = 3
etc.
Then you could do something like this:
MERGE [mssqltable].[Mytable] as s
USING THEORACLETABLE.BLA as t
ON t.[R_ID], 1)) - ASCII('A') + 1 = s.[R_ID]
WHEN MATCHED THEN UPDATE SET [Field1] = s.[Field1], ..., [Brokenfield] = s.[BrokenField]
WHEN NOT MATCHED BY TARGET THEN
... another big statement
Edit: but actually I re-read your question and you are talking about [Brokenfield] being the problem column, so my solution wouldn't work.
I don't really understand now, as it seems as though the MERGE statement is updating the oracle table with numbers, so surely you need the mapping to work the other way, i.e. 1 -> ALFA, 2 -> BETA, etc.?
SQL - How can I return a value from a different table base on a parameter
First time poster, long time reader:
I am using a custom Excel function that allows be to pass parameters and build a SQL string that returns a value. This is working fine. However, I would like to choose among various tables based on the parameters that are passed.
At the moment I have two working functions with SQL statements look like this:
_______FUNCTION ONE________
<SQLText>
SELECT PRODDTA.TABLE1.T1DESC as DESCRIPTION
FROM PRODDTA.TABLE1
WHERE PRODDTA.TABLE1.T1KEY = '&PARM02'</SQLText>
_______FUNCTION TWO________
<SQLText>
SELECT PRODDTA.TABLE2.T2DESC as DESCRIPTION
FROM PRODDTA.TABLE2
WHERE PRODDTA.TABLE2.T2KEY = '&PARM02'</SQLText>
So I am using IF logic in Excel to check the first parameter and decide which function to use.
It would be much better if I could do a single SQL statement that could pick the right table based on the 1st parameter. Logically something like this:
_______FUNCTIONS COMBINED________
IF '&PARM02' = “A” THEN
SELECT PRODDTA.TABLE1.T1DESC as DESCRIPTION
FROM PRODDTA.TABLE1
WHERE PRODDTA.TABLE1.T1KEY = '&PARM02'
ELSE IF '&PARM02' = “B” THEN
SELECT PRODDTA.TABLE2.T2DESC as DESCRIPTION
FROM PRODDTA.TABLE2
WHERE PRODDTA.TABLE2.T2KEY = '&PARM02'
ELSE
DESCRIPTION = “”
Based on another post Querying different table based on a parameter I tried this exact syntax with no success
<SQLText>
IF'&PARM02'= "A"
BEGIN
SELECT PRODDTA.F0101.ABALPH as DESCRIPTION
FROM PRODDTA.F0101
WHERE PRODDTA.F0101.ABAN8 = '&PARM02'
END ELSE
BEGIN
SELECT PRODDTA.F4801.WADL01 as DESCRIPTION
FROM PRODDTA.F4801
WHERE PRODDTA.F4801.WADOCO = '&PARM02'
END</SQLText>
You could try using a JOIN statement.
http://www.sqlfiddle.com/#!9/23461d/1
Here is a fiddle showing two tables.
The following code snip will give you the values from both tables, using the Key as the matching logic.
SELECT Table1.description, Table1.key, Table2.description
from Table1
Join Table2 on Table1.key = Table2.key
Here's one way to do it. If PARM03='Use Table1' then the top half of the union will return records and vice versa. This won't necessarily product good performance though. You should consider why you are storing data in this way. It looks like you are partitioning data across different tables which is a bad idea.
SELECT PRODDTA.TABLE1.T1DESC as DESCRIPTION
FROM PRODDTA.TABLE1
WHERE PRODDTA.TABLE1.T1KEY = '&PARM02'
AND &PARM03='Use Table1'
UNION ALL
SELECT PRODDTA.TABLE2.T2DESC as DESCRIPTION
FROM PRODDTA.TABLE2
WHERE PRODDTA.TABLE2.T2KEY = '&PARM02'</SQLText>
AND &PARM03='Use Table2'
I have read all the related entries (here and at other sites) and have not found my situation.
I have a table (MyTable) with several date fields (MyDateField1, MyDateField2, etc) plus other fields not pertinent to this matter. All the date fields allow null values.
My application's tableadapter's insert method invokes the following stored procedure:
INSERT INTO MyTable VALUES (#MyDateField1Value, #MyDateField2Value,..., <other fields>);
using this VB code (dv is a dataview):
NewRow = dv.AddNew
NewRow(MyDate1Field) = DateValue1 <some date value taken off a window>
NewRow(MyDate2Field) = DateValue2 <some date value taken off a window>
....
NewRow.EndEdit()
MyTableTableAdapter.Insert(DateValue1, DateValue2, <etc>)
This works fine when none of the date fields is null. However, if I set:
NewRow(MyDate1Field) = DBNull.Value
I get the error message that DBNull.Value cannot be converted to a date value. How can I get a null value into MyDateField1, MyDateField2, etc.
The answer is to pass Nothing. Very simple - too bad it was so hard to find.
This is a followup post to this post. I am writing an accounting system backed by an h2 database. The tree of accounts is stored in the ACCOUNTS table, with the PARENT_ID column storing the links in the tree.
To get the path to a given node in the tree, I have the following stored procedure:
public static Long[] getAncestorPKs(Long id)
whose job is to produce an array of integers, being the PARENT_ID values between the given node and the root of the tree. Let's imagine it is defined like this (because I have tried this and I get the same error):
public static Long[] getAncestorPKs(Long id)
{
return new Long[]{new Long(1), new Long(2), new Long(3)};
}
It is properly registered in the database and I can call it from within a SQL query. My problem is that h2 seems to be unable to deal with the return value: if I use it like this:
SELECT ID FROM ACCOUNTS WHERE ID IN (ANCESTOR_PKS(5))
then I get the following error:
Data conversion error converting "(1, 2, 3)"; SQL statement:
SELECT ID FROM ACCOUNTS WHERE ID IN (ANCESTOR_PKS(5)) [22018-167]
If, instead, I send the following to the database:
SELECT ID FROM ACCOUNTS WHERE ID IN (1, 2, 3)
I get back a result set with 3 rows, containing the three integers (exactly what I expect).
I really can't see what is the problem here! I am returning an array of Longs, which are to be used in comparing against a column which contains BIGINTS. Why is h2 refusing to convert this array? I have tried making the return value be Object[], because the h2 documentation is not entirely clear whether this is required on the return side as well as on the call side, but that makes no difference at all. I'm just banging my head against a brick wall here. This ain't rocket science! Surely someone has written similar code before?
Many thanks in advance, before I go mad!
If the method returns an array of objects, then for the database this is one value of data type ARRAY. And not a table with 3 rows. But of course you don't use the data type ARRAY in your table, you use INT or BIGINT. So your query is incorrect.
Either the method needs to return a ResultSet, or you need to convert the array value to a table. To do that, you could use the function TABLE(..) as follows:
select x from table(x bigint = getAncestorPKs(1));
So what you could do is:
drop table accounts;
create table accounts(id int);
insert into accounts values(1), (2), (10), (20);
drop alias getAncestorPKs;
create alias getAncestorPKs as 'Long[] getAncestorPKs(Long id) {
return new Long[]{new Long(1), new Long(2), new Long(3)};
}';
select * from accounts where id in
(select x from table(x bigint = getAncestorPKs(1)));