Note: this is about the same situation as Unload data to a stage with partitions "flattened" One of these might be an X-Y question. I also asked on the Snowflake Community forum: https://community.snowflake.com/s/question/0D53r0000BZbU70CQF/prevent-filename-conflicts-when-unloading-files-with-partition-by.
I have some files in a Snowflake stage called mystage:
+------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------+
| name | size | md5 | last_modified |
|------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------|
| mystage/00/data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_4_0.snappy.parquet | 8645856 | 2e3fe0e95b3b4d39f1b2c2cd2a5e5fd3 | Tue, 7 Jun 2022 13:51:40 GMT |
| mystage/01/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_1_0.snappy.parquet | 8743968 | 4d59bb8cfa355c8d55f05b88aa91002f | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/02/data_01a4c9ff-0602-1baa-0070-30030078eb32_1412_1_0.snappy.parquet | 9719360 | c723aae191fb46c1c83eb660e3dd41cc | Tue, 7 Jun 2022 13:51:43 GMT |
| mystage/03/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_2_0.snappy.parquet | 7786640 | b63ed8a15f18cd97ed8aaeb4fea89dee | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/04/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_1_0.snappy.parquet | 9249120 | 47471b0b0fb72b8d9ae5982176f29af1 | Tue, 7 Jun 2022 13:52:00 GMT |
| mystage/05/data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet | 9385936 | 18b286beb69b876aa3b4e08e857f69e7 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/06/data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet | 9187536 | 2dd0e99c7fa22fe42c624e4298c9d8f1 | Tue, 7 Jun 2022 13:51:58 GMT |
| mystage/07/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_4_0.snappy.parquet | 8804096 | aaf089af92251957502665fb7ab144f5 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/08/data_01a4c9ff-0602-1baa-0070-30030078eb32_312_5_0.snappy.parquet | 9214480 | 7969198b4e826d791e6f99455d01d72c | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/09/data_01a4c9ff-0602-1baa-0070-30030078eb32_512_5_0.snappy.parquet | 8986608 | 159c831c75439dcdc2743dabd91a66bb | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/10/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_7_0.snappy.parquet | 9063392 | 5410728fe0723f350322075f89a7e37f | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/11/data_01a4c9ff-0602-1baa-0070-30030078eb32_912_7_0.snappy.parquet | 9197600 | 68f6c4f81c3d3890a9eb269bfcde97ec | Tue, 7 Jun 2022 13:51:39 GMT |
| mystage/12/data_01a4c9ff-0602-1baa-0070-30030078eb32_1512_0_0.snappy.parquet | 9613440 | 124fdd720b104d4e6a119140409454a3 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/13/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_6_0.snappy.parquet | 8524336 | 17ace7467c50a89d6558a263b40f4036 | Tue, 7 Jun 2022 13:51:40 GMT |
| mystage/14/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_5_0.snappy.parquet | 8830192 | 83bf9a6e027e812583b69d735adda12c | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/15/data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_3_0.snappy.parquet | 9051568 | a75520c1713b6e4cc7a3f28ac4589f42 | Tue, 7 Jun 2022 13:51:40 GMT |
+------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------+
I tried to download these files:
GET #mystage file:///data;
But the files were downloaded without the 00, 01, etc. directory structure. At least two of the files have non-unique names (in this case data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet), so this means I can't actually download all of the files!
In this particular case, it causes a "file not found" error, but in general I want some way to preserve the directory structure when downloading. Is there some other Snowflake command I can use for this purpose?
Related
Note: this is about the same situation as How to download files with GET and preserve directory structure? One of these might be an X-Y question. I also asked on the Snowflake Community forum: https://community.snowflake.com/s/question/0D53r0000BZbU70CQF/prevent-filename-conflicts-when-unloading-files-with-partition-by.
I ran COPY INTO #mystage with PARTITION BY and SINGLE = false, creating several files in a stage with names like this:
+------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------+
| name | size | md5 | last_modified |
|------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------|
| mystage/00/data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_4_0.snappy.parquet | 8645856 | 2e3fe0e95b3b4d39f1b2c2cd2a5e5fd3 | Tue, 7 Jun 2022 13:51:40 GMT |
| mystage/01/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_1_0.snappy.parquet | 8743968 | 4d59bb8cfa355c8d55f05b88aa91002f | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/02/data_01a4c9ff-0602-1baa-0070-30030078eb32_1412_1_0.snappy.parquet | 9719360 | c723aae191fb46c1c83eb660e3dd41cc | Tue, 7 Jun 2022 13:51:43 GMT |
| mystage/03/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_2_0.snappy.parquet | 7786640 | b63ed8a15f18cd97ed8aaeb4fea89dee | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/04/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_1_0.snappy.parquet | 9249120 | 47471b0b0fb72b8d9ae5982176f29af1 | Tue, 7 Jun 2022 13:52:00 GMT |
| mystage/05/data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet | 9385936 | 18b286beb69b876aa3b4e08e857f69e7 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/06/data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet | 9187536 | 2dd0e99c7fa22fe42c624e4298c9d8f1 | Tue, 7 Jun 2022 13:51:58 GMT |
| mystage/07/data_01a4c9ff-0602-1baa-0070-30030078eb32_712_4_0.snappy.parquet | 8804096 | aaf089af92251957502665fb7ab144f5 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/08/data_01a4c9ff-0602-1baa-0070-30030078eb32_312_5_0.snappy.parquet | 9214480 | 7969198b4e826d791e6f99455d01d72c | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/09/data_01a4c9ff-0602-1baa-0070-30030078eb32_512_5_0.snappy.parquet | 8986608 | 159c831c75439dcdc2743dabd91a66bb | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/10/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_7_0.snappy.parquet | 9063392 | 5410728fe0723f350322075f89a7e37f | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/11/data_01a4c9ff-0602-1baa-0070-30030078eb32_912_7_0.snappy.parquet | 9197600 | 68f6c4f81c3d3890a9eb269bfcde97ec | Tue, 7 Jun 2022 13:51:39 GMT |
| mystage/12/data_01a4c9ff-0602-1baa-0070-30030078eb32_1512_0_0.snappy.parquet | 9613440 | 124fdd720b104d4e6a119140409454a3 | Tue, 7 Jun 2022 13:51:44 GMT |
| mystage/13/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_6_0.snappy.parquet | 8524336 | 17ace7467c50a89d6558a263b40f4036 | Tue, 7 Jun 2022 13:51:40 GMT |
| mystage/14/data_01a4c9ff-0602-1baa-0070-30030078eb32_112_5_0.snappy.parquet | 8830192 | 83bf9a6e027e812583b69d735adda12c | Tue, 7 Jun 2022 13:51:41 GMT |
| mystage/15/data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_3_0.snappy.parquet | 9051568 | a75520c1713b6e4cc7a3f28ac4589f42 | Tue, 7 Jun 2022 13:51:40 GMT |
+------------------------------------------------------------------------------------------------------------+---------+----------------------------------+------------------------------+
Note that there is some directory structure here. The directories 00, 01, etc. were created by the COPY INTO command.
As I described in my other question (How to download files with GET and preserve directory structure?), there doesn't seem to be a way to download these files in a way that preserves the directory structure. This is a problem because at least two of the files have identical "tail" names (data_01a4c9ff-0602-1baa-0070-30030078eb32_612_2_0.snappy.parquet), so downloading them all to the same directory causes problems.
As a workaround for the issue described in my linked question, is there any way to change how these file names are created, such that the directory names 00, 01, etc. are converted to filename prefixes? That is, I would want 00_data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_4_0.snappy.parquet instead of 00/data_01a4c9ff-0602-1baa-0070-30030078eb32_1112_4_0.snappy.parquet.
I'm using Vue. Lets say I have a database table with the historical price of a few kinds of fruits for the last few years, so: fruit, year and price columns.
| fruit | year | price |
|--------|------|-------|
| apple | 2018 | 52 |
| apple | 2019 | 57 |
| apple | 2020 | 56 |
| apple | 2021 | 50 |
| banana | 2018 | 25 |
| banana | 2019 | 26 |
| banana | 2021 | 28 |
| pear | 2018 | 61 |
| pear | 2019 | 65 |
| pear | 2020 | 67 |
| pear | 2021 | 64 |
Now I want to create a html table which has fruit names on one axis and years on the other, and the cells contain the price for the given fruit / year combination as below. Some combinations might be missing from the data.
What features and template syntax you'd use? Please do not suggest tranforming the raw data: it comes straight from a database and there will be many tables like this, and I need a generic solution.
| | 2018 | 2019 | 2020 | 2021 |
|--------|------|------|------|------|
| apple | 52 | 57 | 56 | 50 |
| banana | 25 | 26 | n/a | 28 |
| pear | 61 | 65 | 67 | 64 |
I'm looking for elegant "vue-like" solutions. For now I created getRows(), getColumns() functions which collect all possible row and column values and then a getCell(col, row) function to pick up the right value from the dataset - but this might force Vue to rebuild the display more than optimal times when I edit the underlying data.
The broader question is how you work with relational data in Vue, because this is just the basic example, normally the name of the fruit would come from another base table...
I have a panel data set (the identifiers are the variables of cc and year).
I have all the values for the Y variable but I have only the initial (X_{1996}) values of the X variable.
I would like to replace the missing values of X with this pattern:
X_{1997} = Y_{1996} + 0.95*X_{1996}
X_{1998} = Y_{1997} + 0.95*X_{1997}
.
.
.
X_{2015} = Y_{2014} + 0.95*X_{2014}
I would like to iteratively replace X starting from 1997, that is using the last generated value.
Is it possible to write a loop for this purpose?
Example data:
clear
input str3 cc year X Y
"ABW" 1996 467.5953 28.46004
"ABW" 1997 . 30.23142
"ABW" 1998 . 31.70577
"ABW" 1999 . 30.870903
"ABW" 2000 . 25.495903
"ABW" 2001 . 23.71254
"ABW" 2002 . 26.906475
"ABW" 2003 . 29.215036
"ABW" 2004 . 28.77914
"ABW" 2005 . 33.581017
"ABW" 2006 . 35.16129
"ABW" 2007 . 32.51442
"ABW" 2008 . 33.251934
"ABW" 2009 . 30.29287
"ABW" 2010 . 28.931993
"ABW" 2011 . 28.5057
"ABW" 2012 . 26.647564
"ABW" 2013 . 23.999134
"ABW" 2014 . 22.917984
"ABW" 2015 . 21.481943
"AFG" 1996 362.353 11.51442
"AFG" 1997 . 11.557285
"AFG" 1998 . 12.19376
"AFG" 1999 . 14.557285
"AFG" 2000 . 12.906254
"AFG" 2001 . 13.156855
"AFG" 2002 . 11.557285
"AFG" 2003 . 15.758896
"AFG" 2004 . 19.005346
"AFG" 2005 . 23.193766
"AFG" 2006 . 26.407976
"AFG" 2007 . 20.068926
"AFG" 2008 . 19.021326
"AFG" 2009 . 17.991436
"AFG" 2010 . 17.959072
"AFG" 2011 . 16.674591
"AFG" 2012 . 16.906254
"AFG" 2013 . 17.343513
"AFG" 2014 . 18.156855
"AFG" 2015 . 19.454687
If you want to calculate the missing values of X by using the last generated observation, then you can do so as follows:
bysort cc (year): replace X = Y[_n-1] + 0.95 * X[_n-1] if _n > 1
list, sepby(cc)
+----------------------------------+
| cc year X Y |
|----------------------------------|
1. | ABW 1996 467.5953 28.46004 |
2. | ABW 1997 472.6756 30.23142 |
3. | ABW 1998 479.2732 31.70577 |
4. | ABW 1999 487.0153 30.8709 |
5. | ABW 2000 493.5355 25.4959 |
6. | ABW 2001 494.3546 23.71254 |
7. | ABW 2002 493.3494 26.90648 |
8. | ABW 2003 495.5884 29.21504 |
9. | ABW 2004 500.024 28.77914 |
10. | ABW 2005 503.802 33.58102 |
11. | ABW 2006 512.1929 35.16129 |
12. | ABW 2007 521.7445 32.51442 |
13. | ABW 2008 528.1717 33.25193 |
14. | ABW 2009 535.015 30.29287 |
15. | ABW 2010 538.5571 28.93199 |
16. | ABW 2011 540.5613 28.5057 |
17. | ABW 2012 542.0389 26.64756 |
18. | ABW 2013 541.5845 23.99913 |
19. | ABW 2014 538.5045 22.91798 |
20. | ABW 2015 534.4972 21.48194 |
|----------------------------------|
21. | AFG 1996 362.353 11.51442 |
22. | AFG 1997 355.7498 11.55729 |
23. | AFG 1998 349.5196 12.19376 |
24. | AFG 1999 344.2373 14.55729 |
25. | AFG 2000 341.5828 12.90625 |
26. | AFG 2001 337.4099 13.15685 |
27. | AFG 2002 333.6962 11.55729 |
28. | AFG 2003 328.5687 15.7589 |
29. | AFG 2004 327.8992 19.00535 |
30. | AFG 2005 330.5096 23.19377 |
31. | AFG 2006 337.1778 26.40798 |
32. | AFG 2007 346.7269 20.06893 |
33. | AFG 2008 349.4595 19.02133 |
34. | AFG 2009 351.0078 17.99144 |
35. | AFG 2010 351.4489 17.95907 |
36. | AFG 2011 351.8355 16.67459 |
37. | AFG 2012 350.9183 16.90625 |
38. | AFG 2013 350.2787 17.34351 |
39. | AFG 2014 350.1082 18.15685 |
40. | AFG 2015 350.7597 19.45469 |
+----------------------------------+
I'm working in postgres 9.6 and still getting my head around json
i have a column with a json object that is an array of numbers that represent recurrence frequency and the days of the week.
{"every":"1","weekdays":["1"]}
{"every":"1","weekdays":["1","3"]}
{"every":"1","weekdays":["1","2","3","4","5"]}
ROW1 -[1] : MON
ROW2 -[1,3] : MON , WED
ROW3 -[1,2,3,4,5] : MON , TUE , WED , THU , FRI
I want to expand these into columns such that:
|ROW- |MON | TUE| WED|THU|FRI|
------------------------------
|ROW1 |Y |N |N |N |N |
|ROW2 |Y |N |Y |N |N |
|ROW3 |Y |Y |Y |Y |Y |
I can get the elements out using jsonb_array_elements(pattern) but then what?
i thought to use the 'contains' expression to build each column
pattern #> '{1}', pattern #> '{2}' etc but I couldn't construct an object that would give a hit
Example data:
create table my_table(id serial primary key, pattern jsonb);
insert into my_table (pattern) values
('{"weekdays": [1]}'),
('{"weekdays": [1, 3]}'),
('{"weekdays": [1, 2, 3, 4, 5]}');
You can use the operator #> in this way:
select
id,
pattern->'weekdays' #> '[1]' as mon,
pattern->'weekdays' #> '[2]' as tue,
pattern->'weekdays' #> '[3]' as wed,
pattern->'weekdays' #> '[4]' as thu,
pattern->'weekdays' #> '[5]' as fri
from my_table
id | mon | tue | wed | thu | fri
----+-----+-----+-----+-----+-----
1 | t | f | f | f | f
2 | t | f | t | f | f
3 | t | t | t | t | t
(3 rows)
It seems i was on the right track with 'contains' but i had confused myself about what was in the array. I should have been looking for a string not a number
, bookings.pattern->'weekdays' #> '"1"'::jsonb
Thanks to Pitto for the prompt to put the outer json in the question which made it obvious
I haven't been able to find a function or query that would show me a PID that owns the connection.
these tables were created on the 20th and I haven't been able to find which open database connection owns it.
pg_temp_57.tmplb1604202242425
pg_temp_57.tmplb1604202239538
pg_temp_60.tmplb1604202242006
mainDB=# select relname,relcreatetime,relfrozenxid from pg_class where relname ~ 'tmplb16042022';
relname | relcreatetime | relfrozenxid
----------------------------------+---------------+--------------
tmplb1604202242425 | 1461206562 | 372689
tmplb1604202239538 | 1461206393 | 372487
tmplb1604202242006 | 1461206520 | 372626
(6 rows)
[root#postgreDB]# date -d #1461206562
Wed Apr 20 22:42:42 EDT 2016
[root#postgreDB]# date -d #1461206520
Wed Apr 20 22:42:00 EDT 2016
[root#postgreDB]# date -d #1461206393
Wed Apr 20 22:39:53 EDT 2016
[root#postgreDB]# date -d #1461206562
Wed Apr 20 22:42:42 EDT 2016