I am copying the data into snowflake table which has three columns: ID, DATA and ETL_LOAD_TIMESTAMP.
I have a column ETL_LOAD_TIMESTAMP in snowflake of type TIMESTAMP_TZ(9) and I have set its default value as CURRENT_TIMESTAMP().
I get my data from a CSV file, which is of type:
ID, DATA
1, Dummy
I download the csv file at tmpdir location on local. I load the data of this csv into snowflake as:
create_cmd = "CREATE TEMPORARY STAGE teamp123 COMMENT = 'TEMPORARY STAGE FOR TEST_TABLE1 DATA LOAD'"
self.connection.execute("ALTER SESSION SET TIMEZONE = 'UTC';")
self.connection.execute(create_cmd)
self.connection.execute(f"put file://tmpdir/* #temp123 PARALLEL=8")
self.connection.execute("COPY INTO TEST_TABLE1 FROM #temp123 PURGE = TRUE FILE_FORMAT = (TYPE = 'CSV' field_delimiter = ',' FIELD_OPTIONALLY_ENCLOSED_BY = '\"' ESCAPE_UNENCLOSED_FIELD = None error_on_column_count_mismatch=false SKIP_HEADER = 1)")
I get the values of ID and Data but the ETL_LOAD_TIMESTAMP is null.
How do I modify this copy command so that I get the default value of ETL_LOAD_TIMESTAMP which is current timestamp instead of null?
you can use default current_timestamp() while defining datatypes or explicit to_timestamp
https://docs.snowflake.com/en/user-guide/data-load-transform.html#current-time-current-timestamp-default-column-values
I am trying to load a CSV file into Snowflake. The sample format of the input csv table in s3 location is as follows (with 2 columns: ID, Location_count):
Input csv table
I need to transform it in the below format:(with 3 columns:ID, Location, Count)
Output csv table
However when I am trying to load the input file using the following query after creating database, external stage and file format, it returns LOAD_FAILED
create or replace table table_name
(
id integer,
Location_count variant
);
select parse_json(Location_count) as c;
list #stage_name;
copy into table_name from #stage_name file_format = 'fileformatname' on_error = 'continue';
you will probably need to parse_json that 2nd column as part of a copy-transformation. For example:
create file format myformat
type = csv field_delimiter = ','
FIELD_OPTIONALLY_ENCLOSED_BY = '"';
create or replace stage csv_stage file_format = (format_name = myformat);
copy into #csv_stage from
( select '1',
'{"SHS-TRN":654738,"PRN-UTN":78956,"NCT-JHN":96767}') ;
create or replace table blah (id integer, something variant);
copy into blah from (select $1, parse_json($2) from #csv_stage);
I'm new to Snowflake,
I have created a table using DDL into a database called STOCKS_EOD and schema STOCKS
CREATE TABLE DIM_COMPANY_STOCK(
COMPANY_STOCK_SK int NOT NULL,
STOCK_TICKER STRING NOT NULL,
SECTOR STRING NOT NULL,
INDUSTRY STRING NOT NULL,
COMPANY_NAME STRING NULL
);
create or replace file format csvformat
type = 'CSV'
field_delimiter = '|';
Trying to load a staged file into table using the command, results in error : Numeric value '0xFF0xFE10x00' is not recognized
copy into STOCKS_EOD.STOCKS.DIM_COMP_STOCK
from #fileLanding/DIM_COMPANY_STOCK_Export_All.csv.gz
file_format = (format_name = csvformat)
on_error = 'skip_file';
Sample Data:
1|A|CAPITAL GOODS|ELECTRICAL PRODUCTS|AGILENT TECHNOLOGIES
2|AA|CAPITAL GOODS|METAL FABRICATIONS|ALCOA CORP
3|AAN|TECHNOLOGY|DIVERSIFIED COMMERCIAL SERVICES|AARON'S INC
I need to perform location based search based on latitude and longitude pairs, to find out the near by locations of stores within a radius/distance of given KMs.
I am to use (for certain) Oracle's SDO_GEOMETRY for searching the locations.
Also, the table structure are parent-child based such that store address (zip, lat/long) is in parent table but store details (name, contact, etc) are in the child table.
This is to make sure that we do not have redundant data (as multiple store can have same lat/long and by some dark magic same address)
I have the following scenario (tables provided):
Version - Oracle Database 12c Enterprise Edition Release 12.2.0.1.0
Table: STORE_LOCATION
CREATE TABLE STORE_LOCATE
(
ID NUMBER DEFAULT STORE_LOCATE.nextval,
POSTAL_CODE VARCHAR2(18) NOT NULL,
ADDRESS VARCHAR2(382) NOT NULL,
GEO_LOCATION SDO_GEOMETRY NOT NULL
);
Table: STORE_DETAIL
CREATE TABLE STORE_DETAIL
(
ID NUMBER DEFAULT STORE_DETAIL_SEQ.nextval,
STORE_CODE VARCHAR2(20) NOT NULL,
STORE_NAME VARCHAR2(150) NOT NULL,
IS_ACTIVE NUMBER(3) DEFAULT 0 NOT NULL,
fk_store_locate_id INT NOT NULL. -- FK to ID of parent
);
I have inserted data in parent as:
INSERT INTO STORE_LOCATE (ZIP, ADDRESS, GEO_LOCATION) VALUES
('567875', '84 Paddar Road',
SDO_GEOMETRY(2001, 8307, SDO_POINT_TYPE (80.34234431,26.12354422, NULL), NULL, NULL));
Likewise, child table has entries:
INSERT INTO STORE_DETAIL (STORE_CODE, STORE_NAME, FK_STORE_LOCATION_ID) VALUES
('ST01', 'STORE 1', 1);
So, now when a user shares his location to me (lat/long), I need to search all nearby locations of stores within a specified radius.
I tried the following query, but I am getting error with this:
SELECT
s.store_code,
s.store_name,
loc.zip,
loc.address,
loc.geo_location
FROM store_detail s,
store_locate loc
where MDSYS.SDO_WITHIN_DISTANCE(loc.geo_location,
(MDSYS.SDO_GEOMETRY(2001, 8307,
MDSYS.SDO_POINT_TYPE(80.21456732,26.23117864, NULL) ,NULL, NULL)),
'distance=1000 unit=KM') = 'TRUE';
Getting the below error:
ORA-29900: operator binding does not exist
ORA-06553: PLS-306: wrong number or types of arguments in call to 'SDO_WITHIN_DISTANCE'
29900. 00000 - "operator binding does not exist"
*Cause: There is no binding for the current usage of the operator.
*Action: Change the operator arguments to match any of the existing
bindings or add a new binding to the operator.
I am breaking my head around this for a while now but in vain.
Ref:
https://issues.redhat.com/browse/TEIID-751?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
https://docs.oracle.com/cd/E17781_01/appdev.112/e18750/xe_locator.htm#XELOC562
Q's:
Is this the correct way to perform location search on Oracle SQL?
Is there any other way if the above is incorrect?
There are a number of syntax errors in your code (POSTAL_CODE called ZIP, etc)
Here is the corrected example (I also removed the sequences for simplicity)
CREATE TABLE STORE_LOCATE (
ID NUMBER primary key,
POSTAL_CODE VARCHAR2(18) NOT NULL,
ADDRESS VARCHAR2(382) NOT NULL,
GEO_LOCATION SDO_GEOMETRY NOT NULL
);
CREATE TABLE STORE_DETAIL (
ID NUMBER primary key,
STORE_CODE VARCHAR2(20) NOT NULL,
STORE_NAME VARCHAR2(150) NOT NULL,
IS_ACTIVE NUMBER(3) DEFAULT 0 NOT NULL,
fk_store_locate_id INT NOT NULL references STORE_LOCATE
);
INSERT INTO STORE_LOCATE (ID, POSTAL_CODE, ADDRESS, GEO_LOCATION) VALUES (1, '567875', '84 Paddar Road', SDO_GEOMETRY(2001, 8307, SDO_POINT_TYPE (80.34234431,26.12354422, NULL), NULL, NULL));
INSERT INTO STORE_DETAIL (ID, STORE_CODE, STORE_NAME, FK_STORE_LOCATE_ID) VALUES (1001,'ST01', 'STORE 1', 1);
commit;
Here is running your query:
SELECT
s.store_code,
s.store_name,
loc.postal_code,
loc.address,
loc.geo_location
FROM store_detail s, store_locate loc
where SDO_WITHIN_DISTANCE(
loc.geo_location,
SDO_GEOMETRY(2001, 8307,
SDO_POINT_TYPE(80.21456732,26.23117864, NULL),NULL, NULL
),
'distance=1000 unit=KM'
) = 'TRUE';
STORE_ STORE_NAME POSTAL ADDRESS GEO_LOCATION(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES
------ ---------- ------ -------------------- ----------------------------------------------------------------------------------
ST01 STORE 1 567875 84 Paddar Road SDO_GEOMETRY(2001, 8307, SDO_POINT_TYPE(80.3423443, 26.1235442, NULL), NULL, NULL)
1 row selected.
The error you get probably means Oracle Spatial is not or incorrectly installed in your database. What exact database version do you use ?
IMPORTANT NOTE:
Those queries really need a spatial index on the GEO_LOCATION column. Without one, the query is rejected in versions up to 12.1. From 12.2 and later, the query will run, but will be very slow as soon as the number of locations goes above a few 100s. Once you go into the 100000 or more, it will be very slow.
See https://docs.oracle.com/en/database/oracle/oracle-database/19/spatl/indexing-querying-spatial-data.html#GUID-07129836-0DAE-4BCC-B290-942C456AE2EA for details
I am trying to load following data in hive table
data which i am loading
I used following table defination
CREATE EXTERNAL TABLE IF NOT EXISTS YOUTUBE_DATA (
VIDEO_ID STRING,
UPLOADER STRING,
INTERVAL INT,
CATEGORY STRING,
VIDEO_LEN INT,
VIEW_NO INT,
RATING FLOAT,
NO_COMMENTS INT,
RELATED_VIDEOS ARRAY
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
COLLECTION ITEMS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
location '/five';
LOAD DATA INPATH '/DATA/YOUTUBEDATA' OVERWRITE INTO TABLE YOUTUBE_DATA;
on running the query select * from youtube_data limit 10;
following was output.
output of query
can anyone please help with mistake which I am doing and with solution ??