I am using pentaho 7.1 and trying to load data to Snowflake . The SQl is running fine on Snowflake. But in Pentaho I am getting error :
Couldn't execute SQL: copy into "DEMO_DB"."PUBLIC"."STG_DIMACTIVITY" Table 'DEMO_DB.PUBLIC.STG_DIMACTIVITY' does not exist
SQL used is :
copy into "DEMO_DB"."PUBLIC"."STG_DIMACTIVITY"
from #my_s3_stage
FILES = ('MB_ACTIVITY.txt_0')
--pattern='.*MB_ACTIVITY.txt_0.*'
file_format = (type = csv field_delimiter = '|' skip_header = 1)
force=true;
Please let me know what i am missing here.Any help is much appreciated.
Related
create or replace stage AWS_OWNER1
url = 's3 url'
credentials = (aws_key_id = 'aws_key_name'
aws_secret_key = 'aws_secret_key')
file_format = CSV;
when i run above query i will get error as "SQL compilation error: File format 'CSV' does not exist or not authorized."
please send valied answer to solve this issue.
Thank You
the syntax is:
file_format = (type = 'CSV')
However, as CSV is the default, you can leave this out entirely
we are facing below issue while loading csv file from S3 to Snowflake.
SQL Compilation error: Insert column value list does not match column list expecting 7 but got 6
we have tried removing the column from table and tried again but this time it is showing expecting 6 but got 5
below are the the commands that we have used for stage creation and copy command.
create or replace stage mystage
url='s3://test/test'
STORAGE_INTEGRATION = test_int
file_format = (type = csv FIELD_OPTIONALLY_ENCLOSED_BY='"' COMPRESSION=GZIP);
copy into mytable
from #mystage
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
FILE_FORMAT = (TYPE = CSV FIELD_OPTIONALLY_ENCLOSED_BY='"' COMPRESSION=GZIP error_on_column_count_mismatch=false TRIM_SPACE=TRUE NULL_IF=(''))
FORCE = TRUE
ON_ERROR = Continue
PURGE=TRUE;
You can not use MATCH_BY_COLUMN_NAME for the CSV files, this is why you get this error.
This copy option is supported for the following data formats:
JSON
Avro
ORC
Parquet
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html
I am unable to find a way to create an external table in Azure SQL Data Warehouse (Synapse SQL Pool) with Polybase where some fields contain embedded commas.
For a csv file with 4 columns as below:
myresourcename,
myresourcelocation,
"""resourceVersion"": ""windows"",""deployedBy"": ""john"",""project_name"": ""test_project""",
"{ ""ResourceType"": ""Network"", ""programName"": ""v1""}"
Tried with the following Create External Table statements.
CREATE EXTERNAL FILE FORMAT my_format
WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR=',',
STRING_DELIMITER='"',
First_Row = 2
)
);
CREATE EXTERNAL TABLE my_external_table
(
resourceName VARCHAR,
resourceLocation VARCHAR,
resourceTags VARCHAR,
resourceDetails VARCHAR
)
WITH (
LOCATION = 'my/location/',
DATA_SOURCE = my_source,
FILE_FORMAT = my_format
)
But querying this table gives the following error:
Failed to execute query. Error: HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Too many columns in the line.
Any help will be appreciated.
Currently this is not supported in polybase, need to modify the input data accordingly to get it working.
We are using python for data load so we need to create file format in snowflake using python. I have tried to creating file format through python but it got error out.
Could someone please share the sample python script for creating file format using python.
You can execute your file format DDL statement via Python cursor.
snowflake.connector as sfc
# --Snowflake Connection Setup
cnx = sfc.connect(
user='user',
password=pwd,
account='account',
warehouse = 'warehouse',
database='database',
schema='schema',
role = 'role')
cnx.cursor().execute("create or replace file format mycsvformat type = 'CSV'
field_delimiter '|' skip_header = 1;")
dbDir = '/tests/dolphindb/valueDB'
devDir = '/tests/dolphindb/dev.csv'
db = database(dbDir)
dev = db.loadTable(`dev)
saveText(dev, devDir)
I want to export table "dev" as 'csv' file but I encountered this error message:
Execution was completed with exception
SegmentedTable does not support direct access. Please use sql query to retrieve data
I wonder if I have to load all data into memory to export it as 'csv' file.
Yes, the input table for saveText must be a non-partitioned table.