After 4 days of trying everything to load data into snowflake, nothing seems to work at all.
Now as my last option I want to load a local CSV file into snowflake in order to be able to follow the tutorial I am watching.
Unfortunately even this step seems to be a hard one in snowflake. I have seen, that I need to create an internal stage for this. Therefore I went to the Stage and created a "Snowflake Managed", which I think should be an internal stage. I called that Stage "MY_CSV_STAGE".
Internal Stage option on snowflake:
Then I went back to the worksheet and tried the following command:
PUT file://C:\Users\User\Downloads\Projekte/csv_dateien_fuer_snowflake.csv #MY_CSV_STAGE AUTO_COMPRESS=TRUE;
Now by trying to run the command I am just receiving a weired error, which I don't understand:
Error Message:
I really would like to understand what exactly I am doing wrong. I have also read on other places, that I should maybe need Snowsql to import data from local to snowflake. But the installation of the Snowsql I did not figure out.
How can I write this command line in snowflake in order to be able to import the CSV file?
Related
I have a simple requirement where I have to put a file in snowflake internal stage.
put file:///Users/priya/Downloads/vistaslagos.csv #canary2022/apartments;
When I try the above, the command is getting greyed out.
I further tried putting single quotes like below but I am getting the error :-->000002 (0A000): Unsupported feature 'unsupported_requested_format:snowflake'.
put 'file:///Users/priya/Downloads/vistaslagos.csv' #canary2022/apartments;
On further checking the error, it says I need to validate the snowflake edition.
Does anyone has context in this?
It seems you are trying to run the command in worksheets(UI). When you do so and give file:///, it considers the further text in the line as comments which is why you see it as greyed out.
Put command cannot be submitted in UI which is why you see the error feature not supported.
Try using snowsql and this should work. It is also mentioned in official documentation as well. Comments and Links for your reference below.
"The command cannot be executed from the Worksheets Worksheet tab page in either Snowflake web interface; instead, use the SnowSQL client to upload data files, or check the documentation for a specific Snowflake client to verify support for this command"
https://docs.snowflake.com/en/sql-reference/sql/put.html#usage-notes
Cheers!!
I've been looking for information for a long time and I can't get it. I'm starting to think it can't be done if the .parquet are in Azure DataLake Storage.
I have a folder with subfolders in Azure DataLake Storage. In these subfolders there are many .parquet. I manage to get them out using ListAzureDataLakeStorage + FetchAzureDataLakeStorage combination. Then I try to pass them through a PutDatabaseRecord (which I think is the correct processor for the dump in the DB).
I think I have the PutDatabaseRecord well configured. But when executing it gives me an error: "Failed to process session due to Failed to process StandardFlowFileRecord due to java.lang.NullPointerException: Name is null".
I'm not sure I'm using the PutDatabaseRecord right. I thought that PutDatabaseRecord read the flowfiles that came to it interpreting their content as .parquet (it is supposed to use a ParquetReader as a RecordReader), being able to understand the data as records. But it surprises me that it is not necessary to indicate how to interpret the .parquet, nor how to map its columns with those of the DB table. It still doesn't work as I think and it needs the flowfile content to already arrive as records?
The truth is that I can't explain myself better either because I don't really understand what is considered a record in Nifi or how a record is related to a reading of a .parquet.
Either I am missing a processor or something I am configuring wrong. But the only thing I find is the FetchParquet, which seems to be able to read a .parquet and put it into the FlowFile as records. However, it can only be used with ListHDFS or ListFile, which do not allow me to fetch data from Azure Data Lake Storage
After several tests (using the ConvertRecord and QueryRecord processors), I have come to the conclusion that the problem is in the reading that the ParquetReader does of the content of the FlowFiles that arrive. Well, every processor that needs a ParquetReader gives the same error. Downloading the content of the FlowFile that enters the processor that the ParquetReader uses (whatever it is) and using a .parquet viewer I have verified that this content is fine.
Without knowing what to do, I have attached a screenshot of the specific error. I still don't know what "Name" the error refers to.
Error Name is null
Note: I also posted my problem on Cloudera, perhaps better explained. I leave the link in case someone wants to look at it. (https://community.cloudera.com/t5/Support-Questions/How-can-I-dump-the-parquet-data-that-is-in-Azure/td-p/316020)
In the end, the closest thing to the error I was getting was found here (https://issues.apache.org/jira/browse/NIFI-7817). It seems that it is an error related to the creation of the ParquetReader. This makes sense because it would hit any processor that used a ParquetReader. In addition, the FlowFiles did not even enter the processor that used it.
I was using Nifi version 1.12.1. I have downloaded version 1.13.2 and it no longer gives the Name error. In addition, it is seen that the Flow Files already enter the processor. On the download page of the new version (https://nifi.apache.org/download.html) you can access the Release Notes and the Migration Guidance to know what has been fixed with respect to previous versions and with which processors you have to be careful when migrating.
However, even though the data goes into the processor, it still gives me an error, but it is different and I will open it in another post.
I have an SQL Server query and I am using FOR XML PATH to generate the result as XML. I am trying to save the file result.xml to a particular folder of my computer but I do not want to use the "save as" link of SQL Server to do it, because I need to automate my process.
Unfortunately, I cannot use xp_cmdshell solution provided here because I do not have the permissions to do it. In fact I get an error
The EXECUTE permission was denied on the object 'xp_cmdshell', database 'mssqlsystemresource', schema 'sys'
each time I try to execute it.
May anyone help me with that? It would be highly appreciated!
Good day,
like always there are lot of solutions which depend on your knowledge and resources. I will try too give you some ideas here and probably others will be able to add more
Using SQLCLR you can simply create a file and save it (this will require using PERMISSION_SETUNSAFE)
Using Python (basically same as SQLCLR we can do everything with Python including external tasks).
You mentioned that you do not want to use xp_cmdshell at first but after the short discussion I understand that you simply do not succeed to use it, but this is your preferred option. Check this thread on the same topic. You should find the solution there on how to configure the permissions.
Using sp_send_dbmail you can send the result of the query to email. Not like other options this is usually open in most servers. The idea is to send simple email, and using Google Apps you can read the content of the email and create file. You can use for example the methods: GmailApp.getInboxThreads, getMessages(), getRawContent()... and in order to write the file in your Google drive you can use DriveApp.createFile
Using SSIS
Following your advice I took the time to learn how to use C# in order to solve my problem and now it works perfectly! I created a short script that executes my sql query using for xml path and then managed to save the resulting XML file in a specific repertory. Thank you again for the support. So my process is now fully automated. Always nice to learn more!
thanks again for being here!
We have been having this issue for a while but never spent the time to address it.
When we print a page as an excel file through:
Response.ContentType="application/vnd.ms-excel"
We get a download window, as we should, in order to receive the resulting file.
If I save the file rather than open it, my server Event Logs show no error.
When the Open option is chosen, a loading bar appears.
This loading bar seems to be re-querying the page/re-running the asp, because the error we receive boils down to a null variable used in a sql query that is requesting a querystring.
http://s13.postimage.org/5mmkubzlj/ss2.jpg
Do I need to change the method of exporting this data to excel? Is there something I'm missing that will allow Excel to see that the file has already completed processing? Very confused as to what to do here.
Thoughts? Experiences? Thanks again!
I have a client that is looking to create a batch order system that works with Miva.
I am wondering if anyone has created a system that exports orders created that day. The system we need to implement has to create a csv document and send the file to an ftp server.
If anyone has any ideas or examples I would greatly appreciate them as I have not yet worked with that part of Miva's system.
Edit:
After doing some research I found that miva's system saves the export file to the server which can be collected by ftp. I can run the automation and file conversion from our localhost but
I need to find a module or create a module that can create that export daily. Does anyone know of an existing module.
Miva doesn't have a native implementation of cURL.
But it does have, is a "Reports" area, where you can generate reports in CSV of orders placed within the last 24 hours.
It would be up to you to find a way to get the data from the export to the FTP server.
Depending on your setup, you can create a cron to do this for you, and use a different language to FTP the Miva generated CSV file to the proper server (that is, if it isn't the same as the Miva store is running on).