How can I save the output file from Run query and list results in a .PARQUET file format.
This is my current workflow.
My Logic App is working, But the file .parquet created are not valid every time I view it on Apache Parquet Viewer
Can someone help me on this matter. Thank you!
Output:
I see that you are trying to add .parquet to the csv file you are receiving but that's not how it will be converted to parquet file.
One of the workarounds that you can try is to get the csv file and then add Azure function which can convert into parquet file and then adding the azure function to logic app.
Here is the function that worked for me:
BlobServiceClient blobServiceClient = new BlobServiceClient("<YOUR CONNECTION STRING>");
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("<YOUR CONTAINER NAME>");
BlobClient blobClient = containerClient.GetBlobClient("sample.csv");
//Download the blob
Stream file = File.OpenWrite(#"C:\Users\<USER NAME>\source\repos\ParquetConsoleApp\ParquetConsoleApp\bin\Debug\netcoreapp3.1\" + blobClient.Name);
await blobClient.DownloadToAsync(file);
Console.WriteLine("Download completed!");
file.Close();
//Read the downloaded blob
Stream file1 = new FileStream(blobClient.Name, FileMode.Open);
Console.WriteLine(file1.ReadToEnd());
file1.Close();
//Convert to parquet
ChoParquetRecordConfiguration csv = new ChoParquetRecordConfiguration();
using (var r = new ChoCSVReader(#"C:\Users\<USER NAME>\source\repos\ParquetConsoleApp\ParquetConsoleApp\bin\Debug\netcoreapp3.1\" + blobClient.Name))
{
using (var w = new ChoParquetWriter(#"C:\Users\<USER NAME>\source\repos\ParquetConsoleApp\ParquetConsoleApp\bin\Debug\netcoreapp3.1\convertedParquet.parquet"))
{
w.Write(r);
w.Close();
}
}
after this step you can publish to your azure function and add the Azure function connector to your logic app
You can skip the first 2 steps (i.e.. Read and Download the blob) and get the blob directly from logic app and send it to your azure function and follow the same method as above. The generated parquet file will be in this path.
C:\Users\<USERNAME>\source\repos\ParquetConsoleApp\ParquetConsoleApp\bin\Debug\netcoreapp3.1\convertedParquet.parquet
Here convertedParquet.parquet is the name of the parquet file. Now you can read the converted parquet file in Apache Parquet reader.
Here is the output
Related
I have an excel file stored in src folder. I want to use react to read the excel file directly instead of uploading a file using input field and show the object array to the console, does anyone know how to do it? Any reference link about that? I saw a lot about uploading an excel file and output the data, but I don't want to upload the file. I want to import the file directly or fetch data from the file. Let me know if you know how to do in react. Thanks.
You could use FileReader API. Here's a link to similar question. You could parse/read the excel file using this package.
A little suggestion, would be to use .csv or .json files instead of .xlsx files, if you are only dealing in data.
fetch(excelFile).then((res) => res.arrayBuffer())
.then((ab) => {
const wb = XLSX.read(ab, { type: "array" });
const sheetname = wb.SheetNames[0]
const ws = wb.Sheets[sheetname]
const json = XLSX.utils.sheet_to_json(ws,{header: 1,defval:''})
console.log(json)
Here excelFile is file location, you may include your excel file location, Then json contain excel data.
Thanks.
I am developing a Flutter App, and I wish to save a list of objects (Contacts) to the app directory.
As suggested in the Cookbook, I use the path_provider package and the File class of Dart.
Getting to write the objects was easy (I think - I didn't get any error). Here is the code I used for writing the list:
Future<void> writeContact(List<Contact> contacts) async {
final directory = await getApplicationDocumentsDirectory();
final file = File('${directory.path}/contacts.txt');
// Write the file
IOSink sink = file.openWrite(mode: FileMode.write);
sink.write(contacts);
sink.close();
}
The question is - how to I read this file and populate the List of Contacts?
Thanks, Gal.
I was thinking if I store a video or a movie and open that box will that video will be stored in my RAM or else it just load from ROM. I am a bit confused: Can anyone explain this to me?
I think you have misunderstood the concept of Database.
Any Database solution is to only store pure informational organized data. Not to store large files such as media, documents, or images.
On the contrary, storage need not be organized, all files can exist in one folder.
So, any database solution you use, always store Data Types.
In this case you can have a Data Model, which is also an essential thing in using a Database.
#HiveType(typeId: 0)
class Movie extends HiveObject {
#HiveField(0)
String name;
#HiveField(1)
int path;
}
Since Hive supports Dart objects, you don't have to convert toJson or any such for string the Data.
So when you have the file fetched from Storag, you can get the path using path_provider or from the File itself, and then Create a Object
File file = await // get the movie file using any means
final path = file.path
var box = await Hive.openBox('Movies');
var m = Movie()
..name = 'Batman Begins'
..path = path ;
box.add(m);
m.save();
Hope this clears your doubt.
Copy/save your video/media files in the Local File Storage and save file path in Hive Box.
Whenever you need get path from hive then get the file from local storage using that path.
Succesfully i have made to Upload files into firebase storage, but now i want to display all files in table and to have option to download each file.I've read the documentation in firebase but it won't work.When i click the button which function is to get all files and the i want to visualize them in table which users can see:
Show file function:
showFileUrl(){
storageRef.child('UploadedFiles/').listAll().then(function(res) {
res.items.forEach(function(folderRef) {
console.log("folderRef",folderRef.toString());
var blob = null;
var xhr = new XMLHttpRequest();
xhr.open("GET", "downloadURL");
xhr.responseType = "blob";
xhr.onload = function()
{
blob = xhr.response;//xhr.response is now a blob object
console.log(blob);
}
xhr.send();
});
}).catch(function(error) {
});
}
This is log of the network which i found when debugging.What i need to do to get all data and visualize it in table and to hava a download button and when is pressed to download the file
Network log:
Storage in firebase:
Blob object of the files:
Your code gets a list of all the files, but it doesn't actually to anything to read the data for each file.
When using the Web client SDK, the only way to get the data for a file is through a download URL as shown here. So you'll need to:
Loop through all the files you get back from listAll() (you're already doing this).
Call `getDownloadURL as shown here, to get a download URL for each file.
Then use another library/function (such as fetch()/XMLHTTPRequest) to read the data for each file.
Alternatively, if your files are images, you can stuff the download URL in an img tag as the preview.
I've been trying to figure out why part of my Google App script doesn't work, but I've failed to come up with an answer.
The script is downloading an attachment, CSV, from an email in Gmail and stores in with a specific name in a specific folder - this works perfectly fine.
But then I want to edit the CSV, and this is where I run into problems.
var newFolderIterator = DriveApp.getFoldersByName(destinationFolderName)
var newFolderId, myFileName, myFileId
while(newFolderIterator.hasNext()) {
newFolderId = newFolderIterator.next().getId()
var newFileList = DriveApp.getFolderById(newFolderId).getFiles()
while(newFileList.hasNext()) {
myFileName = newFileList.next()
myFileId = myFileName.getId()
var myFileURL = myFileName.getUrl()
Logger.log(myFileName.getId() + " " + myFileName.getName()) //Logs the ID and Name
Logger.log(myFileURL) //Logs the URL
var ss = SpreadsheetApp.openById(myFileName.getId()) //Error, cannot find the ID (error message: perhaps it's missing?)
}
}
I've tried using the openByURL as well, with the same error message.
Probably really easy to fix, any hints and tips is appreciated.
Thanks
The problem here is you are uploading a CSV but attempting to open it with SpreadsheetApp. SpreadsheetApp can only open Google Sheets documents, and your CSV is not automatically converted.
The CSV must first be converted to a Google Sheets document before you can access it with SpreadsheetApp.
You may be able to do this on upload, but I haven't tried this personally. This question looks relevant:
How to automatically import data from uploaded CSV or XLS file into Google Sheets