I want to export a raster product from google earth engine to google drive by Export.image.toDrive command. because the file is large, the output will appear in google drive as too many files [tiles] (for example: 240 files). When I download them az a zip file, the output doesn't look well. Is there any solution or command to make it possible export it as a single file? my code is as below:
Export.image.toDrive({
image: AOIclip,
description:'KhR_Luse_GEE',
folder:'AOI_GEELAyers',
//region:table,
scale: 10,
maxPixels:1e13,
skipEmptyTiles: true,
});
Related
Currently I have deployed a Semantic Segmentation model and an endpoint with which I am able to invoke and get inferences. Now, I am getting inferences for each image at a time.
Now I want to try batch of images at a time using Batch Transform job. It worked perfectly fine but the images that is created is an .out file and I'm not able to open that file using any of the viz library like matplotlib imread, PIL Image and openCV imread. It all says not an image.
Just wanted to understand what is the .out file ? and if it is an segmented mask image which is typically the output of a semantic segmentation model then how can I read that file.
My code for Batch Transformation:
from sagemaker.predictor import RealTimePredictor, csv_serializer, csv_deserializer
class Predictor(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session=None):
super(Predictor, self).__init__(
endpoint_name, sagemaker_session, csv_serializer, csv_deserializer
)
ss_model = sagemaker.model.Model(role =role, image=training_image, model_data = model, predictor_cls=Predictor, sagemaker_session=sess)
transformer = ss_model.transformer(instance_count=1, instance_type='ml.c4.xlarge', output_path=batch_output_data)
transformer.transform(data=batch_input_data, data_type='S3Prefix', content_type='image/png', split_type='None')
transformer.wait()
As the official doc say:
The SageMaker semantic segmentation algorithm provides a fine-grained,
pixel-level approach to developing computer vision applications.
It tags every pixel in an image with a class label from a predefined set of classes.
...
Because the semantic segmentation algorithm classifies every pixel in
an image, it also provides information about the shapes of the objects
contained in the image. The segmentation output is represented as a
grayscale image, called a segmentation mask. A segmentation mask is a
grayscale image with the same shape as the input image.
So, what the .out file has are pixel arrays (assigned class for each pixel).
You need to deserialize the file:
from PIL import Image
import numpy as np
import io
from boto3.session import Session
session = Session(
aws_access_key_id=KEY_ID, aws_secret_access_key=ACCESS_KEY, region_name=REGION_NAME
)
s3 = session.resource("s3")
out_file = io.BytesIO()
s3_object = s3.Object(BUCKET, PATH)
s3_object.download_fileobj(out_file)
out_file.seek(0)
mask = np.array(Image.open(out_file))
Also, I found a class ImageDeserializer in this doc to do the job with a stream of data. Maybe you can adapt it to your file because it read a stream of bytes returned from an inference endpoint (batch transform put this data in the .out file).
what is the .out file?
.out file essentially is the stream(content) in your http response HttpEntity(including header and body). It matches the order of the records in your input file.
To be more specific, for each batch transform, SageMaker splits your input file into records, construct multiple http requests and send them to the model server. The inference result is included in the http response. SageMaker upload the inference results to s3 with ".out" suffix.
I'm fairly new to Dart and Flutter, and I'm having trouble to overwrite an existing assets image from a source image.
My attempt:
try {
File localFile = File('assets/images/myImage.png');
localFile.writeAsBytesSync(originFile.readAsBytesSync());
catch (e) {
log(e.toString());
}
I get:
[log] FileSystemException: Cannot open file, path = 'assets/images/myImage.png' (OS Error: No such file or directory, errno = 2)
I did define the assets folder in pubspec.yaml:
assets:
- assets/images/
Ok, so I've read somewhere that the asset file can be accessed like this:
import 'package:flutter/services.dart' show rootBundle;
final byteData = await rootBundle.load('assets/images/myImage.png');
But I don't know how to convert byteData to a File object that represents the actual file.
I think I'm missing something very basic here. Or maybe is there is a proper way to do this that has nothing to do with this approach?
Please help.
Thanks in advance!
If you want to write a file on a user device you should look here: https://flutter.dev/docs/cookbook/persistence/reading-writing-files
Shared preferences are a space in a phone where you app can write, so it's exactly what you want!
Assets are part of you app and are not meant to be modified within the app.
During a build, Flutter places assets into a special archive called
the asset bundle that apps read from at runtime. According to the flutter website
Hope this helps!
Problem:
Files get pulled automatically from my emails to a folder on my Google Drive.
The files are automatically given a name, which was the subject of the email, e.g. "Beach". Multiple files can thus have the same name if emails have the same subject name.
Once the files have landed in Google Drive, I want to move the files, say the ones called "Beach", to another folder called "Beach".
What is the best way to do this? I have tried using scripts, lists of folders/ID/file names etc in spreadsheets, yet can't quite get it.
According to this article, you can use Google Apps Scripts to move files across folders.
function moveFiles(source_folder, dest_folder) {
var files = source_folder.getFiles();
while (files.hasNext()) {
var file = files.next();
dest_folder.addFile(file);
source_folder.removeFile(file);
}
}
Here are some related threads which might help:
Google Drive: Move file to folder
SCRIPT TO MOVE FILES FROM MYDRIVE TO ANOTHER FOLDER IN GOOGLE DRIVE
This is what I use and it works well
The_unique_File_id is the part after drive.google.com/drive/folders/ the letters and numbers
function copyFiles(source_folder, dest_folder) {
var source_folder = DriveApp.getFolderById('The_unique_File_id');
var dest_folder = DriveApp.getFolderById('The_unique_File_id');
var files = source_folder.getFiles();
while (files.hasNext()) {
var file = files.next();
dest_folder.addFile(file);
source_folder.removeFile(file);
}
}
I'm trying to replace a PDF file in a Google Drive Folder using a script. Since GAS does not provide a method for adding revisions (versions), I'm trying to replace the content of the file, but all I get is a blank PDF.
I can't use the DriveApp.File class since our Admin has disabled the new API, so I have to use DocsList.File instead.
Input:
OldFile.pdf (8 pages)
NewFile.pdf (20 pages)
Output expected:
OldFile.pdf with the same content as NewFile.pdf
Real Output:
OldFile.pdf with 20 empty pages.
Process:
var old = DocsList.getFileById("####");
var new = DocsList.getFileById("####");
old.replace(new.getContentAsString());
Any ideas, please?
Thanks a lot in advance.
PS.: I also tried calling old.clear() first, but I'd say the problem lies on the getContentAsString method.
The Advanced Drive Service can be used to replace the content of an existing PDF file in Google Drive. This answer also includes an example of how to update a PDF file in a shared Drive.
function overwriteFile(blobOfNewContent,currentFileID) {
var currentFile;
currentFile = DriveApp.getFileById(currentFileID);
if (currentFile) {//If there is a truthy value for the current file
Drive.Files.update({
title: currentFile.getName(), mimeType: currentFile.getMimeType()
}, currentFile.getId(), blobOfNewContent);
}
}
References
https://developers.google.com/apps-script/advanced/drive
https://developers.google.com/drive/api/v3/reference/files/update
An example of using with a shared Drive:
Drive.Files.update({ title: currentFile.getName(), mimeType:
currentFile.getMimeType() }, currentFile.getId(), blobOfNewContent,
{supportsTeamDrives: true});
Try to get it as a blob datatype instead.
I'd like to use GAE to allow a few users to upload files and later retrieve them. Files will be relatively small (a few hundred KB), so just storing stuff as a blob should work. I haven't been able to find any examples of something like this. There are a few image uploading examples out there but I'd like to be able to store word documents, pdfs, tiffs, etc. Any ideas/pointers/links? Thanks!
The same logic used for image uploads apply for other archive types. To make the file downloadable, you add a Content-Disposition header so the user is prompted to download it. A webapp simple example:
class DownloadHandler(webapp.RequestHandler):
def get(self, file_id):
# Files is a model.
f = Files.get_by_id(file_id)
if not f:
return self.error(404)
# Set headers to prompt for download.
headers = self.response.headers
headers['Content-Type'] = f.content_type or 'application/octet-stream'
headers['Content-Disposition'] = 'attachment; filename="%s"' % f.filename
# Add the file contents to the response.
self.response.out.write(f.contents)
(untested code, but you get the idea :)
It sounds like you want to use the Blobstore API.
You don't mention if you are using Python or Java so here are links to both.
I use blobstore API that admits any file upload/download up to 50 MB.