I have created a directory and folder for keeping the model and checkpoints in amazon sagemaker. How can i access this directory and folder to see model and checkpoints ?
If you are running training/serving the endpoint in AWS and not locally, all the files are visible under Amazon S3. Likely when you set up the Sagemaker configuration, you had to create a S3 bucket also.
The other option is that the training runs locally. In that case the files are somewhere in your computer.
When a training job ends, Sagemaker copies the resulting model out of the Docker image to S3 or local computer.
In training, the contents of /opt/ml/model are uploaded to the S3 location specified in the OutputDataConfig passed when creating the training job.
In hosting and Batch Transform, the S3 location specified by ModelDataUrl is unarchived and made available to the inference container at /opt/ml/model.
Also see the documentation:
https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html#your-algorithms-training-algo-envvariables
https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html#your-algorithms-inference-code-load-artifacts
https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-batch-code.html#your-algorithms-batch-code-load-artifacts
Related
I have a flask server running on a GAE (flexible env). The application generates certain files during runtime based on the API requests received. But after the instance starts, after idling, the files are lost. How do i prevent this ?
Your application seems to write those files on the filesystem. However, whatever is written in the filesystem will not be persisted and will be lost on instance death. Also, what you write in one instance won't be available to other instances serving your app.
The solution to this is to write your files to Google Cloud Storage. Any instance can write and retrieve its files there and they'll be available for any instance of your service.
How may I upload a data file to my Google app engine folder using any standard file transfer protocol e.g. SFTP, WebDav etc.?
This is so I can use a regular desktop file transfer client
So far the only ways I've found are Google proprietary, and web form upload.
GAE's filesystem is readonly, i.e. you can't really make changes to any of the files once the app is deployed.
You can upload your files to GCS by mounting a bucket as a local directory but as you noted that option doesn't seem to be available on Windows.
You could also use gsutil to transfer files to/from the bucket, like that: gsutil cp *.txt gs://my-bucket, even on Windows (see more details here) but if you want a GUI client - just google for gui clients for gcs, many storage browsers i.e. more or less advanced ftp clients seem to support it (see for example Cyberduck, CrossFTP, etc).
I have an Azure app service with two additional slots - one for QA and testing and one for staging. The deployment process sees me deploying to QA for testing, then to Staging when I'm ready to go live. Staging is then swapped with the Production application to avoid downtime for users.
The problem I have is that I would like to keep some site content - effectively image files uploaded by users. They are located in a specific folder - let's say \wwwroot\images. I can't push these to my TFS system as they are effectively client data files.
Is there a way I can do this deployment without having to back that folder up (using FTP) and restore it after the swap (a 30 minute process)?
In hindsight I could probably have stored the images in the DB - they're not particularly large, but is there another approach that makes more sense? What about using Azure storage? What would you do?
Thanks in advance for any advice!
Typically for any Storage purpose, it's not recommended way to store images in your code by creating any folder(s). Since Azure web app is a PaaS model, you don't own the server. Your code is somewhat deployed in any machine(s) among the Azure DataCenter.
As you mention, you can use Azure Storage.
Azure Blob Storage is a the which you need to look.
Here some good demo for uploading images in Azure Blob Storage
I have uploaded several files into the same folder on Google Cloud Storage using the Google Cloud Console. I would now like to move several of the files to a newly created folder in Google Cloud Storage and I cannot see how to do that via the Google Cloud Console. I found instructions to move the files via command prompt instructions on gsutil. However, I am not comfortable with command line interfaces and have not been able to get gsutil to work on my machine.
Is there a way to move files in Google Cloud Storage from one folder to another via the Google Cloud Console?
Update: Google Cloud Shell provides a terminal within the Google Cloud Console site without having to manually create VMs; it comes with gsutil and Google Cloud SDK pre-installed and pre-authenticated.
Prior answer: If you're having issues installing gsutil on your computer, consider the following approach:
Spin up an f1-micro instance with the Google-provided Debian image which will have gsutil preinstalled.
Use the SSH button to connect to it using the browser interface (you can also use gcutil or gcloud commands, if you have those installed and available).
Run gcloud auth login --no-launch-browser within the instance. It will give you a URL to open with your browser. Once you open it, grant the OAuth permissions, and it will display a code. Paste that code back into the command-line window where you ran the command so that it gets the authentication token.
Run the gsutil mv command, as suggested by Travis Hobrla:
gsutil mv gs://bucket/source-object gs://bucket/dest-object
Once you're done with gsutil, delete the instance by clicking on the Delete button at the top of the VM instance detail page. Make sure that the box marked "Delete boot disk when instance is deleted" on the same VM instance page is checked, so that you don't leave an orphaned disk around, which you will be charged for.
You can also browse your persistent disks on the "Disks" tab right below the "VM instances" tab, and delete disks manually there, or make sure there aren't an orphaned disks in the future.
Given the current price of $0.013/hr for an f1-micro instance, this should cost you less than a penny to do this, as you'll only be charged while the instance exists.
There is not currently a way to do this via the Google Cloud Console.
Because folders in Google Cloud Storage are really just placeholder objects in a flat namespace, it's not possible to do an atomic move or rename of a folder, which is why this scenario is more complex than doing a folder move in a local filesystem (with a hierarchical namespace). That's why a more complex tool like gsutil is needed.
Google Cloud Storage now has the functionality to move files from one folder/bucket to another using Cloud Console. To do this, simply select the file(s), click on the 3 vertical dots to get the option of move. Select the target folder/bucket to move the file.
Lets say on my local machine in the folder which contains my GAE project I have an images folder.
When I upload the app to the GAE with the correct .yaml information the images folder and its contents will be uploaded.
Now lets say, I'm running the APP online and I upload an image to the image folder now on Google's servers. Now the images folders contents on the web and on my local development machine are different.
My question is this:
Next time I upload the app to GAE, how will the discrepancy between the different contents of the image folder be resolved?
You can't add files to the application like that after deployment. The local file system is read-only to GAE applications.
If you were to upload an image (via a handler you create) when the app is deployed you can't save it in the image folder in your application, you can only save it to the data/blob store. The files you uploaded with your app are static, they cannot be changed either by you or the application outside of the deployment tool. You can read them in, sure, but not write to them once deployed/at all.
So the situation will never arise that a deployed version has different files to the local version - they are always identical.