I have a ~2MB file that my Google AppEngine server must use (not serve) as part of a computation for a service request.
That is, a client makes a particular request, my GAE server must first get the data from this ~2MB file, do some computations using this data, then serve a small response back to the client.
Where best do I store this data so that it can be quickly read and used by the server in the computation?
If the following assumptions hold true
the file is not going to require updates outside of appengine code updates
that the file is read only
Then deploy the file with your code and read the file into memory during startup (ideally using warmup requests) and just operate on it from memory. If you code has to have file based semantics to access the data (read,seek, etc) then read the file contents and wrap it in StringIO.
You will need to assign the value read from the file to a module level variable, that way whenever you get a new request you can just get the files contents by importing the module and referencing the name. ie. mymodule.filecontents
Related
I am trying to load Tensorflow.js model via HTTP protocol. Tensorflow.js requires me to store 'model.json' and 'weights.bin' files in the same folder. But I can only call 'model.json' as a parameter. It refers to the binary file by itself. That is how it works as far as I know.
For now, in the local environment, I am loading the model from the localhost(Http://127.0.0.1:8080) and it works fine.
However, the actual application accepts HTTPS protocol only. So I have tried to store them with models and weights in the same buckets in S3 and called via Lambda but it seems like only 'model.json' is retrieved. I am thinking of using EC2 instances where the Python Flask server is running but it seems like the same that only model.json is retrieved, not binary files.
Is there any way that I can retrieve 'model.json' with referring to the weight file? Is there anyway to host file server remotely with HTTPS protocol?
TFJS downloads model JSON, parses it and uses whatever paths are specified in the JSON - you can edit that file and set any URL you want for weights.
Alternatively, you can also use lower-level methods to load weights manually (in case you want to have a custom loader, etc.), but leave that for future until you're more comfortable with TFJS.
On my appspot website, I use a third party API to query a large amount of data. The user then downloads the data in CSV. I know how to generate a csv and download it. The problem is that because the file is huge, I get the DeadlineExceededError.
I have tried tried increasing the fetch deadline to 60 (urlfetch.set_default_fetch_deadline(60)). It doesn't seem reasonable to increase it any further.
What is the appropriate way to tackle this problem on Google App Engine? Is this something where I have to use Task Queue?
Thanks.
DeadlineExceededError means that your incoming request took longer than 60 secs, not your UrlFetch call.
Deploy the code to generate the CSV file into a different module that you setup with basic or manual scaling. The URL to download your CSV will become http://module.domain.com
Requests can run indefinitely on modules with basic or manual scaling.
Alternately, consider creating a file dynamically in Google Cloud Storage (GCS) with your CSV content. At that point, the file resides in GCS and you have the ability to generate a URL from which they can download the file directly. There are also other options for different auth methods.
You can see documentation on doing this at
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/
and
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/functions
Important note: do not use the Files API (which was a common way of dynamically create files in blobstore/gcs) as it has been depracated. Use the above referenced Google Cloud Storage Client API instead.
Of course, you can delete the generated files after they've been successfully downloaded and/or you could run a cron job to expire links/files after a certain time period.
Depending on your specific use case, this might be a more effective path.
Is it possible to use a local database file with html5 without using a server. I would like to create a small application that depends on information from a small database. I do not want to host a server just to pull information. Is it possible to create a database file and pull information from the local files ?
Depends on the following:
The type of application you want to build:
Normal website with some data being pulled from a local storage;
Special purpose hosted website / application with data generated by the user;
Special purpose local application with a dedicated platform (a particular browser) and with access to the browser's non-web API -- in order to access the browser's own persistent storage methods (file storage, SQLite etc.);
Special purpose local application with a dedicated environment -- in order to deploy the application with a local web server and database;
Available options:
Indexed DB
Web Storage
XML files used for storing data and XSLT stylesheets for translating the data into HTML;
Indexed DB and Web Storage ar available in some browsers but you need to make sure the targeted browsers have it. Their features aren't quite as complete and flexible as SQL RDBMSs but they may fit the bill if your application doesn't need all that flexibility.
XML files can contain the data you want to be shown to the user and they can be updated manually (not by the user) or dynamically (by a server script).
For dynamic updating the content of the XML is kept in JavaScript and manipulated / altered (using the XML DOM) and when the session is over the XML content is sent to the server to entirely replace the previous XML file. This works OK if the individual users have a file each and they never write to each other's files.
Reading local files:
Normal file access is prohibited (for security reasons) to all local (JavaScript) code, which means that "having" a file locally implies either downloading it from a known source (a server) or asking the user to offer access to a local file.
Asking the user to offer access to a local file which implies offering the user a "file input" -- like for uploads but without actually uploading the file.
After a file has been selected using FileAPI to read that file should be fairly simple.
This workflow would involve the user "giving" you the database on every page refresh -- but since it's a one page thing it would mean giving you the data on every session as long as your script does not refresh the page.
You can use localstorage but you can run a server from your own computer. You can use Wamp or Xampp. Which use Apache and mysql.
What i'm looking for is a little more robust than a cookie. I am making a web application for a friend that will be 1 page, and have a list of names on the page. The person wants to be able to add names to the list, however they do not want to use a web server. Just want the files locally on a computer so a folder called test-app , with index.html, and possibly a database file that can be stored in the web browser or a way to save information to the web browser for repeated use.
Is it possible to send a http upload request a file to a Apache or IIS that will have a fileName with "../" or ".." that wouldn't be rejected and would be passed to php or ASP.Net engine?
Not really the way you are asking. By the time it gets to the server the browser has read the file and delivered it as a chunk of content with no information about where it came from other than the original file name which you can choose to use or discard.
Generally file uploads go into a temporary storage place (e.g. /tmp) and then need to be moved out of there to somewhere which you can control and name.
This storage is configured on the server, and so any attempt to put path info into the filename should also be blocked by the file upload implementation of the server which should sanitise the filenames again if the browser didn't already do so.
If there's a bug then all bets are off though.
Objective: Suppose the client submits a string or text file to the server (Google App Engine) using a web form. I want the server to modify the original file and serve it back to the client.
I think the only way to serve files from GAE is using the Blobstore, right? Then, as we cannot modify blobs, I believe a solution would be:
Client uploads a file using HttpRequest
Server reads the uploaded file and copies it to a temp buffer (not sure if is there a method to do this)
Server deletes original blob
Server modifies data in the temp buffer
Server writes the modified buffer to the Blobstore
Server serves the new blob to the client
Would this work? Could you think about any other solution?
Thanks
I think the only way to serve files from GAE is using the Blobstore, right?
Wrong. A 'file' is just a way of storing data on disk; there's nothing about serving them from a webserver that requires the data come from an actual, writable disk file. You can simply accept the user's data via a form upload, modify it, and serve it back to them, without it having to ever touch disk, the blobstore, or any other permanent storage medium.
This only becomes a problem if the user's data is too large to fit in memory, in which case you will have to store the data somewhere while you work on it, such as in the blobstore.
http://code.google.com/appengine/kb/java.html#fileforms
shows you how to do it for file upload, which has to be performed thro multipart form-data.
Similarly for non-file data, where you read straight from the request stream.
You don't even have to store the file/input stream. Just spit out the processed data into the output response stream, while reading the input FileItemStream or request inputstream.
If your file/input processing requires look-forward, determine the maximum distance of look-forward and use that distance as your buffer size.
Further Edits
To respond to the client with a file type, set the response content-type or mime-type.
e.g., I've had apps which dynamically generated gifs, jpgs, xls, cvs, etc.
There isn't any difference whether source of response stream is a file you read or a stream that you generate dynamically. Because, even if you had a stored file that needs to be sent as response to client, you could still have to convert it into a response stream and flag the content-type appropriately.
For dynamically generated content, unless you need to cache the output, you need not generate the file into a web URL-visible location and then generate a new html page with the link, and send that html page to the browser. You don't need the user's browser to have to refresh itself just to get that link.
You would simply send the "file" directly with the response stream. You could design your GWT client to accept the "file", perhaps in a named frame, where the named frame src url is the app that performs the dynamic generation of the file.
Read http://en.wikipedia.org/wiki/Mime-type to find the content-type you need.
If the target client's browser does not have the content handler set-up for the response's content-type, it would as for a treatment or be treated as a file download.
I had frequently used jsp or jspx to generate dynamically generated charts or spreadsheets. No stored files involved. The response is written to while the request is being read. Let's look at the jsp page directive to set the content-type to invoke MS Excel on a CSV.
<%# page language="java" contentType="application/vnd-ms-excel; charset=UTF-8"
pageEncoding="UTF-8"%>
For a servlet, ServletResponse.setContentType(String)
is method to set the content-type.