What's the correct way to read a file using Google AppEngine (Go)?
In Java I read there are context.getResourceAsStream, is there any equivalent function for that?
You can read from files on App Engine the same way you can read from files in a Go app running on your computer.
Some things to keep in mind:
You should use relative file paths instead of absolute. The working directory is the root folder of your app (where the app.yaml file resides).
Only files that are application files can be read by Go code, so if you want to read a file from Go code, the file must not be matched by a static file pattern (or if it must be available as a static file too, application_readable option must be specified at the static file handler which includes/applies to the file, details).
The latter is detailed on the Application configuration page, Section Static file handlers. Quoting the relevant part:
For efficiency, App Engine stores and serves static files separately from application files. Static files are not available in the application's file system. If you have data files that need to be read by the application code, the data files must be application files, and must not be matched by a static file pattern.
So let's say you have a folder data in your app's root (next to app.yaml), and a file list.txt in it. You may read its content like this:
if content, err := ioutil.Readfile("data/list.txt"); err != nil {
// Failed to read file, handle error
} else {
// Success, do something with content
}
Or if you want / need an io.Reader (os.File implements io.Reader along with many other):
f, err := os.Open("data/list.txt") // For read access.
if err != nil {
// Failed to open file, log / handle error
return
}
defer f.Close()
// Here you may read from f
Related questions:
Google App Engine Golang no such file or directory
Static pages return 404 in Google App Engine
How do I store the private key of my server in google app engine?
Related
I'm using "github.com/dgrijalva/jwt-go" to create JSON web tokens.
When I hosted my server locally, I could use my private key as usual. But in GAE it won't work because I don't have access to the file system.
How would you guys do it? Store the key in datastore or any other ideas?
Thanks
Edit:
My app.yaml looks like this (below api_version and stuff):
handlers:
- url: /.*
script: _go_app
On AppEngine you don't have access to the file system of the host operating system, but you can access files of your web application (you have read-only permission, you can't change them and you can't create new files in the app's folder).
So the question is: do you want to change this private key from your application without redeploying your app? Or it is perfectly fine if it is deployed "statically" with your app's code?
If you don't need to change it (or only when you redeploy your app), easiest is to store it as a "static" file as part of your webapp. You may refer to files of your app using relative paths, where the current or working directory is your app's root. E.g. if your app contains a key folder in its root (where app.yaml resides), and there is a my_key.txt file inside the key folder, you can refer to it with the path: key/my_key.txt.
Actually it is quite common to "ship" static files with your app's code: just think of HTML templates which are read and processed by the Go code (e.g. package html/template) to produce HTML result; the content of the HTML template files are not served directly to clients.
If you need to change it from time to time without having to redeploy your app, then store it in the Datastore which your app can read and modify.
Note:
One important note: not every file is readable by code, this depends on the app configuration. Quoting from Configuring with app.yaml / Static file handlers:
Static files are files to be served directly to the user for a given URL, such as images, CSS stylesheets, or JavaScript source files. Static file handlers describe which files in the application directory are static files, and which URLs serve them.
For efficiency, App Engine stores and serves static files separately from application files. Static files are not available in the application's file system. If you have data files that need to be read by the application code, the data files must be application files, and must not be matched by a static file pattern.
Static file handlers can be defined in two ways: as a directory structure of static files that maps to a URL path, or as a pattern that maps URLs to specific files.
Read the link how to properly configure application and static files / directories.
The solution was to leave app.yaml as it were. Put app.yaml at root lvl in project. Then change all imports from starting at GOPATH to start at project root instead. The problem that made me choose to put app.yaml and main go file in a different folder under project root was because of double imports. Read this for a better understanding: Google Go AppEngine imports and conflicts when serving / testing
The solution made my project find the files I wanted.
I'm trying to upload files to Google Drive with ProgressListener and ChunkSize enabled (thus with DirectUploadEnabled disabled). This way i have a more reliable upload and the possibility for a progress indication to the user.
I transfer the files from the GWT website to the GAE with a FormPanel and a FileUploadField which POSTS the file to GAE on submit(). On the GAE i receive the file with an UploadServlet which uses org.apache.commons.fileupload to receive the documents as a stream. I don't want to receive the complete documents on the GAE because the documents are to big. Therefore i start the upload (insert) to Google Drive with the received stream from the incoming request.
Now there's a problem; for the insert i need to know the size of the stream;
int lContentLength = getRequest().getContentLength();
FileItemStream lFileItemStream = getFileItemStream();
InputStream lInputStream = lFileItemStream.openStream();
BufferedInputStream lBufferedInputStream = new BufferedInputStream(lInputStream);
InputStreamContent lInputStreamContent = new InputStreamContent(pContentType, lBufferedInputStream);
lInputStreamContent.setLength(lContentLength);
My first guess was the ContentLengt from the incoming Servlet request. But this is not correct because this concerns the complete request (which also contains other fields which are used as parameters). Without the Drive option DirectUploadEnabled i need the exact stream size from the uploaded document, otherwise the upload stall's at the end...
How do i grap this document size? The Google example is stupid because it uses a local file;
https://code.google.com/p/google-api-java-client/wiki/MediaUpload
Yes from a local file it is easy to get the file size (mediaFile.length()). But from a website ... Several sites specify it is not possible to grab the file size before submit() from the website, and it seems also impossible to determine the stream-size on GAE without loading the complete file...
How do i determine this streamsize? Is there another solution for this problem?
I have a ~2MB file that my Google AppEngine server must use (not serve) as part of a computation for a service request.
That is, a client makes a particular request, my GAE server must first get the data from this ~2MB file, do some computations using this data, then serve a small response back to the client.
Where best do I store this data so that it can be quickly read and used by the server in the computation?
If the following assumptions hold true
the file is not going to require updates outside of appengine code updates
that the file is read only
Then deploy the file with your code and read the file into memory during startup (ideally using warmup requests) and just operate on it from memory. If you code has to have file based semantics to access the data (read,seek, etc) then read the file contents and wrap it in StringIO.
You will need to assign the value read from the file to a module level variable, that way whenever you get a new request you can just get the files contents by importing the module and referencing the name. ie. mymodule.filecontents
What I'm trying to do is simple. Load am XML file using ElementTree so I can traverse it.
Here's the code:
_uri = '/news.xml'
self.root = ElementTree.parse(_uri).getroot()
And, the error:
file not accessible: '/news.xml'
From what I can tell, the parser can't find the document. Is there something I need to configure so python can see my site's files?
This is probably similar to:
Read a file on App Engine with Python?
I.e. files that are marked as static are not accessible, but you can also serve it as an application resource file.
I have a static XML file called rules.xml in war/xml/. It is a rules file for the Apache Commons Digester. In order to be able to use the file I need to be able to open it with a Reader. How can I open the file?
try using
final String file = "xml/rules.xml";
FileReader fileReader;
try
{
fileReader = new FileReader(file);
...
}
catch(..)
edit: after some intensive usage, FileReader in GAE seems have some trouble with accentuated characters (Only visible on GAE cloud instances, local tests runs perfectly).
If someone encounters this kind of bugs, use FileInputStream instead. It worked for me.
Check this page for more informations : http://code.google.com/intl/en/appengine/kb/java.html#readfile
Cheers.