I'm trying to setup Cloud Debugging for a Python App Engine module without success. See this question for the specific issue I am having.
I am wondering if the reason for my issue is the locations of the source files in my repository.
My source files are (for various reasons) in rather idiosyncratic locations and I have a "build" step that copies the files into into a staging directory where everything is laid out as app engine expects. It also generates some files (including the app.yaml) based on configuration settings.
I then run appcfg.py update from this staging directory.
Given all this moving around of files, I am wondering how the Cloud Debugger can identify which source file in the module I uploaded corresponds to which file in the repository. Is it designed to look in a specific locations (which will explain my problem), or is it somehow more robust than that?
I don't think the issue is related to location of source files in the repository.
The Python Cloud Debugger loops through all the loaded modules and tries to find the best match. The actual location of the module only matters if there are multiple modules with the same name. In this case, the debugger will try to find the best match given the relative path of the files in the repository. You can see the implementation here.
Related
I'm feeling stupid, but I want to know how GitHub and Dropbox store user files, because I have a similar problem and I need to store user's project files .
Is it just like storing project files somewhere in the server and refer to the location as a field in the database, or there are other better methods ?
Thanks.
GitHub uses Git to store repositories, and accesses those repos from their Ruby application. They used to do this with Grit, a Ruby library. Grit was written to implement Git in Ruby but has been replaced with rugged. There are Git reimplementations in other languages like JGit for Java and Dulwich for Python. This presentation gives some details about how GitHub has changed over the years and is worth watching/browsing the slides.
If you wanted to store Git repositories, what you'd want to do is store them on a filesystem (or a cluster thereof) and then have a pointer in your database to point to where the filesystem is located, then use a library like Rugged or JGit or Dulwich to read stuff from the Git repository.
Dropbox stores files on Amazon's S3 service and then implements some wrappers around that for security and so on. This paper describes the protocol that Dropbox uses.
The actual question you've asked is how do you store user files. The simple answer is... on the filesystem. There are plugins for a lot of popular web frameworks for doing user file uploads and file management. Django has Django-Filer for instance. The difficulty you'll encounter in rolling your own file upload management system is building a sensible way to do permissions (so users can only download the files they are entitled to download), so it is worth looking into how the various framework plugins do it.
When working with Google Cloud Endpoints in a appengine project (eclipse based), some files that describe the api are automatically generated every time the endpoint is edited and for every version.
The files are *-v1.api, *-v1-rest.discovery and *-v1-rpc.discovery (the version number may change) and are placed in WEB-INF.
¿Should these files be committed to source control?
My impression is that if the files are automatically generated, they will always be available and there is no need to track them.
Even if I add more versions of the endpoint in the future, I need to keep all those versions for backwards compatibility so all .api and .discovery files will also be generated for all the versions.
Personally, I don't version control (or even worry about backing up) any generated files. I only worry about source and published binaries. And in theory you don't need the binary either because you should be able to recreate the binary from those source files.
My Java JSP application requires to store permanent files on the Tomcat web server. At the moment I save the files in the "/temp" folder of the System. But this folder gets cleared from time to time. Further, the current solution is hard-coded which makes it less flexible (e.g. when moving to another server).
I would like to now if there is a best practice for defining and accessing a permanent directory in this configuration. In detail, where is the best place to define the app file directory, and how would I access this from within my java application? The goal of this setup would be to cause the least effort when (a) updating the application (i.e. placing a new war file), and (b) moving from one server to another and OS (e.g. Unix, Windows, MacOS).
The research I have done on this topic so far revealed that the following would be solutions (possibly amongst others):
1.) Use of a custom subdirectory in the Tomcat installation directory.
What happens to the files if I deploy a new version on the tomcat via
war file?
Where do I define this directory to be accessed from
within my Java application?
2.) In a separate directory in the file system.
Which are good locations or ways to get the locations without knowing
the system?
Where do I define this directory to be accessed from
within my Java application?
Thank you for your advice!
Essentially, you are creating 'a database' in the form of some files. In the world of Java EE and servlet containers, the only really general approach to this is to configure such a resource via JNDI. Tomcat and other containers have no concept of 'a place for persistent storage for webapps'. If a webapp needs persistent storage, it needs to be configured via JNDI, or -D, or something you tell it by posting something to it. There's no convention or standard practice you can borrow.
You can pick file system pathname by convention and document that convention (e.g. /var/something on Linux, something similar on Windows). But you won't necessarily be aligned with what anyone else is doing.
Is it possible to access files beyond WEB-INF folder in Google App Engine's development server ? Maybe I can somehow force GAE dev server to do not block external connections/file access ?
I'm trying to create Java version of gae-sqlite. But I can't access external sql server (mysql or h2db) due to access denied exceptions.
Call stack:
Caused by: java.security.AccessControlException: access denied (java.io.FilePermission C:\work\test_projects\gae_test\out\artifacts\gae_test_war_exploded\database.lock.db write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:252)
at java.lang.SecurityManager.checkWrite(SecurityManager.java:962)
at java.io.File.createNewFile(File.java:882)
at org.h2.store.fs.FilePathDisk.createFile(FilePathDisk.java:121)
at org.h2.store.fs.FileUtils.createFile(FileUtils.java:59)
at org.h2.store.FileLock.lockFile(FileLock.java:327)
at org.h2.store.FileLock.lock(FileLock.java:128)
at org.h2.engine.Database.open(Database.java:542)
at org.h2.engine.Database.openDatabase(Database.java:222)
According to GAE for Java Questions:
Why can't I read from this file?
It is possible to read from a file which is uploaded as part of your
application provided that it is in the following locations:
war/WEB-INF
in a location matching the <resource-files> pattern in appengine-web.xml (which by default includes everything)
If the file location is not the issue, the problem may be that the
method you are using to read from the file is not whitelisted. Your
application can use any IO classes that are useful for reading from
the file system, such as File, FileInputStream, FileReader, or
RandomAccessFile. For a full list of whitelisted classes, please see
the JRE Class White List. If you need to get file access to your own
resources (such as properties files), you could put these files inside
of jars and use Class or ClassLoader to load them.
So to answer your question, the only available place besides the WEB-INF directory to read files from is whatever you specified in the <resource-files> entry in appengine-web.xml.
Now one thing I noticed is that you're trying to read a DB file which should be okay if you're only considering it read-only. You cannot write to any "local" files because (from the same link):
Why can't I write to this file?
Writing to local files is not supported in App Engine due to the
distributed nature of your application. Instead, data which must be
persisted should be stored in the distributed datastore. For more
information see the documentation on the runtime sandbox.
As of GAE SDK 1.6.1 it's not possible. If you guys later find the way to do it please let me know.
I created a web2py app that is extremely light, with the goal of eventually making the app support JSON-RPC calls, and maybe a few other things.
I found some tutorial online that (on winxp) had me get the source code for web2py and extract it on top of the compiled program. At the top level, I edited app.yaml with my program name and used the GAE SDK to upload the program. It looks like that uploaded everything including example applications. I think it's including a whole gluon directory, and other dir's full of py files. Is there a way to setup web2py to only upload my application, and what's minimally required to run it?
The app.yaml that comes with web2py includes a section skip_files and it should contain, among others, this line:
(applications/(admin|examples)/.*)|
You can change it to
(applications/(admin|examples|welcome)/.*)|
So that welcome app is not deployed. You add more apps that you may have and do not want deployed.
At minimum you need:
web2py/gaehandler.py
web2py/gluon/* (and subfolders, this is web2py)
web2py/applications/theoneappyouwanttodeploy/* (and subfolders)