I have been trying to deploy a reinforcement learner that i've contained in a flask app to google app engine. However I am getting an unexpected error from a package I don't use
[Errno 2] No such file or directory: '/sys/fs/cgroup/memory/memory.usage_in_bytes'
My requirements.txt are as follows:
appdirs==1.4.4
appnope==0.1.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
autopep8==1.5.4
backcall==0.2.0
bleach==3.2.1
cffi==1.14.4
decorator==4.4.2
defusedxml==0.6.0
distlib==0.3.1
dm-tree
entrypoints==0.3
filelock==3.0.12
flask
flask_restful
grpcio==1.32.0
gunicorn
gym
jedi
Jinja2
jsonschema==3.2.0
jupyter
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.0
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
lz4
MarkupSafe
mistune==0.8.4
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
nest-asyncio==1.4.3
notebook==6.1.6
numpy==1.19.5
opencv-python-headless
packaging==20.8
pandas
pandocfilters==1.4.3
parso==0.8.1
pbr==5.5.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
prometheus-client==0.9.0
prompt-toolkit==3.0.10
ptyprocess==0.7.0
pycodestyle==2.6.0
pycparser==2.20
Pygments==2.7.4
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2021.1
pyzmq==20.0.0
qtconsole==5.0.1
QtPy==1.9.0
ray==0.8.4
requests
Send2Trash==1.5.0
six==1.15.0
stevedore==3.3.0
tabulate==0.8.9
terminado==0.9.2
testpath==0.4.4
tensorflow==2.2
toml==0.10.2
tornado
traitlets
virtualenv==20.3.1
virtualenv-clone==0.5.4
virtualenvwrapper==4.8.4
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
My yaml is as follows:
runtime: python37
instance_class: B4
manual_scaling:
instances: 1
entrypoint: gunicorn -b :$PORT main:app --timeout 120
I don't use cgroup, and am unsure on how to fix this error, is this a configuration problem and do I need more memory?
Related
I am trying to setup and connect to a Cassandra single node instance using docker and Golang and it is not working.
The closest information I could find to addressing connection issues between the golang gocql package and Cassandra is available here: Cassandra cqlsh - connection refused, however there are many different upvote answers with no clear indication of which is preferred. It is also a protected question (no "me toos"), so a lot of community members seem to be having trouble with this.
This problem should be slightly different, as it is using Docker and I have tried most (if not all of the solutions linked to above).
version: "3"
services:
cassandra00:
restart: always
image: cassandra:latest
volumes:
- ./db/casdata:/var/lib/cassandra
ports:
- 7000:7000
- 7001:7001
- 7199:7199
- 9042:9042
- 9160:9160
environment:
- CASSANDRA_RPC_ADDRESS=127.0.0.1
- CASSANDRA_BROADCAST_ADDRESS=127.0.0.1
- CASSANDRA_LISTEN_ADDRESS=127.0.0.1
- CASSANDRA_START_RPC=true
db:
restart: always
build: ./db
environment:
POSTGRES_USER: patientplatypus
POSTGRES_PASSWORD: SUPERSECRETFAKEPASSD00T
POSTGRES_DB: zennify
expose:
- "5432"
ports:
- 5432:5432
volumes:
- ./db/pgdata:/var/lib/postgresql/data
app:
restart: always
build:
context: .
dockerfile: Dockerfile
command: bash -c 'while !</dev/tcp/db/5432; do sleep 10; done; realize start --run'
# command: bash -c 'while !</dev/tcp/db/5432; do sleep 10; done; go run main.go'
ports:
- 8000:8000
depends_on:
- db
- cassandra00
links:
- db
- cassandra00
volumes:
- ./:/go/src/github.com/patientplatypus/webserver/
Admittedly, I am a little shaky on what listening addresses I should pass to Cassandra in the environment section, so I just passed 'home':
- CASSANDRA_RPC_ADDRESS=127.0.0.1
- CASSANDRA_BROADCAST_ADDRESS=127.0.0.1
- CASSANDRA_LISTEN_ADDRESS=127.0.0.1
If you try and pass 0.0.0.0 you get the following error:
cassandra00_1 | Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: listen_address cannot be a wildcard address (0.0.0.0)!
cassandra00_1 | listen_address cannot be a wildcard address (0.0.0.0)!
cassandra00_1 | ERROR [main] 2018-09-10 21:50:44,530 CassandraDaemon.java:708 - Exception encountered during startup: listen_address cannot be a wildcard address (0.0.0.0)!
Overall, however I think that I am getting the correct start up procedure for Cassandra (afaict) because my terminal outputs that Cassandra started up as normal and is listening on the appropriate ports:
cassandra00_1 | INFO [main] 2018-09-10 22:06:28,920 StorageService.java:1446 - JOINING: Finish joining ring
cassandra00_1 | INFO [main] 2018-09-10 22:06:29,179 StorageService.java:2289 - Node /127.0.0.1 state jump to NORMAL
cassandra00_1 | INFO [main] 2018-09-10 22:06:29,607 NativeTransportService.java:70 - Netty using native Epoll event loop
cassandra00_1 | INFO [main] 2018-09-10 22:06:29,750 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
cassandra00_1 | INFO [main] 2018-09-10 22:06:29,754 Server.java:156 - Starting listening for CQL clients on /127.0.0.1:9042 (unencrypted)...
cassandra00_1 | INFO [main] 2018-09-10 22:06:29,990 ThriftServer.java:116 - Binding thrift service to /127.0.0.1:9160
In my golang code I have the following package that is being called (simplified to show relevant section):
package data
import(
"fmt"
"github.com/gocql/gocql"
)
func create_userinfo_table() {
<...>
fmt.Println("replicating table in cassandra")
cluster := gocql.NewCluster("localhost") //<---error here!
cluster.ProtoVersion = 4
<...>
}
Which results in the following error in my terminal:
app_1 | [21:52:38][WEBSERVER] : 2018/09/10
21:52:38 gocql: unable to dial control conn 127.0.0.1:
dial tcp 127.0.0.1:9042: connect: connection refused
app_1 | [21:52:38][WEBSERVER] : 2018/09/10
21:52:38 gocql: unable to dial control conn ::1:
dial tcp [::1]:9042: connect: cannot assign requested address
app_1 | [21:52:38][WEBSERVER] : 2018/09/10
21:52:38 Could not connect to cassandra cluster: gocql:
unable to create session: control: unable to connect to initial hosts:
dial tcp [::1]:9042: connect: cannot assign requested address
I have tried several variations on the connection address
cluster := gocql.NewCluster("localhost")
cluster := gocql.NewCluster("127.0.0.1")
cluster := gocql.NewCluster("127.0.0.1:9042")
cluster := gocql.NewCluster("127.0.0.1:9160")
These seemed likely candidates for example, but no luck.
Does anyone have any idea what I am doing wrong?
Use the service name cassandra00 for the hostname per the docker-compose documentation https://docs.docker.com/compose/compose-file/#links
Containers for the linked service are reachable at a hostname identical to the alias, or the service name if no alias was specified.
Leave the CASSANDRA_LISTEN_ADDRESS envvar unset (or pass auto) per https://docs.docker.com/samples/library/cassandra/
The default value is auto, which will set the listen_address option in cassandra.yaml to the IP address of the container as it starts. This default should work in most use cases.
I am deploying a Go application using AppEngine flexible. Below is my app.yaml. Sometimes after I deploy it stabilizes at 1 instance (it's a very low load application), but most of the time it constantly respawns upwards of 6 instances. My logs are filled with messages showing the new instances being created. There is nearly zero load on this application, why is AppEngine flexible constantly destroying and respawning instances?
Log showing constant respawning:
app.yaml
runtime: go
api_version: go1
env: flex
handlers:
- url: /.*
script: _go_app
health_check:
enable_health_check: True
check_interval_sec: 10
timeout_sec: 4
unhealthy_threshold: 2
healthy_threshold: 2
automatic_scaling:
min_num_instances: 1
max_num_instances: 10
cool_down_period_sec: 120 # default value
cpu_utilization:
target_utilization: 0.5
The problem was with my health check function. It originally looked like this:
func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
return
}
I then discovered this sentence in the documentation on how instances are managed:
You can write your own custom health-checking code. It should reply to /_ah/health requests with a HTTP status code 200. The response must include a message body, however, the value of the body is ignored (it can be empty).
So I changed the health check function to write a simple "ok" in response:
func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
return
}
The instances now behave according to my autoscale settings! The respawning is gone.
I obviously should have read the documentation closer, but there was zero indication of a problem in the health check logs. All health checks looked like they were passing. Hopefully this info is helpful to others.
I have latest tools installed, in the app.go I have following imports
"github.com/go-martini/martini"
"github.com/martini-contrib/binding"
"github.com/martini-contrib/render"
"github.com/moovweb/gokogiri"
"github.com/moovweb/gokogiri/xml"
And in app.yaml
application: app
version: 1
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
I saved external dependencies using godep save tool. But while trying to execute goapp serve command following error occured
SRSLY-2:app bezigon$ goapp serve
INFO 2015-05-27 09:11:22,645 devappserver2.py:745] Skipping SDK update check.
INFO 2015-05-27 09:11:23,048 api_server.py:190] Starting API server at: http://localhost:57449
INFO 2015-05-27 09:11:23,071 dispatcher.py:192] Starting module "default" running at: http://localhost:8080
INFO 2015-05-27 09:11:23,073 admin_server.py:118] Starting admin server at: http://localhost:8000
ERROR 2015-05-27 09:11:24,772 go_runtime.py:180] Failed to build Go application: (Executed command: /Users/bezigon/google-cloud-sdk/platform/google_appengine/goroot/bin/go-app-builder -app_base /Users/bezigon/go/src/github.com/bezigon/app -arch 6 -dynamic -goroot /Users/bezigon/google-cloud-sdk/platform/google_appengine/goroot -nobuild_files ^^$ -unsafe -gopath /Users/bezigon/go -print_extras_hash Godeps/_workspace/src/github.com/moovweb/gokogiri/html/xpath_test.go Godeps/_workspace/src/github.com/martini-contrib/binding/validate_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/attribute.go Godeps/_workspace/src/github.com/martini-contrib/binding/file_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/document_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xpath/util.go Godeps/_workspace/src/github.com/go-martini/martini/logger.go Godeps/_workspace/src/github.com/martini-contrib/binding/misc_test.go Godeps/_workspace/src/github.com/go-martini/martini/response_writer_test.go Godeps/_workspace/src/github.com/codegangsta/inject/inject_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/text.go Godeps/_workspace/src/github.com/go-martini/martini/go_version.go Godeps/_workspace/src/github.com/go-martini/martini/router_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/nodeset.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xpath/xpath_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xpath/xpath.go Godeps/_workspace/src/github.com/martini-contrib/binding/errors.go Godeps/_workspace/src/github.com/moovweb/gokogiri/mem/mem_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/help/help_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/fragment_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/node_test.go Godeps/_workspace/src/github.com/martini-contrib/render/render.go Godeps/_workspace/src/github.com/moovweb/gokogiri/help/help.go Godeps/_workspace/src/github.com/moovweb/gokogiri/mem/mem.go Godeps/_workspace/src/github.com/go-martini/martini/recovery.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/document.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/document.go Godeps/_workspace/src/github.com/moovweb/gokogiri/gokogiri.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/element.go Godeps/_workspace/src/github.com/martini-contrib/binding/errors_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/util/util_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/fragment.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/attribute_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/css/css_test.go Godeps/_workspace/src/github.com/oxtoacart/bpool/bytepool.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/fragment_test.go Godeps/_workspace/src/github.com/martini-contrib/binding/errorhandler_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/utils_test.go Godeps/_workspace/src/github.com/oxtoacart/bpool/bpool.go Godeps/_workspace/src/github.com/moovweb/gokogiri/util/util.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/utils_test.go Godeps/_workspace/src/github.com/go-martini/martini/response_writer.go Godeps/_workspace/src/github.com/moovweb/gokogiri/css/css.go Godeps/_workspace/src/github.com/go-martini/martini/env.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/search_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/gokogiri_test.go Godeps/_workspace/src/github.com/go-martini/martini/static_test.go Godeps/_workspace/src/github.com/codegangsta/inject/inject.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xpath/expression.go Godeps/_workspace/src/github.com/martini-contrib/binding/form_test.go Godeps/_workspace/src/github.com/go-martini/martini/static.go Godeps/_workspace/src/github.com/go-martini/martini/martini.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/fragment.go Godeps/_workspace/src/github.com/martini-contrib/binding/bind_test.go app.go Godeps/_workspace/src/github.com/martini-contrib/render/render_test.go Godeps/_workspace/src/github.com/martini-contrib/binding/binding.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/cdata.go Godeps/_workspace/src/github.com/go-martini/martini/martini_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/node_test.go Godeps/_workspace/src/github.com/oxtoacart/bpool/bufferpool.go Godeps/_workspace/src/github.com/martini-contrib/binding/json_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xpath/util_test.go Godeps/_workspace/src/github.com/go-martini/martini/router.go Godeps/_workspace/src/github.com/martini-contrib/binding/common_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/pi.go Godeps/_workspace/src/github.com/go-martini/martini/logger_test.go Godeps/_workspace/src/github.com/go-martini/martini/recovery_test.go Godeps/_workspace/src/github.com/go-martini/martini/return_handler.go Godeps/_workspace/src/github.com/moovweb/gokogiri/help/util_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/document_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/crash_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/comment.go Godeps/_workspace/src/github.com/martini-contrib/binding/multipart_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/xml/node.go Godeps/_workspace/src/github.com/go-martini/martini/env_test.go Godeps/_workspace/src/github.com/moovweb/gokogiri/html/encoding_test.go)
2015/05/27 12:11:24 go-app-builder: Failed parsing input: package github.com/moovweb/gokogiri/html required, but all its files were excluded by nobuild_files
What should I do?
By the way it looks to https://blog.golang.org/the-app-engine-sdk-and-workspaces-gopath, but I have not found the answer.
UPD: I have added a testcase, check it out https://github.com/bezigon/testcase
Following link has the answer
https://groups.google.com/d/msg/google-appengine-go/mGuMV4tVR0U/A3qaCau7zY4J
Briefly, the reason is
"cgo" is not supported on classic App Engine. If you want to build your app, you will need to use Go on Managed VMs ("vm: true" in your app.yaml).
Please see https://cloud.google.com/appengine/docs/go/managed-vms/ for more information.
I have a php app, for the app engine. My app.yaml file is the following:
application: myproject-testing-112
version: 1
runtime: php
api_version: 1
threadsafe: yes
handlers:
- url: /edit.php
script: edit.php
....
- url: /profile.php
script: profile.php
my problem is that I have about 300 urls (i have 300 php files). As I can see app.yaml allows you 100 URLS. How to deal with this problem? Any ideas?
You can use regex patterns with PHP GAE just like in the Python GAE example referenced in the comment. So, give the following a try:
application: myproject-testing-112
version: 1
runtime: php
api_version: 1
threadsafe: yes
handlers:
- url: /(.+\.php).*
script: \1
To be clear, this one entry will map every URL that matches to a PHP file of the same name.
/edit.php --> edit.php
/profile.php --> profile.php
/profile.php/foobar --> profile.php (the ".*" at the end of the regex allows this behavior)
/someOtherPath will not match the above entry at all, since it doesn't have ".php"
Obviously, you can tweak the regular expression to get the exact behavior you're looking for.
Could anyone point to roughly where in the python sdk code static routes get loaded into or accessed by http_server. This is to debug a failure to load static images. In eclipse I can see the static routes loading into var appinfo from the yaml file, and later can see the dynamic routes being checked during a request, but having trouble following the in-between steps.
Thanks
Update 11/30
Previously tried variations on the yaml, path, etc that were suggested in some docs and postings.
Here is one of them. In this case there is no 404 error, but image doesn't load and Firebug reports "Failed to the load the given URL".
app.yaml
application: crazywidget2
version: 1
runtime: python27
api_version: 1
threadsafe: false
handlers:
- url: /images
static_dir: /images
secure: always
-url: /.*
script: crazywidget2.py
secure: always
libraries:
- name: jinja2
version: latest
index.html
...
<img src="/images/xyz.gif" alt="XYZ illustration" />
...
crazywidget2.py
...
class MainPage(webapp2.RequestHandler):
def get(self):
template = jinja_environment.get_template('index.html')
self.response.out.write(template.render({}))
...
...
app = webapp2.WSGIApplication([('/script_send', ScriptSend),
('/resetkey', ResetKey),
('/admin', Admin),
('/start', Start),
('/', MainPage)],
debug=True)
def main():
app.run()
if __name__=='__main__':
main()
Update 12/3
Turns out that in the above case it works if the static_dir is relative, "images" instead of "/images". In the absolute case it tries to open that path as is. Maybe some other variations would work as well.
Here are three relevant code pointers (all in google/appengine/tools/dev_appserver.py):
class FileDispatcher
CreateURLMatcherFromMaps()
DevAppServerRequestHandler._Dispatch()
I would assume though, there are easier ways to debug your problem. If you would post your app.yaml and the path you access and the response you get, people here could start to help you.
Just a hunch: does it work if you add a slash to the end of /images in your handlers? Try replacing /images with /images/ in both places that you use it in app.yaml.