bad import "syscall" for cloud storage APIs - google-app-engine

I am following the instructions on https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download to begin migrating some code from the, now deprecated, Files API to the new Cloud Storage API without success.
The steps I'm following are ...
I'm running appengine v1.9.23 which is later than the required appengine v1.8.1.
My $GOPATH is set, so I skip step #1.
I proceed to step #2:
goapp get -u golang.org/x/oauth2
goapp get -u google.golang.org/cloud/storage
I am not developing on a managed VM, so I skip step #3.
Now when I run the application, I get:
go-app-builder: Failed parsing input: parser: bad import "syscall" in goapp/src/golang.org/x/net/internal/nettest/error_posix.go
What am I doing wrong?
Steps to reproduce
Download an install the Google Appengine runtime, version 1.9.23 from https://console.cloud.google.com/storage/browser/appengine-sdks/featured/ . Follow the installation instructions documented on https://cloud.google.com/appengine/downloads?hl=en
Create an appengine project directory:
% mkdir $HOME/myapp
Create a new app.yaml file as ~/myapp/app.yaml. Read the directions on the Google website for details: https://cloud.google.com/appengine/docs/go/config/appconfig
I use a version that does not have the static resources:
application: myapp
version: alpha-001
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
Create a location for the Go source files.
% mkdir $HOME/myapp/go
Set your GOPATH to the location of your sources
% export GOPATH=$HOME/myapp/go
Get the Go appengine example project: https://github.com/golang/example
% goapp get github.com/golang/example/appengine-hello
This command will download the example app to the first path entry in the GOPATH
Install the Google Cloud Storage client libraries as directed in https://cloud.google.com/appengine/docs/go/googlecloudstorageclient/download . Reference the steps at the top of this question for more details. Following the directions should result in you running 2 commands:
% go get -u golang.org/x/oauth2
% go get -u google.golang.org/cloud/storage
Attempt to run your go application
% goapp serve
You will see the following compilation error (no stack trace):
2015/12/23 10:37:07 go-app-builder: Failed parsing input: parser: bad import "syscall" in go/src/golang.org/x/net/ipv6/control_unix.go

This error is caused by either of two scenarios:
1) Implicitly importing syscall by importing another package that uses it, as referenced in this related question.
2) Having your package source files in your GOPATH located in a directory at or below the same level as your project's app.yaml (eg. app.yaml in ~/go, and packages sources in ~/go/gopath/src). If a package like x/net/internal/nettest exists in your GOPATH the syscall import will be parsed by goapp at compile time and throw the compilation error.
Avoiding these two scenarios should be sufficient to prevent any bad import "syscall" errors or related compilation errors.

Reproduced the initial steps above and got a similar error, even if not explicitly mentioning syscall. However, running “goapp serve” in the appengine-hello directory results in no error at all.
Adam’s explanation at point 2 applies here correctly: one needs to place the app.yaml file at the right level in the directory structure.

sirupsen/logrus references syscall.
They have an appengine tag specified, not to include syscall so it's usable in AppEngine, something like go build -tags appengine as per issue 310.
However I haven't yet succeeded including it in an AppEngine project so that this build param could be forwarded and specified somewhere so that it goes through. I'll come back to update if I manage.

Related

Unable to provide custom app.yaml file for Java GCP App Engine project

I have a Java 11 GCP App Engine project and I'm trying to use different app.yaml files depending on the environment (e.g., app-dev.yaml, app-prod.yaml, etc). The yaml files in the /appengine directory like src/main/appengine/app-dev.yaml and so on.
There is an SO post about this already, but the answer doesn't work because it clobbers the descriptor which in Java should be the pom.xml (see my Approach 2 for more information).
Approach #1
UPDATE: Solved! In order to use this approach you must be on gcloud 298.0.0+
First, I tried using the --appyaml=APPYAML argument found in https://cloud.google.com/sdk/gcloud/reference/app/deploy#--appyaml:
gcloud app deploy [DEPLOYABLES …] [--appyaml=APPYAML] [--bucket=BUCKET] ...
I ran the following and received an error that the appyaml argument isn't recognized.
$ gcloud --project=my-project app deploy --appyaml=app-dev.yaml
ERROR: (gcloud.app.deploy) unrecognized arguments: --appyaml=app-dev.yaml
The fully qualified path to app-dev.yaml doesn't work either.
Approach #2
Next I found a slighly different syntax in https://cloud.google.com/appengine/docs/flexible/java/configuring-your-app-with-app-yaml that looks like this:
gcloud app deploy service-name-app.yaml
I tried the same locally but pointed to my custom app-dev.yaml like so, but it breaks:
$ gcloud --project=my-project app deploy src/main/appengine/app-dev.yaml
...
descriptor: [/Users/SomeDev/IdeaProjects/my-project/app-server/src/main/appengine/app-dev.yaml]
source: [/Users/SomeDev/IdeaProjects/my-project/app-server/src/main/appengine]
target project: [my-project]
target service: [default]
target version: [20200831abcdefg]
target url: [https://my-project.uc.r.appspot.com]
This breaks because it thinks the app-dev.yaml is the descriptor file instead of a pom.xml, so it errors out with the following:
Error message: did not find any jar files with a Main-Class manifest entry
To compare, I ran a normal deployment without a custom yaml file and you can see the pom.xml is the value of the descriptor.
$ gcloud --project=my-project app deploy
...
descriptor: [/Users/SomeDev/IdeaProjects/my-project/app-server/pom.xml]
source: [/Users/SomeDev/IdeaProjects/my-project/app-server]
target project: [my-project]
target service: [default]
target version: [20200831abcdefg]
target url: [https://my-project.uc.r.appspot.com]
Is there a recommended way to make this work, or is this the wrong approach entirely?
Looking at your "Approach #1" you have to upgrade your gcloud to version >= 298.0.0 here --appyaml parameter have been added - quite recently in Jun'20.
Looking at your "Approach #2". If you are running gcloud app deploy (without parameters) it search for descriptor app.yaml in current directory and if not found - than for pom.xml. If you want to use pom.xml from different localization you have to remove it from current directory. I didn't test it to the very end, just tested the descriptor value in summary.
Anyway I don't think using above is best way to do it. When you use pom.xml as descriptor it means that you are using feature called "deploy your Maven project as source code". Which is not main way to deploy app engine with maven.
According to my understanding if maven was used for build, its possible to use the jar in entrypoint of app.yaml file (reference) or maven goal appengine:deploy (reference + article that should be interesting).

Spacy on AppEngine standard

I'm trying to use Spacy on the new AppEngine Standard Python 3.7 runtime.
When I try to deploy I get:
ERROR: (gcloud.app.deploy) Cannot upload file
[/my/project/path/venv/lib/python3.7/site-packages/spacy/lang/tr/lemmatizer.py],
which has size [41523943] (greater than maximum allowed size of
[33554432]). Please delete the file or add to the skip_files entry in
your application .yaml file and try again.
A few oddities:
The docs seem to indicate that I don't need to upload the virtual environment and it will be created from requirements.txt
Looking at the log file, it seems to ignore .pyc files, but not the venv directory
The error message says to add to the skip_files in your application .yaml file and try again., but the docs say the python3.7 runtime doesn't use skip files and to use a .gcloudignore file instead, but adding venv/ or venv/* doesn't work (it appears to be ignored)
To fix this, I needed up update gcloud and reauthenticate:
gcloud components update
gcloud auth login

Google App Engine goapp "deploy" works but "serve" fails, what's going on?

I started using Google App Engine today, so hopefully the answer will be obvious to someone with experience. After simply following this method to get App Engine interfacing with Cloud SQL, I'm having a frustrating issue: goapp deploy succeeds but goapp serve fails (using the cloud shell).
Here's the error:
XXXX#cloudshell:~/src/XXXX/app-engine (XXXX)$ goapp serve app.yaml
INFO 2018-06-21 07:29:08,115 devappserver2.py:764] Skipping SDK update check.
INFO 2018-06-21 07:29:08,235 api_server.py:268] Starting API server at: http://0.0.0.0:60628
INFO 2018-06-21 07:29:08,305 dispatcher.py:199] Starting module "default" running at: http://0.0.0.0:8080
INFO 2018-06-21 07:29:08,306 admin_server.py:116] Starting admin server at: http://0.0.0.0:8000
ERROR 2018-06-21 07:29:12,254 go_runtime.py:181] Failed to build Go application: (Executed command: /google/go_appengine/goroot/bin/go-app-builder -app_base /home/XXX/app-engine -arch 6 -dynamic -goroot /google/go_appengine/goroot -gopath /home/XXX/gopath:/google/gopath -nobuild_files ^^$ -incremental_re
build -unsafe -binary_name _go_app -extra_imports appengine_internal/init -work_dir /tmp/tmpT1RTRMappengine-go-bin -gcflags -I,/google/go_appengine/goroot/pkg/linux_amd64_appengine -l
dflags -L,/google/go_appengine/goroot/pkg/linux_amd64_appengine hello.go)
/home/XXX/gopath/src/github.com/go-sql-driver/mysql/connection.go:12: can't find import: "context"
2018/06/21 07:29:09 Can't find package "context" in $GOPATH: cannot find package "context" in any of:
/google/go_appengine/goroot/src/context (from $GOROOT)
/home/XXX/gopath/src/context (from $GOPATH)
/google/gopath/src/context
2018/06/21 07:29:12 go-app-builder: build timing: 0×skip (3ms total), 9×compile (2.731s total), 0×link (0 total)
2018/06/21 07:29:12 go-app-builder: failed running compile: exit status 2
I'm suspicious that the version of go being used is wrong. The stated version is 1.6.3 but as I understand "context" requires 1.7. Here's the relevant cloud shell output for that:
XXXX#cloudshell:~ (XXX)$ goapp version
go version go1.6.3 (appengine-1.9.48) linux/amd64
XXXX#cloudshell:~ (XXX)$ go version
go version go1.10 linux/amd64
However, despite my best efforts I have been unable to find any resources on how to manipulate the "goapp" go version.
My app.yaml includes:
runtime: go
api_version: go1.8
And goapp get yields this:
XXXX#cloudshell:~/src/XXX/app-engine (XXX)$ goapp get
package context: unrecognized import path "context" (import path does not begin with hostname)
The go file itself is an exact copy of the demo from the linked above.
As confirmed on a comment on the question the solution was using dev_appserver.py, instead of goapp serve.
goapp is the old tool to handle GAE tasks. Now it's recommended to use dev_appserver.py for the local development, and gcloud app commands for tasks in the cloud (e.g. gcloud app deploy instead of goapp deploy).
Note that goapp is not deprecated, but dev_appserver.py provides more flexibility on local dev, while gcloud app uses the properly documented and quite useful App Engine Admin API (unlike the older tools), making debugging a lot easier in case something goes wrong with your deployment.
In Google App Engine, if you need to get a context, you should probably use google.golang.org/appengine.NewContext() for a context during an HTTP request, or google.golang.org/appengine.BackgroundContext() if it's outside a request. Both return context.Context, which is the same that you would get by creating a context through the context package, but with a context prepared to be used within GAE.
This does not tell you why the imports are not working (other than goapp looks to be using 1.6), but will just work around the issue avoiding you to import any context package.

App Engine deploy with Go libraries

I'm new on Google App Engine. And, I'm getting an issue that I can't solve.
I've a very simple app (developped in Go) like this :
main/
| model/
| | user.go
| main.go
| app.yaml
These are the imports of main.go :
import (
"github.com/julienschmidt/httprouter"
"log"
"net/http"
)
My code works well when I run it locally.
But, when I try to publish it on my Google App Engine instance, I receive this error :
$ gcloud app deploy
You are about to deploy the following services:
- <MY_APP_ENGINE_URL> (from [<MY_LOCAL_YAML_PATH>])
Deploying to URL: [<MY_APP_ENGINE_URL>]
Do you want to continue (Y/n)? Y
Beginning deployment of service [default]...
Some files were skipped. Pass `--verbosity=info` to see which ones.
You may also view the gcloud log file, found at
[<LOCAL_APP_ENGINE_LOG>].
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
2017/05/27 14:48:24 go-app-builder: build timing: 5×compile (301ms total), 0×link (0s total)
2017/05/27 14:48:24 go-app-builder: failed running compile: exit status 2
main.go:4: can't find import: "github.com/julienschmidt/httprouter"
What did I do wrong ?
EDIT :
This is the content of my app.yaml file :
runtime: go
api_version: go1
handlers:
- url: /.*
script: _go_app
App Engine environment doesn't contain your dependencies, you can add an script to do a go get ... for each one but it's too hacky and Go has a solution for that, we can save our dependencies in a vendor folder on the root of our project.
Quick solution:
# Instal godep:
go get -v -u github.com/tools/godep
cd your/project/path
godep save
Now try to deploy again, you'll see a vendor folder in your project, don't remove it and add it to your git source, that folder contains all your third party dependencies like your httprouter (it's my favorite :) )
Note You can use other tools to save your dependencies
I haven't used the gcloud tool, but back in the day when goapp was the tool you had to create github.com/julienschmidt/httprouter (with the lib's source in it, of course) directly under you'r main and then deploy.
AFAIK the App Engine's go version is currently 1.6 which means that while the vendoring is on by default, it can be switched off - perhaps thats the case and thats why #Yandry Pozo's suggestion doesn't work.

Error with module using Cloud Storage with Python and his tutorial

I'm trying to test Google Cloud Storage to store images (I need it in an app that I'm developing) and I'm following the Bookshelf App tutorial that they have in his webpage.
I'm using python and the problem is that when I execute the requirementes.txt all packages have been installed fine, but when I try execute the code, I see this error:
...sandbox.py", line 948, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named cryptography.hazmat.bindings._openssl
I have been trying hundred of posibles solutions, reinstalling only the cryptography package, trying to use different versions of the same module, and installing other packages that contains it but anything resolved the problem.
The requirements contains this:
Flask==0.10.1
gcloud==0.9.0
gunicorn==19.4.5
oauth2client==1.5.2
Flask-SQLAlchemy==2.1
PyMySQL==0.7.1
Flask-PyMongo==0.4.0
PyMongo==3.2.1
six==1.10.0
I'm sure that it is a simple error but I don't find the way to solve it.
Any help will be welcome. Thanks.
EDIT:
When I try do this with a python program this work fine:
import os
from gcloud import storage
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key.json'
client = storage.Client(project='xxxxxxx')
bucket = client.get_bucket('yyyyyyy')
f = open('profile.jpg', 'rb')
blob = bucket.blob(f.name)
blob.upload_from_string(f.read(), 'image/jpeg')
url = blob.public_url
print url
Why I don't can use gcloud library without erros in a GAE app?
It seems you're following the bookshelf tutorial, but according to this line in your stacktrace:
...sandbox.py", line 948, in load_module
It hints that you're using dev_appserver.py to run the code. This isn't necessary for Managed VMs/Flexible unless you're using the compat runtime.
If this is the case, the tutorial provides correct instructions:
$ virtualenv env
$ source env/bin/activate
$ pip install -r requirements.txt
$ python main.py
(If this is not the case, please feel free to comment on this with more details about how you're running your application).

Resources