Execution of this this line:
logging = LoggingOptions.getDefaultInstance().getService()
On Google App Engine standard environment instance results with:
com.google.api.server.spi.SystemService invokeServiceMethod: exception occurred while calling backend method (SystemService.java:398)
java.lang.NoClassDefFoundError: java.nio.channels.spi.SelectorProvider is a restricted class. Please see the Google App Engine developer's guide for more details.
at java.nio.channels.spi.SelectorProvider.<clinit>(SelectorProvider.java)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:187)
at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:171)
at io.grpc.internal.SharedResourceHolder.getInternal(SharedResourceHolder.java:121)
at io.grpc.internal.SharedResourceHolder.get(SharedResourceHolder.java:91)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:311)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:280)
at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:230)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:239)
at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:123)
at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:110)
at com.google.api.gax.grpc.ProviderManager.getChannel(ProviderManager.java:105)
at com.google.api.gax.grpc.ChannelAndExecutor.create(ChannelAndExecutor.java:67)
at com.google.api.gax.grpc.ClientSettings.getChannelAndExecutor(ClientSettings.java:81)
at com.google.cloud.logging.spi.v2.ConfigServiceV2Client.<init>(ConfigServiceV2Client.java:131)
at com.google.cloud.logging.spi.v2.ConfigServiceV2Client.create(ConfigServiceV2Client.java:122)
at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:141)
at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:503)
at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:490)
Is there any way to get it to work?
You will have to use the v2 api of google-api-services. Explanation:
Google provides two sets of supported libraries for interfacing with their cloud services.
Google API Client Libraries[1] (google-api-services): older libraries that provide a programmatic overlay on top of each API's REST interface
Google Cloud Libraries[2] (google-cloud): alpha/beta libraries that are based on gRPC. They are allegedly much more performant, but are NOT supported in Standard Appengine (found this little note in their github page.. [3]) because of network security limitations on gRPC.
Each set of libraries now has a v1 and v2 set of apis for both sets of libraries (v1 is discontinued).
google-api-services v2 documentation[4]
google-cloud v2 documentation[5]
Sources:
Main docs page: https://cloud.google.com/apis/docs/client-libraries-explained
[1]: https://developers.google.com/api-client-library/
[2]: http://googlecloudplatform.github.io/google-cloud-java/0.10.0/index.html
[3]: https://github.com/GoogleCloudPlatform/google-cloud-java
[4]: http://googlecloudplatform.github.io/google-cloud-java/0.10.0/apidocs/index.html
[5]: https://developers.google.com/resources/api-libraries/documentation/logging/v2beta1/java/latest/com/google/api/services/logging/v2beta1/package-summary.html
Related
We are using Spring Cloud Gateway [JavaDsl] as our API Gateway. For the proxies we have multiple microservices [running on different ip:port] as target's. Would like to know either we can configure multiple targets to spring cloud gateway proxies, similar to apache camel load balancer eip.
camel.apache.org/manual/latest/loadBalance-eip.html
We are looking for software load balancing with in spring cloud gateway [similar to netflix/apache-camel] instead of another dedicated LB.
Able to get Spring Cloud Gateway load balanced Route working using spring-cloud-starter-netflix-ribbon. However when one of the server instance is down, load-balancing-fails. Code Snippets below.
Version :
spring-cloud-gateway : 2.1.1.BUILD-SNAPSHOT
Gateway Route
.route(
r ->
r.path("/res/security/")
.filters( f -> f
.preserveHostHeader()
.rewritePath("/res/security/", "/targetContext/security/")
.filter(new LoggingFilter())
.uri("lb://target-service1-endpoints")
)
application.yml
ribbon:
eureka:
enabled: false
target-service1-endpoints:
ribbon:
listOfServers: 172.xx.xx.s1:80, 172.xx.xx.s2:80
ServerListRefreshInterval: 1000
retryableStatusCodes: 404, 500
MaxAutoRetriesNextServer: 1
management:
endpoint:
health:
enabled: true
Here is the response from Spring Cloud Team.
What you've described, indeed, happens. However, it is not gateway-specific. If you just use Ribbon in a Spring Cloud project with listOfServers, the same thing will happen. This is because, unlike for eureka, the IPing for non-discovery-service scenario is not instrumented (a DummyPing instance is used).
You could probably change this behaviour by providing your own IPing, IRule or ServerListFilter implementation and overriding the setup we provide in the autoconfiguration in this way.
https://github.com/spring-cloud/spring-cloud-gateway/issues/1482
I have a google app engine project in python 3.7 in which I would like to write some logs.
I am used to program in app engine python 2.7 and I was using the simple code:
logging.info('hi there!')
to write any log onto the google cloud log console.
That command above now doesn't work anymore and it says:
logging has no attribute 'info'
I searched and I found this possible new code
from flask import Flask
from google.cloud import logging
app = Flask(__name__)
#app.route('/l')
def hello():
logging_client = logging.Client()
log_name = LOG_NAME
logger = logging_client.logger(LOG_NAME)
text = 'Hello, world!'
logger.log_text(text, severity='CRITICAL')
return text
This code above doesn't give any error in the stack-driver report page BUT it displays nothing at all in the log page.
So how can I write a log for my app engine project in python3.7?
The second generation standard environment (which includes python 3.7) is IMHO closer to the flexible environment than to the first generation standard environment (which includes python 2.7).
Many of the APIs which had customized versions for the 1st generation (maintained by the GAE team) were not (or at least not yet) ported in the 2nd generation if the respective functionality was more or less covered by alternate, more generic approaches, already in use in the flexible environment (most of them based on services developed and maintained by teams other than the GAE one).
You'll notice the similarity between many of the service sections in these 2 migration guides (which led me to the above summary conclusion):
Understanding differences between Python 2 and Python 3 on the App Engine standard environment
Migrating Services from the Standard Environment to the Flexible Environment (i.e 1st generation standard to flexible)
Logging is one of the services listed in both guides. The 1st generation used a customized version of the standard python logging library (that was before Stackdriver became a standalone service). For the 2nd generation logging was simply delegated to using the now generally available Stackdriver logging service (which is what the snippet you shown comes from). From Logging (in the 1st guide):
Request logs are no longer automatically correlated but will still
appear in Stackdriver Logging. Use the Stackdriver Logging client
libraries to implement your desired logging behavior.
The code snippet you show corresponds, indeed, to Stackdriver Logging. But you seem to be Using the client library directly. I don't know if this is a problem (GAE is often a bit different), but maybe you could also try using the standard Python logging instead:
Connecting the library to Python logging:
To send all log entries to Stackdriver by attaching the Stackdriver
Logging handler to the Python root logger, use the setup_logging
helper method:
# Imports the Google Cloud client library
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
# Connects the logger to the root logging handler; by default this captures
# all logs at INFO level and higher
client.setup_logging()
Using the Python root logger:
Once the handler is attached, any logs at, by default, INFO level or
higher which are emitted in your application will be sent to
Stackdriver Logging:
# Imports Python standard library logging
import logging
# The data to log
text = 'Hello, world!'
# Emits the data using the standard logging module
logging.warn(text)
There are some GAE-specific notes in there as well (but I'm not sure if they also cover the 2nd generation standard env):
Google App Engine grants the Logs Writer role by default.
The Stackdriver Logging library for Python can be used without needing
to explicitly provide credentials.
Stackdriver Logging is automatically enabled for App Engine
applications. No additional setup is required.
Note: Logs written to stdout and stderr are automatically sent to Stackdriver Logging for you, without needing to use
Stackdriver Logging library for Python.
Maybe worth noting that the viewing the logs would likely be different as well outside the 1st generation standard env (where app logs would be neatly correlated to the request logs).
And there's also the Using Stackdriver Logging in App Engine apps guide. It doesn't specifically mention the 2nd generation standard env (so it might need an update) but has good hints for the flexible environment which might be useful. For example the Linking app logs and requests section might be of interest if the missing request logs correlation has anything to do with it.
Even though logging works differently in Python 2.7 and 3.7, the same method of logging provided in Reading and Writing Application Logs in Python 2.7 should work for Python 3.7 as well since logs written to stdout and stderr will still appear in the Stackdriver Logging.
Import logging
logging.getLogger().setLevel(logging.DEBUG)
logging.debug('This is a debug message')
logging.getLogger().setLevel(logging.INFO)
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
#logging.warn is deprecated
logging.warn('This is a warning' message)
=======================================
Import logging
app.logger.setLevel(logging.ERROR)
app.logger.error('This is an error message')
However, log entries are no longer automatically correlated with requests as in Python 2.7, which is why you see them in plain text. I have created a feature request to address this which you can follow here.
I'm getting this error on app engine using flask to make a Slack bot. It happens whenever I send a POST request from Slackbot.
Unfortunately, the url provided in the error is a dead link. How do I go about using sockets instead of URLFetch?
/base/data/home/apps/[REDACTED]/lib/requests/packages/urllib3/contrib/appengine.py:115:
AppEnginePlatformWarning: urllib3 is using URLFetch on Google App
Engine sandbox instead of sockets. To use sockets directly instead of
URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
As detailed on Google's Sockets documentation, sockets can be used by setting the GAE_USE_SOCKETS_HTTPLIB environment variable. This feature seems to be available only on paid apps, and impacts billing.
Though the error you posted gets logged as an Error in App Engine, this thread suggests (see reply #8) that the error is actually meant as a warning, which the text "AppEnginePlatformWarning" seems to suggest anyway.
The comment block on the source page for appengine.py is also instructive.
You didn't post any information about your implementation, but on Google App Engine Standard edition, using URLFetch via the AppEngineManager should be just fine, though you will get the error.
You can use the following to silence this:
import warnings
import urllib3.contrib.appengine
warnings.filterwarnings('ignore', r'urllib3 is using URLFetch', urllib3.contrib.appengine.AppEnginePlatformWarning)
For me, turns out the presence of requests_toolbelt dependency in my project was the problem: it somehow forced the requests library to use urllib3, which requires URLFetch to be present, otherwise it raises an AppEnginePlatformError. As suggested in the app engine docs, monkey-patching Requests with requests_toolbelt forces the former to use URLFetch, which is no longer supported by GAE in a Python 3 runtime.
The solution was to remove requests_toolbelt from my requirements.txt file
I want to put a scraping service using Apache HttpClient to the Cloud. I read problems are possible with Google App Engine, as it's direct network access and threads creation are prohibited. What's about other cloud hosting providers? Have anyone experince with Apache HttpClient + cloud?
AppEngine has threads and direct network access (HTTP only). There is a workaround to make it work with HttpClient.
Also, if you plan to use many parse tasks in parallel, you might check out Task Queue or even mapreduce.
Btw, there is a "misfeature" in GAE that you can not fully set custom User-agent header on your requests - GAE always adds "AppEngine" to the end of it (this breaks requests to certain sites - most notably iTunes).
It's certainly possible to create threads and access other websites from CloudFoundry, you're just time limited for each process. For example, if you take a look at http://rack-scrape.cloudfoundry.com/, it's a simple rack application that inspects the 'a' tags from Google.com;
require 'rubygems'
require 'open-uri'
require 'hpricot'
run Proc.new { |env|
doc = Hpricot(open("http://www.google.com"))
anchors = (doc/"a")
[200, {"Content-Type" => "text/html"}, [anchors.inspect]]
}
As for Apache HttpClient, I have no experience of this but I understand it isn't maintained any more.
I created one google app engine project. There I configured DWR(DirectWebRemoting).
I have created one ajax functionality which will checks the username and password of the user. But it does give me any output rather it gives Error:
dwr is not defined
Source File: http://localhost:8081/dwr.jsp
Line: 16
Where as it works fine in Tomcat web application.
So My question is that does Google app engine not support DWR configuration?
DWR is compatible with GAE according to "Will it play in Java" (broken link. possibly: a list of GAE-compatible java libraries) page.
Several links to threads with proof of success and potential limitations:
thread about "DWR 3 RC1" from "users#dwr.java.net"
StackOverflow question "DWR sometimes die on the GAE server"
Most useful post in "DWR 3 RC1" thread is this one (about potential problems).
Do you require reverse ajax? (Current implementation uses a thread to clean up expired sessions. Current implementation uses javax.swing.event.EventListenerList which is blacklisted by GAE/J.)
Do you require file upload? (Can potentially write to disk)
Do you require file download? (Current implementation uses a thread to clean up expired downloads)
If you don't need anything from the above, then you could replace problematic classes with dummy implementation.
I don't think this is supported, but the same functionality is offered by channels :
https://developers.google.com/appengine/docs/java/channel/