I successfully deployed a twitter screenshot bot on Google App Engine.
This is my first time deploying.
First thing I noticed was that the app didn't start running until I clicked the link.
When I did, the app worked successfully (replied to tweets with screenshots) as long as the tab was loading and open.
When I closed the tab, the bot stopped working.
Also, in the cloud shell log, I saw:
Handling signal: term
[INFO] Worker exiting (pid 18)
This behaviour surprises me as I expect it to keep running on google server indefinitely.
My bot works by streaming with Twitter api. Also the "worker exiting" line above surprises me.
Here is the relevant code:
def get_stream(set):
global servecount
with requests.get(f"https://api.twitter.com/2/tweets/search/stream?tweet.fields=id,author_id&user.fields=id,username&expansions=author_id,referenced_tweets.id", auth=bearer_oauth, stream=True) as response:
print(response.status_code)
if response.status_code == 429:
print(f"returned code 429, waiting for 60 seconds to try again")
print(response.text)
time.sleep(60)
return
if response.status_code != 200:
raise Exception(
f"Cannot get stream (HTTP {response.status_code}): {response.text}"
)
for response_line in response.iter_lines():
if response_line:
json_response = json.loads(response_line)
print(json.dumps(json_response, indent=4))
if json_response['data']['referenced_tweets'][0]['type'] != "replied_to":
print(f"that was a {json_response['data']['referenced_tweets'][0]['type']} tweet not a reply. Moving on.")
continue
uname = json_response['includes']['users'][0]['username']
tid = json_response['data']['id']
reply_tid = json_response['includes']['tweets'][0]['id']
or_uid = json_response['includes']['tweets'][0]['author_id']
print(uname, tid, reply_tid, or_uid)
followers = api.get_follower_ids(user_id='1509540822815055881')
uid = int(json_response['data']['author_id'])
if uid not in followers:
try:
client.create_tweet(text=f"{uname}, you need to follow me first :)\nPlease follow and retry. \n\n\nIf there is a problem, please speak with my creator, #JoIyke_", in_reply_to_tweet_id=tid, media_ids=[mid])
except:
print("tweet failed")
continue
mid = getmedia(uname, reply_tid)
#try:
client.create_tweet(text=f"{uname}, here is your screenshot: \n\n\nIf there is a problem, please speak with my creator, #JoIyke_", in_reply_to_tweet_id=tid, media_ids=[mid])
#print(f"served {servecount} users with screenshot")
#servecount += 1
#except:
# print("tweet failed")
editlogger()
def main():
servecount, tries = 1, 1
rules = get_rules()
delete = delete_all_rules(rules)
set = set_rules(delete)
while True:
print(f"starting try: {tries}")
get_stream(set)
tries += 1
If this is important, my app.yaml file has only one line:
runtime: python38
and I deployed the app from cloud shell with gcloud app deploy app.yaml
What can I do?
I have searched and can't seem to find a solution. Also, this is my first time deploying an app sucessfully.
Thank you.
Google App Engine works on demand i.e. when it receives an HTTP(s) request.
Neither Warmup requests nor min_instances > 0 will meet your needs. A warmup tries to 'start up' an instance before your requests come in. A min_instance > 0 simply says not to kill the instance but you still need an http request to invoke the service (which is what you did by opening a browser tab and entering your Apps url).
You may ask - since you've 'started up' the instance by opening a browser tab, why doesn't it keep running afterwards? The answer is that every request to a Google App Engine (Standard) app must complete within 1 - 10 minutes (depending on the type of scaling) your App is using (see documentation). For Google App Engine Flexible, the timeout goes up to 60 minutes. This tells you that your service will timeout after at most 10 minutes on GAE standard or 60 minutes on GAE Flexible.
I think the best solution for you on GCP is to use Google Compute Engine (GCE). Spin up a virtual server (pick the lowest configuration so you can stick within the free tier). If you use GCE, it means you spin up a Virtual Machine (VM), deploy your code to it and kick off your code. Your code then runs continuously.
App Engine works on demand, i.e, only will be up if there are requests to the app (this is why when you click on the URL the app works). As well you can set 1 instance to be "running all the time" (min_instances) it will be an anti-pattern for what you want to accomplish and App Engine. Please read How Instances are Managed
Looking at your code you're pulling data every minute from Twitter, so the best option for you is using Cloud Scheduler + Cloud Functions.
Cloud Scheduler will call your Function and it will check if there is data to process, if not the process is terminated. This will help you to save costs because instead of have something running all the time, the function will only work the needed time.
On the other hand I'm not an expert with the Twitter API, but if there is a way that instead of pulling data from Twitter and Twitter calls directly your function it will be better since you can optimize your costs and the function will only run when there is data to process instead of checking every n minutes.
As an advice, first review all the options you have in GCP or the provider you'll use, then choose the best one for your use case. Just selecting one that works with your programming language does not necessarily will work as you expect like in this case.
I have an GAE app PHP72, env: standard which is hanging intermittently (once or twice a day for about 5 mins).
When this occurs I see a large spike in GAE dashboard's Traffic Sent graph.
I've reviewed all uses of file_get_contents and curl_exec within the app's scripts, not including those in /vendor/, and don't believe these to be the cause.
Is there a simple way in which I can review more info on these outbound requests?
There is no way to get more details in that dashboard. You're going to need to check your logs at the corresponding times. Obscure things to check for:
Cron jobs coming in at the same times
Task Queues spinning up
With an Instance ID, can the details of the instance can be read from the google console?
To see the startTime of an instance in the Google Cloud Console click on the Navigation Menu > App Engine > Instances. Select the service you are interested on and below the graphic there will be displayed a table with the startTime of each instance.
Or you can run the following command:
gcloud app instances describe <INSTANCE_ID> -s=<SERVICE> -v=<VERSION> | grep startTime
EDIT:
In case the instances no longer exists you can use this filter in the Google Cloud Stackdriver Logging Console:
resource.type="gae_app"
resource.labels.module_id="SERVICE"
resource.labels.version_id="VERSION"
protoPayload.instanceId="INSTANCE_ID"
The timestamp of the first entry will roughly coincide with the StartTime of the instance.
We were running on AppEngine but recently moved over to Managed VMs. For some reason AppStats is no longer available? We just get a 404 not found error when browsing to our appstats URL. Is appstats not supported on Managad VMs? If not, is there a way of isolating poorly performing endpoints within our application?
One way to isolate poorly performing endpoints is to use the advanced filter search in the GCP Logs Viewer. It is a little hard to find at first.
To get there, in your Google Cloud console, navigate to Logging for your project. At the right of the text box for "Filter by label or text search" you will see a small dropdown arrow. Click that and select "Convert to advanced filter". This will allow you to write your own sql-ish query where you can find requests that took longer than n to complete.
For example, add the following to the filter:
protoPayload.latency>"0.300s"
This will return a list of all requests that took longer than 300 milliseconds to process. If you have Cloud Trace enabled, you can click on the request response time to see the timeline for the individual service calls.
Titanium SDK version: 1.6.2
iPhone SDK version: 4.2
I am developing an iOS app in Appcelerator. In this app I am getting all the users facebook friends and send them (all) to my API (built in Rails3). If a user got alot of friends this requests takes a long time and time out sometimes.
How can I alter my code so that I can send the facebook friend ids in "batches"?
This is my code: http://pastie.org/2043126
Thankful for all input!
When you get the response from Facebook, parse it and store the ids in an array.
Loop through the array, pop X ids, and ApiPOST them.
At the end of the onload function for this request, get the next X items until the array is empty.
When the array is empty do the table.setData().
You can also launch multiple xhr requests at the same time using different xhr objects. In one of my Appcelerator apps, I have a pool of 10 xhr requests caching images at the same time.