Non-interactive auto-refresh stale OAuth Token with Googlesheets package - launchd

I'm trying to automatically run an r script to download a private Google Sheet every hour. It always works fine when I'm interactively using R. It also works fine during the first hour after I automate the script with launchd.
It stops working an hour after I start automating it with launchd. I think the problem is that after one hour the access token changes, and the non-interactive version isn’t waiting for the auto refreshing of the OAuth token. Here is the error that I get from the error report:
Auto-refreshing stale OAuth token.
Error in gzfile(file, mode) : cannot open the connection
Calls: gs_auth ... -> -> cache_token -> saveRDS -> gzfile
In addition: Warning message:
In gzfile(file, mode) :
cannot open compressed file '.httr-oauth', probable reason 'Permission denied'
Execution halted
I'm using Jenny Bryan's googlesheets package. Here is the code that I initially use to register the sheet, and then save the oAuth token:
gToken <- gs_auth() # Run this the first time to get the oAuth information
saveRDS(gToken, "/Users/…/gToken.rds") # Save the oAuth information for non-interactive use
I then use the following script in the file that I automate with launchd:
gs_auth(token = "/Users/…/gToken.rds")
How can I avoid this error when running the script automatically with launchd?

I don't know about launchd but I had the same problem when I wanted to run a R script automatically from the Windows task planer. Changing the 'cache' attribute value to FALSE did the trick for me [1]: https://i.stack.imgur.com/pprlC.png
You can find the solution here: https://github.com/jennybc/googlesheets/issues/262
To authenticate once in the browser in order to get a token file, I did this:
token_file <- gs_auth(new_user = TRUE, cache = FALSE)
saveRDS(token_file, "googlesheets_token.rds")
Automatic login afterwards via:
gs_auth(token = paste0(path_scripts, "googlesheets_token.rds"),
verbose = TRUE, cache = FALSE)

Related

Google AppEngine application log assigned to the wrong request log

When I look at the logs in the Google Log Viewer for my GAE project, I see that often the logs that I write myself in the code are assigned to the wrong request. Most of the time the log is assigned to the request directly after the request that produced the log entry.
As the root of every application log in GAE must be a request, this means that the wrong request is sometimes marked as error, because another request before produced an error, but the log is somehow assigned to the request after that.
I don't really do anything special, I use Ktor as my servlet and have an interceptor that creates a log when an exception occurs before returning status 500.
I use Java logging via SLF4J with the google cloud logging handler, but before that I used logback via SLf4J and had the same problem.
The content of the logs itself is also correct, the returned status of the request, the level of the log entry, the message, everything is ok.
I thought that it may be because I use kotlin and switch coroutine contexts during a single request, but in some cases the point where I write the log and where I send the response are exactly next to each other, so I'm not sure if kotlin has anything to do with it.
My logging.properties:
# To use this configuration, add to system properties : -Djava.util.logging.config.file="/path/to/file"
#
.level = INFO
# it is recommended that io.grpc and sun.net logging level is kept at INFO level,
# as both these packages are used by Stackdriver internals and can result in verbose / initialization problems.
io.grpc.netty.level=INFO
sun.net.level=INFO
handlers=com.google.cloud.logging.LoggingHandler
# default : java.log
com.google.cloud.logging.LoggingHandler.log=custom_log
# default : INFO
com.google.cloud.logging.LoggingHandler.level=INFO
# default : ERROR
com.google.cloud.logging.LoggingHandler.flushLevel=WARNING
# default : auto-detected, fallback "global"
#com.google.cloud.logging.LoggingHandler.resourceType=container
# custom formatter
com.google.cloud.logging.LoggingHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.SimpleFormatter.format=%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS %4$-6s %2$s %5$s%6$s%n
#optional enhancers (to add additional fields, labels)
#com.google.cloud.logging.LoggingHandler.enhancers=com.example.logging.jul.enhancers.ExampleEnhancer
My logging relevant dependencies:
implementation "org.slf4j:slf4j-jdk14:1.7.30"
implementation "com.google.cloud:google-cloud-logging:1.100.0"
An example logging call:
exception<Throwable> { e ->
logger().error("Error", e)
call.respondText(e.message ?: "", ContentType.Text.Plain, HttpStatusCode.InternalServerError)
}
with logger() being:
import org.slf4j.Logger
import org.slf4j.LoggerFactory
inline fun <reified T : Any> T.logger(): Logger = LoggerFactory.getLogger(T::class.java)
Edit:
An example of the log in Google cloud. The first request has the query parameter GAID=cdda802e-fb9c-47ad-0794d394c913, but as you can see the error log for that request is in the one below, marked in red.

Going Headless in Selenium with Chrome using Python Executing, But Throwing Errors

Based off the answer here I am trying to get Chrome to run headless in my script.
The snippet of code below is inside of a function called login() that logs into our ERP system:
if headless == True:
options = Options()
options.headless = True
#Load webdriver
driver = webdriver.Chrome(options=options, executable_path=r'C:/Users/d.kelly/Desktop/Python/chromedriver_win32/chromedriver.exe')
if headless == False:
driver = webdriver.Chrome('C:/Users/d.kelly/Desktop/Python/chromedriver_win32/chromedriver.exe')
window_before_login = driver.window_handles[0]
### Removed Code Block that fills out login form and clicks 'Login' button ###
# Switch to new window ERP (PLEX) launches and close original blank one no longer needed.
window_before_login = driver.window_handles[0]
window_title = driver.title
driver.switch_to.window(window_before_login)
driver.close()
driver.switch_to.window(driver.window_handles[0])
When I call my function like so:
login(headless=False)
It throws no errors and my entire script executes just fine.
When I call my function like this:
def login(headless=True)
I get the following errors:
DevTools listening on ws://127.0.0.1:57567/devtools/browser/69f9e357-dccf-4e38-8d6b-78030462379a
[0204/072436.206:INFO:CONSOLE(6)] "Error parsing a meta element's content: ';' is not a valid key-value pair separator. Please use ',' instead.", source: https://test.plexonline.com/modules/systemadministration/login/index.aspx? (6)
[0204/072437.699:INFO:CONSOLE(6)] "Error parsing a meta element's content: ';' is not a valid key-value pair separator. Please use ',' instead.", source: https://test.plexonline.com/Modules/SystemAdministration/Login/Index.aspx (6)
[0204/072437.722:INFO:CONSOLE(1)] "Scripts may close only the windows that were opened by it.", source: (1)
[0204/072441.162:INFO:CONSOLE(751)] "Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.", source: https://test.plexonline.com/Modules/scripts/ajax.js (751)
I am using Chrome Version 79.0.3945.130 (Official Build) (64-bit), Selenium 3.141.0, and Python 3.7.4.
Any ideas what I am doing wrong? Thank you!

Getting error asInvalidSession id while invoking upsert bulk operation in mule salesforce connector

Polling app daily once. We are loading data using upsert bulk operation in salesforce. First day it is working fine . Second day on wards we are getting the below error(InvalidSession id ). I found workaround as reconnection strategy and configured even after it is not working.
[2017-06-07 10:02:24.519] ERROR org.mule.retry.notifiers.ConnectNotifier [[test-salesforce-bulk].checkbulkupsert.stage1.05]: Failed to connect/reconnect: Work Descriptor. Root Exception was: InvalidSessionId : Invalid session id. Type: class com.sforce.async.AsyncApiException
[2017-06-07 10:02:24.547] ERROR org.mule.exception.CatchMessagingExceptionStrategy [[test-salesforce-bulk].checkbulkupsert.stage1.05]:
********************************************************************************
Message : Failed to invoke upsertBulk.
Element : /test-salesforce-bulk/processors/4/prepareAccountRequestSubFlow/subprocessors/1 # test-salesforce-bulk
--------------------------------------------------------------------------------
Exception stack is:
Failed to invoke upsertBulk. (org.mule.api.MessagingException)
com.sforce.async.BulkConnection.parseAndThrowException(BulkConnection.java:180)
com.sforce.async.BulkConnection.createOrUpdateJob(BulkConnection.java:164)
com.sforce.async.BulkConnection.createOrUpdateJob(BulkConnection.java:132)
com.sforce.async.BulkConnection.createJob(BulkConnection.java:122)
org.mule.modules.salesforce.SalesforceConnector.createJobInfo(SalesforceConnector.java:2570)
org.mule.modules.salesforce.SalesforceConnector.upsertBulk(SalesforceConnector.java:672)
org.mule.modules.salesforce.generated.processors.UpsertBulkMessageProcessor$1.process(UpsertBulkMessageProcessor.java:153)
(50 more...)
(set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
Can you please help on this.
There error says Invalid Session Id.
This means that the initial session has expired. Check your login history and see if there's even an attempt to login. If there's not, you need to update your username in Mulesoft. If there is and it was unsuccessful, you need to update your password/security token. If there is and it was successful, then it could be that your Session Settings are set to limit calls to the initial IP address or something like that.

How can I get AppEngine to log info level only for my app?

So I've tried configuring AppEngine logging according to this guide, ensuring I've configured the logging.properties file to be used in web.xml. I've configured logging.properties the following way:
.level = WARNING
nilsnett.chinese.backend.level = INFO
The package name of my logging wrapper is nilsnett.chinese.backend. The problem is that even with this configuration, info-level log output from my app is filtered. Evidence:
I've also tried the following config, which yielded the same result (including the logger class name at the end of the package name):
.level = WARNING
nilsnett.chinese.backend.JavaUtilLogger.level = INFO
To demonstrate that the logging.properties-file is actually read, and that I actually do write info-level logging data to app-engine in this service call, let me show you what happens when I set.level=INFO:
So my desired result is to have INFO and higher-level log outputs from my packages, while other packages, like org.datanucleus, only shows output if WARNING or more severe. In the example above, I want only the two lines marked with the purple star. Am I doing anything wrong?
change your config to:
.level = WARNING
# Set the default logging level for the datanucleus loggers
DataNucleus.JDO.level=WARNING
DataNucleus.Persistence.level=WARNING
DataNucleus.Cache.level=WARNING
DataNucleus.MetaData.level=WARNING
DataNucleus.General.level=WARNING
DataNucleus.Utility.level=WARNING
DataNucleus.Transaction.level=WARNING
DataNucleus.Datastore.level=WARNING
DataNucleus.ClassLoading.level=WARNING
DataNucleus.Plugin.level=WARNING
DataNucleus.ValueGeneration.level=WARNING
DataNucleus.Enhancer.level=WARNING
DataNucleus.SchemaTool.level=WARNING
# FinalizableReferenceQueue tries to spin up a thread and fails. This
# is inconsequential, so don't scare the user.
com.google.common.base.FinalizableReferenceQueue.level=WARNING
com.google.appengine.repackaged.com.google.common.base.FinalizableReferenceQueue.level=WARNING
this is are coming from logging config template, so to set datanucleus to warning you have todo like in this template.
https://developers.google.com/appengine/docs/java/#Logging
and then just add your own logging config:
nilsnett.chinese.backend.level = INFO
this should solve it

Authentication fails in databasedotcom

I am trying to adopt databasedotcom gem, but couldn't get beyond the authentication. Here is what I did (after installing databasedotcom gem):
rails c (or irb then require 'databasedotcom')
client=Databasedotcom::Client.new :client_id => 'foo', :client_secret=>'bar'
client.ca_file = '/Users/tjiang/missioncontrol/tmp/ca-bundle.crt'
client.verify_mode = OpenSSL::SSL::VERIFY_PEER
client.authenticate :username=>'myusername', :password=>'mypassword'
All credentials are copy-and-pasted in the process so no mistake there; the certificate was downloaded here: http://certifie.com/ca-bundle/ca-bundle.crt.txt
I tried Ruby 187 and 193 as well as inside and outside Rails, repeatedly, but always got this error message:
Databasedotcom::SalesForceError: authentication failure from /Library/Ruby/Gems/1.8/gems/databasedotcom-1.3.0/lib/databasedotcom/client.rb:112:in `authenticate'
I wonder what I have missed here? Particularly, I am concerned about the Callback URL I used when creating a Remote Access in Salesforce (I tried 'oob', 'http://localhost:3000', and 'https://www.salesforce.com', but none made any difference).
It turns out this is due to a bug in databasedotcom. When you use username and password to authenticate, it puts them into an url query string WITHOUT encoding and POST a request with that url. As a result, the plus sign in my username will be interpreted as a blank space.
Solution: CGI::escape() both your username and password.

Resources