Github Link
Missing username and password properties from the solr connector configuration to authenticate with the solr.
Is that possible to authenticate with solr via connector config?
How to pass username n password within the connector config?
Looks like solr.username and solr.password aren't actually used in the connector. The SolrClient is created with a default HttpSolrClient without any credentials in this source file in Github. But MatsLindh's point about embedding username and password in the URL is a good one and I'd have expected it to work.
There's a relevant thread here:
Solr6.3.0 SolrJ API for Basic Authentication
I haven't vetted the last answer in the thread, which doesn't really address the original topic in that post, but it is a concise example of creating a SolrClient with authentication. The SolrClient needs to wrap an underlying HttpClient that provides the basic auth, and the Kafka Solr sink connector isn't doing that.
I wrote my first Gmail API program in C# starting with the help I found
at:
https://developers.google.com/gmail/api/quickstart/dotnet
The problem is the name of the app, according to Google, listed under "Third-party apps with account access", is "Quickstart", which is not a very descriptive name for my app. Am I stuck with that name?
I've tried to search the source files for "Quickstart" and can't find out where the name is coming from. I've made sure to specify my preferred name in the ApplicationName parameter of the service connect.
// Create Gmail API service.
var service = new GmailService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "EMailChecker",
});
But that makes no difference. I've tried removing access via my Google account and re-running the program to let it authenticate, but that does not affect. It still shows up as "Quickstart".
So am I stuck with that name? Other than that, the program is working well.
It is also stated in the documentation that if you are modifying these scopes, delete your previously saved credentials at ~/.credentials/gmail-dotnet-quickstart.json.
You may check this link for reference: Where is ~/.credentials/drive-python-quickstart.json saved for Google Drive API Python Implementation?
How do I get the AppEngine service name and version in the GAE flexible env from my Java code at runtime in the Java 8/Jetty runtime?
I need the service and version to populate the ServiceContext info in the stackdriver error reporting on GCP. https://cloud.google.com/error-reporting/docs/formatting-error-messages
I am the Stackdriver Error Reporting product manager.
To answer your question:
While I cannot find a clear documentation page for it. It seems that the environment variables GAE_MODULE_NAME and GAE_MODULE_VERSION contain the data you are looking for.
However, we recently changed how errors are processed on App Engine flexible environment: service name and version are now extracted automatically are not needed in the log entry payload. The serviceContext field is now optional on GAE Flex.
The formatting error messages page should be updated in the following days to reflect this change.
According to the current documentation the environment variables GAE_SERVICE and GAE_VERSION should be used.
It is also possible to get the instance ID with the GAE_INSTANCE environment variable.
https://cloud.google.com/appengine/docs/flexible/java/migrating#modules
In SalesForce you can schedule up to weekly "backups"/dumps of your data here: Setup > Administration Setup > Data Management > Data Export
If you have a large Salesforce database there can be a significant number of files to be downloading by hand.
Does anyone have a best practice, tool, batch file, or trick to automate this process or make it a little less manual?
Last time I checked, there was no way to access the backup file status (or actual files) over the API. I suspect they have made this process difficult to automate by design.
I use the Salesforce scheduler to prepare the files on a weekly basis, then I have a scheduled task that runs on a local server which downloads the files. Assuming you have the ability to automate/script some web requests, here are some steps you can use to download the files:
Get an active salesforce session ID/token
enterprise API - login() SOAP method
Get your organization ID ("org ID")
Setup > Company Profile > Company Information OR
use the enterprise API getUserInfo() SOAP call to retrieve your org ID
Send an HTTP GET request to https://{your sf.com instance}.salesforce.com/ui/setup/export/DataExportPage/d?setupid=DataManagementExport
Set the request cookie as follows:
oid={your org ID}; sid={your
session ID};
Parse the resulting HTML for instances of <a href="/servlet/servlet.OrgExport?fileName=
(The filename begins after fileName=)
Plug the file names into this URL to download (and save):
https://{your sf.com instance}.salesforce.com/servlet/servlet.OrgExport?fileName={filename}
Use the same cookie as in step 3 when downloading the files
This is by no means a best practice, but it gets the job done. It should go without saying that if they change the layout of the page in question, this probably won't work any more. Hope this helps.
A script to download the SalesForce backup files is available at https://github.com/carojkov/salesforce-export-downloader/
It's written in Ruby and can be run on any platform. Supplied configuration file provides fields for your username, password and download location.
With little configuration you can get your downloads going. The script sends email notifications on completion or failure.
It's simple enough to figure out the sequence of steps needed to write your own program if Ruby solution does not work for you.
I'm Naomi, CMO and co-founder of cloudHQ, so I feel like this is a question I should probably answer. :-)
cloudHQ is a SaaS service that syncs your cloud. In your case, you'd never need to upload your reports as a data export from Salesforce, but you'll just always have them backed up in a folder labeled "Salesforce Reports" in whichever service you synchronized Salesforce with like: Dropbox, Google Drive, Box, Egnyte, Sharepoint, etc.
The service is not free, but there's a free 15 day trial. To date, there's no other service that actually syncs your Salesforce reports with other cloud storage companies in real-time.
Here's where you can try it out: https://cloudhq.net/salesforce
I hope this helps you!
Cheers,
Naomi
Be careful that you know what you're getting in the back-up file. The backup is a zip of 65 different CSV files. It's raw data, outside of the Salesforce UI cannot be used very easily.
Our company makes the free DataExportConsole command line tool to fully automate the process. You do the following:
Automate the weekly Data Export with the Salesforce scheduler
Use the Windows Task Scheduler to run the FuseIT.SFDC.DataExportConsole.exe file with the right parameters.
I recently wrote a small PHP utility that uses the Bulk API to download a copy of sObjects you define via a json config file.
It's pretty basic but can easily be expanded to suit your needs.
Force.com Replicator on github.
Adding a Python3.6 solution. Should work (I haven't tested it though). Make sure the packages (requests, BeautifulSoup and simple_salesforce) are installed.
import os
import zipfile
import requests
import subprocess
from datetime import datetime
from bs4 import BeautifulSoup as BS
from simple_salesforce import Salesforce
def login_to_salesforce():
sf = Salesforce(
username=os.environ.get('SALESFORCE_USERNAME'),
password=os.environ.get('SALESFORCE_PASSWORD'),
security_token=os.environ.get('SALESFORCE_SECURITY_TOKEN')
)
return sf
org_id = "SALESFORCE_ORG_ID" # canbe found in salesforce-> company profile
export_page_url = "https://XXXX.my.salesforce.com/ui/setup/export/DataExportPage/d?setupid=DataManagementExport"
sf = login_to_salesforce()
cookie = {'oid': org_id, 'sid':sf.session_id}
export_page = requests.get(export_page_url, cookies=cookie)
export_page = export_page.content.decode()
links = []
parsed_page = BS(export_page)
_path_to_exports = "/servlet/servlet.OrgExport?fileName="
for link in parsed_page.findAll('a'):
href = link.get('href')
if href is not None:
if href.startswith(_path_to_exports):
links.append(href)
print(links)
if len(links) == 0:
print("No export files found")
exit(0)
today = datetime.today().strftime("%Y_%m_%d")
download_location = os.path.join(".", "tmp", today)
os.makedirs(download_location, exist_ok=True)
baseurl = "https://zageno.my.salesforce.com"
for link in links:
filename = baseurl + link
downloadfile = requests.get(filename, cookies=cookie, stream=True) # make stream=True if RAM consumption is high
with open(os.path.join(download_location, downloadfile.headers['Content-Disposition'].split("filename=")[1]), 'wb') as f:
for chunk in downloadfile.iter_content(chunk_size=100*1024*1024): # 50Mbs ??
if chunk:
f.write(chunk)
I have added a feature in my app to automatically backup the weekly/monthly csv files to S3 bucket, https://app.salesforce-compare.com/
Create a connection provider (currently only AWS S3 is supported) and link it to a SF connection (needs to be created as well).
On the main page you can monitor the progress of the scheduled job and access the files in the bucket
More info: https://salesforce-compare.com/release-notes/
I'm trying to download data from my Google App Engine app, using the official instructions
Remote API is set up & I call:
appcfg.py download_data --application=appname --url=http://app.address.com/_ah/remote_api --filename=alldata.csv
The connection is established, I'm being asked for my e-mail & password, and then a long trace appears ending up with this:
google.appengine.api.datastore_errors.BadRequestError: app s~appname cannot access app appname's data
Any ideas?
If using the high-replication datastore, change the value passed to --application from appname to s~appname.
Found the problem - as an URL I was giving my own app's domain name instead of the 1.appname.appspot.com.
I ran into this same issue, but when attempting to use the remote API from a custom script as opposed to appcfg.py's bulk downloader.
This issue may have been addressed in more recent GAE SDK releases, but for various reasons I'm still using the older 1.7 release. I found the answer in this bug report: https://code.google.com/p/googleappengine/issues/detail?id=4374
Basically, when calling ConfigureRemoteApi, don't specify the APP_ID argument (just pass None) and specify your app ID via the host argument, e.g. 'myapp-hrd.appspot.com'. ConfigureRemoteApi will figure out your app ID correctly and won't add the 's~' that causes this problem.