.I am trying to create a data fusion pipeline which fetches data from pub sub and the used projection to convert message into string (as told by you) and thens ave it to GCS.
howeer,its giving me error: Parameter subscription must conform to pattern ^projects/[^/]+subscriptions/[^/]+$
I am using correct topic and subsctiption name projects/test/subscriptions/dataFusionSubscribe projects/test/topics/DataFusionTopic
You don't need to include the full subscription and topic name in the config, you only need to provide project id as test, subscription id as dataFusionSubscribe, and topic as DataFusionTopic
Attached is a screenshot showing the config for the Pubsub
Related
How can I query Solr, using the HTTP API, for information about a collection? I'm not talking about the collection's indexes, which I could query using the COLSTATUS command. I'm just talking about the basic details of a collection, which you can see when you click on a collection in the Solr web admin page, such as config name.
When wondering where information provided in the web interface comes from, the easiest way is to bring up your browser's development tools and go to the Network section. Since the interface is a small Javascript application, it uses the available REST API in the background - the same that you'd query yourself.
Extensive collection information can be retrieved by querying:
/solr/admin/collections?action=CLUSTERSTATUS&wt=json
(Any _ parameter is just present for cache busting).
This will return a list of all the collections present and their metadata, such as which config set they use and what shards the collection consists of. This is the same API endpoint that the web interface uses.
collections":{
"aaaaa":{
"pullReplicas":"0",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{"core_node2":{
"core":"aaaaa_shard1_replica_n1",
"base_url":"http://...:8983/solr",
"node_name":"...:8983_solr",
"state":"down",
"type":"NRT",
"force_set_state":"false",
"leader":"true"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"0",
"znodeVersion":7,
"configName":"_default"},
...
}
Please try the below code.
getConfigName(String collectionName){
//provide the list of zookeeper instances
List<String> zkHosts = ""
// get the solr cloud client
CloudSolrClient cloudSolrClient = new CloudSolrClient.Builder (zkHosts, Optional.empty
()).build ();
// get the config for the collection
String configName = solrConnectionProvider.getCloudSolrClient().getZkStateReader().readConfigName(collectionName);
return configName;
}
Please handle the exception(s) from your end.
I want to implement URL for search specific data in my base using filterByFormula
below are my link and I got an error and how to resolve that
my url:
api.airtable.com/v0/APPID/Stories?filterByFormula=(FIND(“Car (in robot form) will offer a hug when I am stressed out”,{User want}) &api_key=MYKEY
Error :
{
"error": {
"type": "INVALID_FILTER_BY_FORMULA",
"message": "The formula for filtering records is invalid: Invalid formula. Please check your formula text."
}
}
I tried using postman, please help me.
While querying filtered data from Airtable via API, you don't have to use FIND keyword. You can filter data with simple Airtable formula like structure. For example,
{Email} = 'johnwick#neverdie.com'
Above filter retrieve all the records from the table whose Email is simply johnwick#neverdie.com
To limit number of records retrieve by API maxRecord parameter is available for that. For more info about various parameters please refer my answer here AirTable API Find record by email or official Airtable API Documentation
In your case
API url would be structured like,
api.airtable.com/v0/APPID/Stories?filterByFormula=Car+(in+robot+form)+will+offer+a+hug+when+I+am+stressed+out%3D%22User+want%22&api_key=MYKEY
For more info about Airtable API encoding, check this https://codepen.io/airtable/full/rLKkYB?baseId=app1C5TVoophmmr8M&tableId=tblvILr4aSAYI98oa
Hope this helps!
I'd like to start syncing a users mailbox going forward so I need the most recent historyId of the users mailbox. There doesn't seem to be a way to get this with one API call.
The gmail.users.history.list endpoint contains a historyId which seems to be what I need, from the docs:
historyId unsigned long The ID of the mailbox's current history record.
However to get a valid response from this endpoint you must provide a startHistoryId as a parameter.
The only alternative I see is to make a request to list the users messages, get the most recent history id from that, then make a request to gmail.users.history.list providing that historyid to get the most recent one.
Other ideas?
Did you check out https://developers.google.com/gmail/api/guides/sync ?
Depending on what your use-case is, to avoid races between your current state and when you start to forward sync, you'll need to provide an appropriate historyId. If there were a "get current history ID" then anything between your previous state and when you got those results would be lost. If you don't have any particular existing state (e.g. only want to get updates and don't care about anything before that) then you can use any historyId returned (e.g. on a message or thread) as you mention.
Small example for C# users (mentioned in comments by #EricDeFriez).
Nuget package Google.Apis.Gmail.v1 must be installed. See also quickstart for .NET developers.
var service = new GmailService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = ApplicationName,
});
var req = service.Users.GetProfile("me");
req.Fields = "historyId";
var res = req.Execute();
Console.WriteLine("HistoryId: " + res.HistoryId);
This answer is related to the Java Gmail API Client Library using a service account.
I found that the gmail.users.getprofile() will not work as the object that it returns is of type Class Gmail.Users.GetProfile which does not have an interface to getting a historyId.
com.google.api.services.gmail.model.Profile actually has a getHistoryId() function, but calling service.users().getProfile() will return a Class Gmail.Users.GetProfileobject instead.
To get around this, I use the history.list() function which will always return the latest historyId as part of its response.
Gmail service = createGmailService(userId); //Authenticate
BigInteger startHistoryId = BigInteger.valueOf(historyId);
ListHistoryResponse response = service.users().history().list("me")
.setStartHistoryId(startHistoryId).setMaxResults(Long.valueOf(1)).execute();
I set the max number of results to be 1 to limit the unnecessary data that I get returned back and I will receive a payload that looks like:
{"history":[{"id":"XXX","messages":[{"id":"XXX","threadId":"XXX"}]}],"historyId":"123456","nextPageToken":"XXX"}
The historyId (123456) will be the current historyId of the user. You can grab that historyId using response.getHistoryId()
You can also see that the latest historyId is given in the response if you use the API tester for Users.history: list
https://developers.google.com/gmail/api/v1/reference/users/history/list
I have an object with a java.util.Date attribute I am serializing with the AE Cloud Endpoints service. From the client, when I just send a date, everything works fine (ex: '2013-05-27'). When I try adding a time, it fails:
{
"error" : {
"message" : "com.google.appengine.repackaged.org.codehaus.jackson.map.JsonMappingException: Invalid date/time format: 2013-05-27T12:00 (through reference chain: com.foo.MyObject[\"date\"])"
}
}
The best resource for jackson's default date format I have been able to find is here: http://wiki.fasterxml.com/JacksonFAQDateHandling. I tried the full ISO-8601 "1970-01-01T00:00:00.000+0000" which failed as well. I also tried an UNIX timestamp which didn't fail on parsing, but set a date in the 1372's.
So two part question. One, what's the correct default format to use? And two, can we modify (do we have access to) the jackson config so we can set our own serialization format?
Looks like it's close to the RFC 3339 standard, the fractional second appears to require 3 digits of precision, e.g.:
1985-04-12T23:20:50.520Z
This matches what's returned by the APIs Explorer, if you try it with your own API that includes a Date field.
With regards to handling the serialization yourself, you do not have access to the Jackson config, but you do have access to custom Endpoints serialization options (see #ApiSerializationProperty for instance).
I need to extract attatchments out of salesforce? I need to transfer some notes and attachments into another environmant. I am able to extract the notes but not sure how to go about extracting the attatchments
Thanks
Prady
This mostly depends on what tools/utilities you use to extract. The SOQL for Attachment sObject will always return one row at a time if Body field is included in the query. This is enforced to conserver resources and prevent overbearing SOQL scripts.
Approach #1, if queryMore is not available: Issue a SOQL without Body field to enumerate all attachments, then issue one SOQL per attachment ID to retrieve Body
Approach #2: Issue a SOQL to retrieve all needed attachments then loop using queryMore to get them one at a time.
Approach #3: If you can "freeze" the SF environment and just want to take snapshot of the system to pre-load a different one to be used going forward you can use "data exports". In setup menu, in data management there is an export data command, make sure you click "Include in export" to include all binary data. After due process it will give you a complete data backup you can crunch offline.
Btw, body is base64 encoded, you'll need to decode it to get the actual binary
Here is the solution I've used to get the attachment binary content from SalesForce. The example in their documentation points the following:
curl
https://na1.salesforce.com/services/data/v20.0/sobjects/Document/015D0000000NdJOIA0/body
-H "Authorization: Bearer token"
So there are a couple different elements here. The host (https://na1.salesforce.com) you should be able to get after the login process, this host is session based so it can always change. Second element is the rest of the URL, that you will get from the "body" field of the Attachment object. Third and last element is the Authorization header, which is composed by the string "Bearer ", plus the token that is given to you after you authenticate with the SF backend.
The response is the binary file, it is NOT in base64, just save it to a file and you are good to go.
Here is an example of how I did it in Objective C:
// How to get the correct host, since that is configured after the login.
NSURL * host = [[[[SFRestAPI sharedInstance] coordinator] credentials] instanceUrl];
// The field Body contains the partial URL to get the file content
NSURL *url = [NSURL URLWithString:[NSString stringWithFormat:#"%#%#", [host absoluteString], {AttachmentObject}.body]];
// Creating the Authorization header. Important to add the "Bearer " before the token
NSString *authHeader = [NSString stringWithFormat:#"Bearer %#",[[[[SFRestAPI sharedInstance] coordinator] credentials] accessToken]];
NSMutableURLRequest * urlRequest = [NSMutableURLRequest requestWithURL:url cachePolicy:NSURLRequestReloadIgnoringCacheData timeoutInterval:60.0];
[urlRequest addValue:authHeader forHTTPHeaderField:#"Authorization"];
[urlRequest setHTTPMethod:#"GET"];
urlConnection = [NSURLConnection connectionWithRequest:urlRequest delegate:self];
Hope it helps.
You can use SOQl Query options, or if you are looking for some automation tool that will help you with quick export, then you can try AppExchange App Satrang Mass File Download - https://appexchange.salesforce.com/listingDetail?listingId=a0N3A00000EcsAOUAZ&tab=e
Disclaimer: I work at Satrang Technologies, the publisher of this Mass File Download AppExchange App.
In SalesForce attachment will be against an Object for e.g. Account object.
Steps to retrieve attachment (in Java)
Get the ID of the Object to which a file is attached. e.q. Account Object
String pid = Account__r().getId();
Execute a Query on Salesforce Object "Attachment" for the ID in Step 1
*String q = "Select Name, Body, ContentType from Attachment
where ParentId = '" + pid + "'";
QueryResult qr = connection.query(q);
SObject[] sarr = qr.getRecords();*
SObject so = sarr[0];
Typecast Salesforce Generic Object (SObject) to "Attachment" object
*Attachment att = (Attachment)so;*
Retrieve the Byte Array Stream from Body of Attachment, and do the operation needed on byte array.
*byte[] bName = att.getBody();
// Do your operation in byte array stream*