I am trying to send post request to mapreduce in GAE.
Google provided
private String generateHtml(String configXml) {
return "<html>"
+ "<body>"
+ "<form action=\"/mapreduce/start\" method=\"POST\">"
+ "<textarea name=\"configuration\" rows=20 cols=80>"
+ configXml
+ "</textarea>"
+ "<input type=\"submit\" value=\"Start\">"
+ "</form>";
}
How I configure:
String configString = ConfigurationXmlUtil.convertConfigurationToXml(config);
String body = "configuration=" + configString;
Doing so I get null pointer exception when server tries to get parameter "configuration";
How should I build request so it would be identical to the html scenario with textarea?
Here's how I did it once using TaskQueue:
import static com.google.appengine.api.datastore.FetchOptions.Builder.withLimit;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;
import com.google.appengine.api.taskqueue.TaskOptions;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.Mapper;
//... other imports
// mapreduce config
Configuration conf = new Configuration(false);
conf.setClass("mapreduce.map.class", YourMapperClass.class, Mapper.class);
conf.setClass("mapreduce.inputformat.class", DatastoreInputFormat.class, InputFormat.class);
// setup callbacks if you need to know when the job is done.
//conf.set(AppEngineJobContext.DONE_CALLBACK_URL_KEY, "/job-done-callback");
// you can also specify a queue. Defaults to "default" :)
//conf.set(AppEngineJobContext.CONTROLLER_QUEUE_KEY, "mrqueue");
//conf.set(AppEngineJobContext.DONE_CALLBACK_QUEUE_KEY, "mrqueue");
//conf.set(AppEngineJobContext.WORKER_QUEUE_KEY, "mrqueue");
conf.set(DatastoreInputFormat.ENTITY_KIND_KEY, 'YourEntityKind');
String xml = ConfigurationXmlUtil.convertConfigurationToXml(conf);
TaskOptions opts = withUrl("/mapreduce/start")
.param("configuration", xml)
.method(TaskOptions.Method.POST)
.header("X-Requested-With", "XMLHttpRequest");
// use our dedicated dbm4g queue
// this needs to be configured by a user
QueueFactory.getQueue("mrqueue").add(opts);
I used a non default queue to make sure my MapReduce jobs won't hit limits too much. But you can safely use the default queue too.
Related
The code:
package simulations
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class StarWarsBasicExample extends Simulation
{
// 1 Http Conf
val httpConf = http.baseUrl("https://swapi.dev/api/films/")
// 2 Scenario Definition
val scn = scenario("Star Wars API")
.exec(http("Get Number")
.get("4")
.check(jsonPath("$.episode_id")
.saveAs("episodeId"))
)
.exec(session => {
val movie = session("episodeId").as[String]
session.set("episode",movie)
}).pause(4)
.exec(http("$episode")
.get("$episode"))
// 3 Load Scenario
setUp(
scn.inject(atOnceUsers(1)))
.protocols(httpConf)
}
Trying to grab a variable from the first Get request, and inject that variable into a second request, but unable to do so despite using the documentation. There might be something I'm not understanding.
When I use breakpoints, and navigate through the process, it appears the session execution happens AFTER both of the other requests have been completed (by which time is too late). Can't seem to make that session execution happen between the two requests.
Already answered on Gatling's community mailing list.
"$episode" is not correct Gatling Expression Language syntax. "${episode}" is correct.
How can I post message as a bot(async) in new hangouts chat without using the Google App Engine. I have gone through the examples, but all of them use App Engine for authentication, but i need to authenticate it without using the same.
Here is a code sample that connects to a chat using an http request and a webhook from Google Hangout Chat with a Python script. Webhooks are the only alternative to using a service account. More info here: https://developers.google.com/hangouts/chat/how-tos/webhooks
`from httplib2 import Http
from json import dumps
#
# Hangouts Chat incoming webhook quickstart
#
def main():
url = '<webhook url here>'
bot_message = {
'text' : 'text go here'}
message_headers = { 'Content-Type': 'application/json; charset=UTF-8'}
http_obj = Http()
response = http_obj.request(
uri=url,
method='POST',
headers=message_headers,
body=dumps(bot_message),
)
print(response)
if __name__ == '__main__':
main()
`
If your bot implementation is with google app script try to do it with google service account and as indicated here an example of async message
// Example bot for Hangouts Chat that demonstrates bot-initiated messages
// by spamming the user every minute.
//
// This bot makes use of the Apps Script OAuth2 library at:
// https://github.com/googlesamples/apps-script-oauth2
//
// Follow the instructions there to add the library to your script.
// When added to a space, we store the space's ID in ScriptProperties.
function onAddToSpace(e) {
PropertiesService.getScriptProperties()
.setProperty(e.space.name, '');
return {
'text': 'Hi! I\'ll post a message here every minute. ' +
'Please remove me after testing or I\'ll keep spamming you!'
};
}
// When removed from a space, we remove the space's ID from ScriptProperties.
function onRemoveFromSpace(e) {
PropertiesService.getScriptProperties()
.deleteProperty(e.space.name);
}
// Add a trigger that invokes this function every minute via the
// "Edit > Current Project's Triggers" menu. When it runs, it will
// post in each space the bot was added to.
function onTrigger() {
var spaceIds = PropertiesService.getScriptProperties()
.getKeys();
var message = { 'text': 'Hi! It\'s now ' + (new Date()) };
for (var i = 0; i < spaceIds.length; ++i) {
postMessage(spaceIds[i], message);
}
}
var SCOPE = 'https://www.googleapis.com/auth/chat.bot';
// The values below are copied from the JSON file downloaded upon
// service account creation.
var SERVICE_ACCOUNT_PRIVATE_KEY = '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n';
var SERVICE_ACCOUNT_EMAIL = 'service-account#project-id.iam.gserviceaccount.com';
// Posts a message into the given space ID via the API, using
// service account authentication.
function postMessage(spaceId, message) {
var service = OAuth2.createService('chat')
.setTokenUrl('https://accounts.google.com/o/oauth2/token')
.setPrivateKey(SERVICE_ACCOUNT_PRIVATE_KEY)
.setClientId(SERVICE_ACCOUNT_EMAIL)
.setPropertyStore(PropertiesService.getUserProperties())
.setScope(SCOPE);
if (!service.hasAccess()) {
Logger.log('Authentication error: %s', service.getLastError());
return;
}
var url = 'https://chat.googleapis.com/v1/' + spaceId + '/messages';
UrlFetchApp.fetch(url, {
method: 'post',
headers: { 'Authorization': 'Bearer ' + service.getAccessToken() },
contentType: 'application/json',
payload: JSON.stringify(message),
});
}
You need to perform some below steps.
Create a service-account in console.developers.google.com and download the private key in JSON format
Use below modules if you code in python.
from oauth2client.service_account import ServiceAccountCredentials
from googleapiclient.discovery import build, build_from_document
from httplib2 import Http
Below snippet will post the message to user via chat.google.
scopes = ['https://www.googleapis.com/auth/chat.bot']
credentials = ServiceAccountCredentials.from_json_keyfile_name('/path/to/json',
scopes)
http = Http()
credentials.authorize(http)
chat = build('chat', 'v1', http=http)
resp = chat.spaces().messages().create(
parent=space,
body={'text': 'HELLO WORLD'}).execute()
You would require a space name where you can post the code. You will get the same from hangout chat response.
It’s possible to do so using JavaScript, python, (possibly more). You can check out examples here: https://github.com/gsuitedevs/hangouts-chat-samples/tree/master/node/basic-cloud-functions-bot
If you’re using cards and JavaScript I would encourage you to checkout my library here: https://github.com/BaReinhard/hangouts-card-helper
I’m also in the process of creating another example for JavaScript that is more async focused that should provide and example that’s a bit easier to reason about the code. Will link when the PR is pushed.
Edit:
I realize that you mentioned REST api. The above answer is more useful for a specific bot that can be accessed #mentions. However, if you can provide us with a bit more information I can better fix my answer to answer your question.
I'm currently using Solr 4.3.1. i have configured dih for my solr. i would like to do a full import through command prompt. I know the url will be something like this http://localhost:8983/solr/corename/dataimport?command=full-import&clean=true&commit=true is there any method i can do this without using curl ?
Thanks
Edit
string Text = "http://localhost:8983/solr/Latest_TextBox/dataimport?command=full-import&clean=true&commit=true";
var wc = new WebClient();
var Import = wc.DownloadString(Text);
Currently using the above code
Call it like a normal REST url that's it !! I am using it in my application for importing and indexing data from my Local drive and it just works fine ! :) . Use HttpURLConnection to make a request and capture response to see whether it was successful or not . You don't need any specific API to do that . This is a sample code to make a GET request correctly in C# .Try data import handler url with this, it may work !
Console.WriteLine("Making API Call...");
using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
{
client.BaseAddress = new Uri("https://api.stackexchange.com/2.2/");
HttpResponseMessage response = client.GetAsync("answers?order=desc&sort=activity&site=stackoverflow").Result;
response.EnsureSuccessStatusCode();
string result = response.Content.ReadAsStringAsync().Result;
Console.WriteLine("Result: " + result);
}
Console.ReadLine();
}
}
}
You'll have to call the URL in some way - Solr only operates through a REST API. There is no command line API (the command line tools available just talk to the API). So use your preferred way to talk to a HTTP endpoint, that being curl, wget, GET or what's available for your programming language of choice.
The bundled solrCli application does not have any existing command for triggering a full-import as far as I were able to see (which would just talk to the REST API by calling the URL you've already referenced).
I try to call AppIdentityService#getDefaultGcsBucketName but it throws an exception: Response contained no data. According to the docs, the default bucket should be ready to use. Code:
//Field
private AppIdentityService service;
//in ctor
this.service = AppIdentityServiceFactory.getAppIdentityService();
//call inside a method
final String baseUrl = GCS_URL + "/" + service.getDefaultGcsBucketName() + "/" + fileName;
If you created your app prior to the 1.9.0 SDK release, you have to manually setup your default GCS bucket. Read this article for more information (I know its for PHP, but its applicable across all languages).
I'm having trouble setting a parameter when kicking off a mapreduce via start_map so I can access it in done_callback. Numerous things I've read imply that it's possible, but somehow I've not got the earth-moon-stars properly aligned. Ultimately, what I'm trying to accomplish is to delete the temporary blob I created for the mapreduce job.
Here's how I kick it off:
mrID = control.start_map(
"Find friends",
"findfriendshandler.findFriendHandler",
"mapreduce.input_readers.BlobstoreLineInputReader",
{"blob_keys": blobKey},
shard_count=7,
mapreduce_parameters={'done_callback': '/fnfrdone','blobKey': blobKey})
In done_callback, the context object isn't available:
class FindFriendsDoneHandler(webapp.RequestHandler):
def post(self):
ctx = context.get()
if ctx is not None:
params = ctx.mapreduce_spec.mapper.params
try:
blobKey = params['blobKey']
logging.info(['BLOBKEY ' + blobKey])
except KeyError:
logging.info('blobKey key not found in params')
else:
logging.info('context.get did not work') #THIS IS WHAT GETS OUTPUT
Thanks!
EDIT: It seems like there may be more than one MR library, so I wanted to include my various imports:
from mapreduce import control
from mapreduce import operation as op
from mapreduce import context
from mapreduce import model
Below is the code I used in my done_callback handler to retrieve my blobKey user parameter:
class FindFriendsDoneHandler(webapp.RequestHandler):
mrID = self.request.headers['Mapreduce-Id']
try:
mapreduceState = MapreduceState.get_by_key_name(mrID)
mrSpec = mapreduceState.mapreduce_spec
jsonSpec = mrSpec.to_json()
jsonParams = jsonSpec['params']
blobKey = jsonParams['blobKey']
blobInfo = BlobInfo.get(blobKey)
blobInfo.delete()
logging.info('Temp blob deleted successfully for mapreduce:' + mrID)
except:
logging.warning('Unable to delete temp blob for mapreduce:' + mrID)
This uses the mapreduce ID passed into done callback via the header to retrieve the mapreduce state model object from the mapreduce state table. The model stores any user params sent via start_map in a mapreduce_spec property which is in json format.
Note that MR, itself, actually stores the blob_key elsewhere in mapreduce_spec.
Thanks again to #Nick for pointing me to the model.py source file.
I'd love to hear if there's a simpler way to get at MR user params...
Context is only available to mappers/reducers - it's largely concerned with things that don't make sense outside the context of one. As you can see from the source, however, the "Mapreduce-Id" header is set, from which you can get the ID of the mapreduce job.
You shouldn't have to do your own cleanup, though - mapreduce has a handler that does it for you.