Flink REST API POST error while trying to start a new job using the uploaded jar - apache-flink

I am trying to hit the /jars/:jarid/run endpoint to start a Flink job as follows after reading up this SO post -
curl -k -v -X POST -H "Content-Type: application/json" --data '
{
"programArgsList": [
"--runner",
"FlinkRunner",
"--inputTopicName",
"inputTopicNameValue",
"--Argument",
"Value",
"--streaming",
"true"]
}
' http://<JobManager-hostname>:<port>/jars/MyApplication.jar/run
I get the following error when I try the above command -
{"errors":["Internal server error.","<Exception on server side:\norg.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Argument 'FlinkRunner' does not begin with '--'\n\tat
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)\n\tat
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)\n\tat
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)\n\tat
org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:80)
Argument 'FlinkRunner' does not begin with '--' leads me to think that argument values are not being provided correctly in my example. I understand that the Flink documentation provides the JSON schema definition and not the sample request in the REST API docs. What is the correct way to provide argument values? My example is following what the accepted solution suggested in this post.

The following POST request worked for me so I am documenting it here -
curl -k -v -X POST -H "Content-Type: application/json" --data '
{
"programArgsList": [
"--runner=FlinkRunner",
"--inputTopicName=inputTopicNameValue",
"--Argument=Value",
"--streaming=true"]
}
' http://<JobManager-hostname>:<port>/jars/MyApplication.jar/run

Related

Http get method for Splunk saved search using access token

What would be the correct HTTP get request call syntax to fetch saved search data from Splunk if we're accessing it through an access token?
My curl command is working but http.get is not.
curl command:
#os.system('curl -H "Authorization: Bearer <token>"
<baseurl>:8089/services/search/jobs/export --data search="savedsearch abc_backup_status" -d output_mode=csv')
request call ::::
BASE_URL = '<baseurl>:8089/services/search/jobs/export'
data = {"search":"savedsearch abc_backup_status"}
headers = {'Authorization': "Bearer <token>"}
auth_response = requests.get(BASE_URL, headers=headers, data = data, verify=False)
this is giving 400 errors.
The curl options -d OR --data imply a POST method by default.
From: https://man7.org/linux/man-pages/man1/curl.1.html
-d, --data <data>
(HTTP MQTT) Sends the specified data in a POST request to
the HTTP server, in the same way that a browser does when
a user has filled in an HTML form and presses the submit
button. This will cause curl to pass the data to the
server using the content-type application/x-www-form-
urlencoded. Compare to -F, --form.
It is interesting that Splunk Docs claim that search/jobs/export takes a GET, but you're creating a job to immediately export, which feels like a POST type of operation.
Also I notice that your search starts with the savedsearch command, if that's a regularly scheduled savedsearch, you may want to GET saved/searches/{name}/history to get the last execution SID, followed either by the results or events endpoint of that already executed job, instead of a new search.... but that's a use case question

Solr 6.2.1 - Bad Request (400) - Invalid content type application/x-www-form-urlencoded; (ManagedSynonymsFilterFactory)

Using following query to add a new synonym mapping, you can PUT/POST a single mapping. (ManagedSynonymFilterFactory)
curl -X PUT -H 'Content-type:application/json' --data-binary
'{"mad":["angry","upset"], "happy": ["smiling", "jolly"]}'
"localhost:8983/solr/techproducts/schema/analysis/synonyms/english"
and getting the following response with error:
curl: (6) Could not resolve host: happy
curl: (3) [globbing] bad range in column 2
curl: (3) [globbing] unmatched close brace/bracket in column 6
{
"responseHeader":{
"status":500,
"QTime":111},
"error":{
"msg":"Bad Request",
"trace":"Bad Request (400) - Invalid content type application/x-www-form-
urlencoded; only application/json is supported.\r\n\tat only application/json is supported.\r\n\tat org.apache.solr.rest.RestManager$ManagedEndpoint.parseJsonFromRequestBody(RestManager.java:407)\r\n\tat
org.apache.solr.rest.RestManager$ManagedEndpoint.put(RestManager.java:340)\r\n\tat org.restlet.resource.ServerResource.doHandle(ServerResource.java:447)\r\n\tat
org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:359)\r\n\tat org.restlet.resource.ServerResource.handle(ServerResource.java:1044)\r\n\tat
org.restlet.resource.Finder.handle(Finder.java:236)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat org.restlet.routing.Router.doHandle(Router.java:422)\r\n\tat
org.restlet.routing.Router.handle(Router.java:639)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)\r\n\tat org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat
org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat
org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)\r\n\tat org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)\r\n\tat
org.restlet.Application.handle(Application.java:385)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat org.restlet.routing.Router.doHandle(Router.java:422)\r\n\tat org.restlet.routing.Router.handle(Router.java:639)\r\n\tat
org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat
org.restlet.routing.Router.doHandle(Router.java:422)\r\n\tat org.restlet.routing.Router.handle(Router.java:639)\r\n\tat org.restlet.routing.Filter.doHandle(Filter.java:150)\r\n\tat
org.restlet.routing.Filter.handle(Filter.java:197)\r\n\tat org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)\r\n\tat
org.restlet.Component.handle(Component.java:408)\r\n\tat org.restlet.Server.handle(Server.java:507)\r\n\tat
org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)\r\n\tat org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)\r\n\tat
org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1117)\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\r\n\tat
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)\r\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\r\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)\r\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\r\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\r\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\r\n\tat
org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:199)\r\n\tat org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:74)\r\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:266)\r\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\r\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\r\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\r\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\r\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\r\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\r\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\r\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\r\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\r\n\tat org.eclipse.jetty.server.Server.handle(Server.java:518)\r\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\r\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\r\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\r\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\r\n\tat
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\r\n\tat
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\r\n\tat
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)\r\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\r\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\r\n\tat java.lang.Thread.run(Unknown Source)\r\n",
"code":500}}
I was following an example in Solr document but still got this error. Why this error occured even when I'm following each step from the doc. I'm using Solr 6.2.1 on Windows
document for above example: https://lucene.apache.org/solr/guide/6_6/managed-resources.html
You have to use "" around parameters under cmd in Windows, not ''. To escape the " inside the JSON string, use "", or add the JSON content to a file and use -d "#request.json" instead, avoiding the need to escape anything explicitly.
I have seen such errors many times. I cannot pinpoint the right issue right now, but it has nothing to do with Solr, but with Curl.
As you are in windows, I guess you are using cmd, I strongly advise you set up WSL and do all curl-ing to solr in a ubuntu (or whatever linux you prefer) terminal, you will be able to use all snippets from the doc/examples without converting them to Windows world, and your life will be much easier.
Solve your problem with the below example
Error Request: curl -X PUT -H 'Content-type:application/json' --data-binary '["foo"]' "http://localhost:8983/solr/collection1/schema/analysis/stopwords/
english"
Correct Request: curl -X POST -H "Content-type:application/json" --data-binary "["foo"]" "http://localhost:8983/solr/techproducts/schema/analysis/stopwords/english"
Note: Simple replace ' with " in the request to make it simple.

org.apache.solr.common.SolrException Stream Body is disabled

I have setup apache solr 7.1 and using postman tool to query it. But when I am trying to delete indexed data using postman I get following error.
Request:
GET http://localhost:8983/solr/solr-sample3/update?stream.body={
"delete": {
"query": "*:*"
},
"commit": { }
}
Body:
{
"error": {
"metadata": [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"org.apache.solr.common.SolrException"
],
"msg": "Stream Body is disabled. See http://lucene.apache.org/solr/guide/requestdispatcher-in-solrconfig.html for help",
"code": 400
}
}
It was working in previous solr version solr 6.6. I went through the lucene documentation but I am not able to figure it out.
You don't need to enable the stream body. Just use a curl POST request specifying the data type as text/xml
curl http://localhost:8983/solr/solr-sample3/update?commit=true -H "Content-Type: text/xml" --data-binary '<delete><query>*:*</query></delete>'
Or if you're using the Post Tool included in solr:
bin/post -c core_name -type text/xml -out yes -d $'<delete><query>*:*</query></delete>'
I went though the documentation, it says i need to enable stream body as it has been disabled in solr 7.1 .
to enable use :
curl http://localhost:8983/solr/solr-sample3/config -H 'Content-type:application/json' -d'{
"set-property" : {"requestDispatcher.requestParsers.enableRemoteStreaming":true},
"set-property" : {"requestDispatcher.requestParsers.enableStreamBody":true}
}'
Here is what worked for me, using cURL and avoiding to enable stream body:
curl http://localhost:8983/solr/solr-sample3/update?commit=true -X POST -H "Content-Type: text/xml" --data-binary "<delete><query>*:*</query></delete>"

Error when adding user for Solr Basic Authentication

When I try to add the user for the Solr Basic Authentication using the following method in curl
curl --user user:password http://localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d '{
"set-user": {"tom" : "TomIsCool" ,
"harry":"HarrysSecret"}}'
I get the following error:
{
"responseHeader":{
"status":400,
"QTime":0},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"No contentStream",
"code":400}}
curl: (3) [globbing] unmatched brace in column 1
枩]?V7`-{炘9叡 t肤 ,? E'qyT咐黣]儎;衷 鈛^W褹?curl: (3) [globbing] unmatched cl
ose brace/bracket in column 13
What does this error means and how should we resolve it?
I'm using SolrCloud on Solr 6.4.2.
Regards,
Edwin
If you're using curl under Windows, this is a known issue with cmd.exe's escaping of single quotes. Use double quotes around your JSON string (or use cygwin, powershell, etc.)
curl --user user:password http://localhost:8983/solr/admin/authentication -H
"Content-type:application/json" -d "{
\"set-user\": {\"tom\" : \"TomIsCool\" ,
\"harry\":\"HarrysSecret\"}}"
The "globbing" message from curl is the hint that curl is doing something else than what you intended, and that the actual body of the request isn't getting to Solr (which is complaining about no message body being present).
You could also get around this by using stream.body in the URL and making the request from your browser.

Error while adding new field in Solr with default value

I am trying to add new field in Solr with default value but getting below error. I am sure that I am using wrong key for default but not able to get correct key in Google.
Command
curl -H 'Content-type: application/json' -d '{"name": "ALL_MODE_INDC","stored": true,"indexed": true,"field_type": "string" ,"default":"N"}' http://<solrServer>:<solrIP>/api/collections/ALGORITHM_AU_TEST/fields
Error message
{"errors":[{"message":"Unknown or dissallowed key found:default","code":"error.forbidden.key","key":"default"}],"http_status_name":"Unprocessable Entity","http_status_code":422}
Just got the solution (change key name to "default_value") -
curl -H 'Content-type: application/json' -d '{"name": "ALL_MODE_INDC","stored": true,"indexed": true,"field_type": "string" ,"default_value":"N"}' http://<solrServer>:<solrIP>/api/collections/ALGORITHM_AU_TEST/fields

Resources