How to use RESTfulWebServiceClient for php server in CN1 - codenameone

I want to generate the code for the server in php and use it as the backend instead of webservices in java. I noticed that, webservices Wizard in CN1 generated details on the server side. In my own case I want to create the server in php and use RESTfulWebServiceClient to point to the url.
Pls how do I go about this?
Is there any webservice Wizard that can generate the php server like java in netbeans?
In the auto re-new subscription in in-app purchase can I interact with the mysql database without using RESTfulWebServiceClient?
Thanks for your response
I have created the webservices in php and used the url in codename one. I was able to use the url to submit receipt into the database. see the image below
I still have challenges, the error code below was generated
[EDT] 0:0:0,0 - Codename One revisions: 3dc2fe6c4df57ff264ea094d13e0275639b780862293
[EDT] 0:0:0,0 - Receipts were last refreshed at Fri Mar 17 07:30:21 WAT 2017 so we won't refetch.
WARNING: Apple will no longer accept http URL connections from applications you tried to connect to http://slimapp/api/customer/add to learn more check out https://www.codenameone.com/blog/ios-http-urls.html
[Task Thread] 0:0:0,417 - Expected null for key value while parsing JSON token at row: 1 column: 44 buffer:
[Task Thread] 0:0:0,417 - Expected true for key value while parsing JSON token at row: 1 column: 47 buffer:
[Task Thread] 0:0:0,417 - Expected null for key value while parsing JSON token at row: 1 column: 49 buffer:
[Task Thread] 0:0:0,417 - Expected null for key value while parsing JSON token at row: 1 column: 52 buffer:
[Task Thread] 0:0:0,417 - Expected true for key value while parsing JSON token at row: 1 column: 57 buffer: l
[Task Thread] 0:0:0,417 - Expected null for key value while parsing JSON token at row: 1 column: 70 buffer: 1048l
[Task Thread] 0:0:0,417 - Expected null for key value while parsing JSON token at row: 1 column: 88 buffer: 1048l
java.lang.NumberFormatException: For input string: "1048le"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at com.codename1.io.JSONParser.parse(JSONParser.java:286)
at com.codename1.io.JSONParser.parseJSON(JSONParser.java:427)
at com.codename1.ws.RESTfulWebServiceClient.lambda$null$6(RESTfulWebServiceClient.java:204)
at com.codename1.ui.Display$1.run(Display.java:806)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
WARNING: Apple will no longer accept http URL connections from applications you tried to connect to http://slimapp/api/customer/add to learn more check out https://www.codenameone.com/blog/ios-http-urls.html
[Task Thread] 0:0:8,888 - Expected null for key value while parsing JSON token at row: 1 column: 44 buffer:
[Task Thread] 0:0:8,888 - Expected true for key value while parsing JSON token at row: 1 column: 47 buffer:
[Task Thread] 0:0:8,888 - Expected null for key value while parsing JSON token at row: 1 column: 49 buffer:
[Task Thread] 0:0:8,888 - Expected null for key value while parsing JSON token at row: 1 column: 52 buffer:
[Task Thread] 0:0:8,888 - Expected true for key value while parsing JSON token at row: 1 column: 57 buffer: l
[Task Thread] 0:0:8,888 - Expected null for key value while parsing JSON token at row: 1 column: 70 buffer: 1048l
[Task Thread] 0:0:8,888 - Expected null for key value while parsing JSON token at row: 1 column: 88 buffer: 1048l
java.lang.NumberFormatException: For input string: "1048le"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at com.codename1.io.JSONParser.parse(JSONParser.java:286)
at com.codename1.io.JSONParser.parseJSON(JSONParser.java:427)
at com.codename1.ws.RESTfulWebServiceClient.lambda$null$6(RESTfulWebServiceClient.java:204)
at com.codename1.ui.Display$1.run(Display.java:806)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
After subscribing and I press Syns Receipts. Subscribe to remove ads is still there. Pls help Thanks
This is my php webserver code.
<?php
use \Psr\Http\Message\ServerRequestInterface as Request;
use \Psr\Http\Message\ResponseInterface as Response;
$app = new \Slim\App;
$app->get('/api/customers', function (Request $request, Response $response) {
echo "customers";
// Get All customers
$sql = "SELECT * FROM receipts";
try{
$db = new db();
$db = $db->connect();
$stmt = $db->query($sql);
$customers = $stmt->fetchAll(PDO::FETCH_OBJ);
$db = null;
echo json_encode($customers);
}catch(PDOException $e){
echo '{"error": {"text": '.$e->getMessage().'}';
}
});
$app->get('/api/customers/{transactionId}', function (Request $request, Response $response) {
echo "customers";
$id = $request ->getAttribute("transactionId");
// Get single customers
$sql = "SELECT * FROM receipts where transaction_id = $id";
try{
$db = new db();
$db = $db->connect();
$stmt = $db->query($sql);
$customers = $stmt->fetchAll(PDO::FETCH_OBJ);
$db = null;
echo json_encode($customers);
}catch(PDOException $e){
echo '{"error": {"text": '.$e->getMessage().'}';
}
});
//add a user
$app->post('/api/customer/add', function (Request $request, Response $response) {
//echo "customers";
$transactionId = $request->getParam('transactionId');
$sku = $request->getParam('sku');
$purchaseDate = $request->getParam('purchaseDate');
$orderData = $request->getParam('orderData');
$storeCode = $request->getParam('storeCode');
echo "emmy ";
// (:transactionId,:sku,:purchaseDate,:orderData,:storeCode)";
// 'transactionid'
$sql = "INSERT INTO receipts(transaction_id,sku,purchase_date,order_data,store_code)VALUES
(:transactionid,:sku,:purchaseDate,:orderData,:storeCode)";
try{
$db = new db();
$db = $db->connect();
$stmt = $db->prepare($sql);
$stmt->bindParam(':transactionid', $transactionId);
$stmt->bindParam(':sku', $sku);
$stmt->bindParam(':purchaseDate', $purchaseDate);
$stmt->bindParam(':orderData', $orderData);
$stmt->bindParam(':storeCode', $storeCode);
$stmt->execute();
echo '{"notice": {"text": "customar added"}';
}catch(PDOException $e){
echo '{"error": {"text": '.$e->getMessage().'}';
}
});
This is my url and part of my code in codename name:
private static final String receiptsEndpoint = "http://slimapp/api/customers";
private static final String receiptsEndpoint2 = "http://slimapp/api/customer/add";
private ReceiptStore createReceiptStore() {
return new ReceiptStore() {
RESTfulWebServiceClient client = createRESTClient(receiptsEndpoint);
RESTfulWebServiceClient client2 = createRESTClient(receiptsEndpoint2);
#Override
public void fetchReceipts(SuccessCallback<Receipt[]> callback) {
RESTfulWebServiceClient.Query query = new RESTfulWebServiceClient.Query() {
#Override
protected void setupConnectionRequest(RESTfulWebServiceClient client, ConnectionRequest req) {
super.setupConnectionRequest(client, req);
// req.setPost(true);
// req.setHttpMethod("post"); //Change to GET if necessary
req.setUrl(receiptsEndpoint);
}
protected void setupConnectionRequest2(RESTfulWebServiceClient client2, ConnectionRequest req2) {
super.setupConnectionRequest(client, req2);
req2.setPost(true);
req2.setHttpMethod("post"); //Change to GET if necessary
req2.setUrl(receiptsEndpoint2);
}
};
client.find(query, rowset->{
List<Receipt> out = new ArrayList<Receipt>();
for (Map m : rowset) {
Result res = Result.fromContent(m);
Receipt r = new Receipt();
r.setTransactionId(res.getAsString("transactionId"));
System.out.println(r.getTransactionId()+" ko");
r.setPurchaseDate(new Date(res.getAsLong("purchaseDate")));
r.setQuantity(1);
r.setSku(res.getAsString("sku"));
if (m.containsKey("cancellationDate") && m.get("cancellationDate") != null) {
r.setCancellationDate(new Date(res.getAsLong("cancellationDate")));
}
if (m.containsKey("expiryDate") && m.get("expiryDate") != null) {
r.setExpiryDate(new Date(res.getAsLong("expiryDate")));
}
out.add(r);
}
callback.onSucess(out.toArray(new Receipt[out.size()]));
});
}
#Override
public void submitReceipt(Receipt r, SuccessCallback<Boolean> callback) {
Map m = new HashMap();
m.put("transactionId", r.getTransactionId());
m.put("sku", r.getSku());
m.put("purchaseDate", r.getPurchaseDate().getTime());
m.put("orderData", r.getOrderData());
m.put("storeCode", r.getStoreCode());
System.out.println(r.getTransactionId());
client2.create(m, callback);
}
};
}
pls help to review my code in above what I am missing maybe in php webserver code or in codename one code. I was able to subscribe and receive the message that the subscription was succesfull and the records was inserted into the database, when I click on Syns Receipts Button, No receipt was updated and the record goes to the database. Pls help, I dont know what is going wrong

Our webservice wizard expects a Java server and doesn't generate PHP code.
You can use any restful service from Codename One by using ConnectionRequest and invoking a URL with GET, POST, PUT & DELETE.
See the developer guide section on networking. If you create a webservice in PHP and have the curl usage I can help with the Codename One equivalent.

Related

Flink Kinesis Sink Decoding Issue

I have Flink job running in AWS Kinesis Analytics that does the following.
1 - I have Table on a Kinesis Stream - Called MainEvents.
2 - I have a Sink Table that is pointing to Kinesis Stream - Called perMinute.
The perMinute is populated using the MainEvents table as input and generates a sliding window(hop) agg.
So far so good.
My final consumer is a Kineis Python Script that reads the input from perMinute Kinesis Stream.
This is my Consumer Script.
stream_name = 'perMinute'
ses = boto3.session.Session()
kinesis_client = ses.client('kinesis')
response = kinesis_client.describe_stream(StreamName=stream_name)
shard_id = response['StreamDescription']['Shards'][0]['ShardId']
response = kinesis_client.get_shard_iterator(
StreamName=stream_name,
ShardId=shard_id,
ShardIteratorType='LATEST'
)
shard_iterator = response['ShardIterator']
while shard_iterator is not None:
result = kinesis_client.get_records(ShardIterator=shard_iterator, Limit=1)
records = result["Records"]
shard_iterator = result["NextShardIterator"]
for record in records:
data = str(record["Data"])
print(data)
time.sleep(1)
The issue i have is that i get encoded data, that looks like.
b'{"window_start":"2022-09-28 04:01:46","window_end":"2022-09-28 04:02:46","counts":300}'
b'{"window_start":"2022-09-28 04:02:06","window_end":"2022-09-28 04:03:06","counts":478}'
b'\xf3\x89\x9a\xc2\n$4a540599-485d-47c5-9a7e-ca46173b30de\n$2349a5a3-7949-4bde-95a8-4019a077586b\x1aX\x08\x00\x1aT{"window_start":"2022-09-28 04:02:16","window_end":"2022-09-28 04:03:16","counts":504}\x1aX\x08\x01\x1aT{"window_start":"2022-09-28 04:02:18","window_end":"2022-09-28 04:03:18","counts":503}\xc3\xa1\xfe\xfa9j\xeb\x1aP\x917F\xf3\xd2\xb7\x02'
b'\xf3\x89\x9a\xc2\n$23a0d76c-6939-4eda-b5ee-8cd2b3dc1c1e\n$7ddf1c0c-16fe-47a0-bd99-ef9470cade28\x1aX\x08\x00\x1aT{"window_start":"2022-09-28 04:02:30","window_end":"2022-09-28 04:03:30","counts":531}\x1aX\x08\x01\x1aT{"window_start":"2022-09-28 04:02:36","window_end":"2022-09-28 04:03:36","counts":560}\x0c>.\xbd\x0b\xac.\x9a\xe8z\x04\x850\xd5\xa6\xb3'
b'\xf3\x89\x9a\xc2\n$2cacfdf8-a09b-4fa3-b032-6f1707c966c3\n$27458e17-8a3a-434e-9afd-4995c8e6a1a4\n$11774332-d906-4486-a959-28ceec0d134a\x1aY\x08\x00\x1aU{"window_start":"2022-09-28 04:02:42","window_end":"2022-09-28 04:03:42","counts":1625}\x1aY\x08\x01\x1aU{"window_start":"2022-09-28 04:02:50","window_end":"2022-09-28 04:03:50","counts":2713}\x1aY\x08\x02\x1aU{"window_start":"2022-09-28 04:03:00","window_end":"2022-09-28 04:04:00","counts":3009}\xe1G\x18\xe7_a\x07\xd3\x81O\x03\xf9Q\xaa\x0b_'
Some Records are valid, the first two and the other records seems to have multiple entries on the same row.
How can i get rid of the extra characters that are not part of the json payload and get one line per invocation ?
If i would use decode('utf-8'), i get few record out ok but when it reaches a point if fails with:
while shard_iterator is not None:
result = kinesis_client.get_records(ShardIterator=shard_iterator, Limit=1)
records = result["Records"]
shard_iterator = result["NextShardIterator"]
for record in records:
data = record["Data"].decode('utf-8')
# data = record["Data"].decode('latin-1')
print(data)
time.sleep(1)
{"window_start":"2022-09-28 03:59:24","window_end":"2022-09-28 04:00:24","counts":319}
{"window_start":"2022-09-28 03:59:28","window_end":"2022-09-28 04:00:28","counts":366}
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-108-0e632a57c871> in <module>
39 shard_iterator = result["NextShardIterator"]
40 for record in records:
---> 41 data = record["Data"].decode('utf-8')
43 print(data)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-2: invalid continuation byte
If i use decode('latin-1') it does not fail but i get alot of crazy text out
{"window_start":"2022-09-28 04:02:06","window_end":"2022-09-28 04:03:06","counts":478}
óÂ
$4a540599-485d-47c5-9a7e-ca46173b30de
$2349a5a3-7949-4bde-95a8-4019a077586bXT{"window_start":"2022-09-28 04:02:16","window_end":"2022-09-28 04:03:16","counts":504}XT{"window_start":"2022-09-28 04:02:18","window_end":"2022-09-28 04:03:18","counts":503}áþú9jëP7FóÒ·
óÂ
here is the stream producer flink code
-- create sink
CREATE TABLE perMinute (
window_start TIMESTAMP(3) NOT NULL,
window_end TIMESTAMP(3) NOT NULL,
counts BIGINT NOT NULL
)
WITH (
'connector' = 'kinesis',
'stream' = 'perMinute',
'aws.region' = 'ap-southeast-2',
'scan.stream.initpos' = 'LATEST',
'format' = 'json',
'json.timestamp-format.standard' = 'ISO-8601'
);
%flink.ssql(type=update)
insert into perMinute
SELECT window_start, window_end, COUNT(DISTINCT event) as counts
FROM TABLE(
HOP(TABLE MainEvents, DESCRIPTOR(eventtime), INTERVAL '5' SECOND, INTERVAL '60' SECOND))
GROUP BY window_start, window_end;
Thanks

Rest API throws JSON parsing error: Exception during JSON parsing at row: 1 column: 1 buffer

I am using Rest API and when I use Rest.put(), it throws an error. It does not even reach the server at all. I don't know where I am going wrong.
Here is my code:
Response<String> res = Rest.put(URLLinks.getMainBackend() + "items")
.body(reqJson.toString())
.jsonContent()
.bearer(initForm.data.getString("jwt"))
.header("token", initForm.data.getString("token"))
.pathParam("id", item.getId())
.onErrorCodeJSON((errorData) -> {
if (errorData.getResponseCode() == 404) {
Dialog dlg = rich.Dialog("Not found!");
Button yes = new Button("Close");
yes.addActionListener(ev -> {
dlg.dispose();
});
dlg.add(new SpanLabel("//////!"));
dlg.add(FlowLayout.encloseRight(yes));
dlg.showPacked(BorderLayout.CENTER, true);
}else if (errorData.getResponseCode() == 402){
}
})
.getAsString();
It throws an error:
java.io.IOException: Stream closed
[Network Thread] 0:0:12,855 - Codename One revisions: 4afb54f6a5cecd2b6fbee170262d5c3c8d9431f9
[Network Thread] 0:0:12,855 - Exception: java.io.IOException - Stream closed
[Network Thread] 0:0:12,858 - Exception during JSON parsing at row: 1 column: 1 buffer:
[EDT] 0:0:12,874 - Exception: java.lang.NullPointerException - null
at com.codename1.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:140)
at com.codename1.io.BufferedInputStream.read1(BufferedInputStream.java:338)
at com.codename1.io.BufferedInputStream.read(BufferedInputStream.java:445)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at com.codename1.io.JSONParser$ReaderClass.read(JSONParser.java:124)
at com.codename1.io.JSONParser.parse(JSONParser.java:188)
at com.codename1.io.JSONParser.parseJSON(JSONParser.java:475)
at com.codename1.io.rest.RequestBuilder$Connection.readUnzipedResponse(RequestBuilder.java:683)
at com.codename1.io.gzip.GZConnectionRequest.readResponse(GZConnectionRequest.java:67)
at com.codename1.io.ConnectionRequest.performOperation(ConnectionRequest.java:809)
at com.codename1.io.NetworkManager$NetworkThread.run(NetworkManager.java:282)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
java.lang.NullPointerException
at java.base/java.lang.String.<init>(String.java:537)
at com.codename1.io.rest.RequestBuilder.getAsString(RequestBuilder.java:361)
at com.falcontechnology.items.NewItem.backendItem(NewItem.java:425)
at com.falcontechnology.items.NewItem.lambda$itemFields$7(NewItem.java:296)
at com.codename1.ui.util.EventDispatcher.fireActionEvent(EventDispatcher.java:349)
at com.codename1.ui.Button.fireActionEvent(Button.java:570)
at com.codename1.ui.Button.released(Button.java:604)
at com.codename1.ui.Button.pointerReleased(Button.java:708)
at com.codename1.ui.Form.pointerReleased(Form.java:3356)
at com.codename1.ui.Component.pointerReleased(Component.java:4552)
at com.codename1.ui.Display.handleEvent(Display.java:2080)
at com.codename1.ui.Display.edtLoopImpl(Display.java:1052)
at com.codename1.ui.Display.mainEDTLoop(Display.java:970)
at com.codename1.ui.RunnableWrapper.run(RunnableWrapper.java:120)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
The null pointer on line NewItem.java:425 points at the line where there is .getAsString().
What could i be doing wrong?
EDIT
I have updated my project libraries which solved the null pointer exception on .getAsString() but it still leaves another error which is:
java.io.IOException: Stream closed
[Network Thread] 0:0:54,316 - Exception: java.io.IOException - Stream closed
[Network Thread] 0:0:54,344 - Exception during JSON parsing at row: 1 column: 1 buffer:
at com.codename1.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:140)
at com.codename1.io.BufferedInputStream.read1(BufferedInputStream.java:338)
at com.codename1.io.BufferedInputStream.read(BufferedInputStream.java:445)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at com.codename1.io.JSONParser$ReaderClass.read(JSONParser.java:191)
at com.codename1.io.JSONParser.parse(JSONParser.java:278)
at com.codename1.io.JSONParser.parseJSON(JSONParser.java:568)
at com.codename1.io.rest.RequestBuilder$Connection.readUnzipedResponse(RequestBuilder.java:785)
at com.codename1.io.gzip.GZConnectionRequest.readResponse(GZConnectionRequest.java:67)
at com.codename1.io.ConnectionRequest.performOperationComplete(ConnectionRequest.java:1002)
at com.codename1.io.NetworkManager$NetworkThread.run(NetworkManager.java:340)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
Apparently when you use .pathparam(), you also have to add the parameter in the url. Hence I adjusted my code to:
Response<String> res = Rest.put(URLLinks.getMainBackend() + "items"+item.getId())
.body(reqJson.toString())
.jsonContent()
.bearer(initForm.data.getString("jwt"))
.header("token", initForm.data.getString("token"))
.pathParam("id", item.getId())
.onErrorCodeJSON((errorData) -> {
if (errorData.getResponseCode() == 404) {
Dialog dlg = rich.Dialog("Not found!");
Button yes = new Button("Close");
yes.addActionListener(ev -> {
dlg.dispose();
});
dlg.add(new SpanLabel("//////!"));
dlg.add(FlowLayout.encloseRight(yes));
dlg.showPacked(BorderLayout.CENTER, true);
}else if (errorData.getResponseCode() == 402){
}
})
.getAsString();

cron job throwing DeadlineExceededError

I am currently working on a google cloud project in free trial mode. I have cron job to fetch the data from a data vendor and store it in the data store. I wrote the code to fetch the data couple of weeks ago and it was all working fine but all of sudden , i started receiving error "DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded" for last two days. I believe cron job is supposed to timeout only after 60 minutes any idea why i am getting the error?.
cron task
def run():
try:
config = cron.config
actual_data_source = config['xxx']['xxxx']
original_data_source = actual_data_source
company_list = cron.rest_client.load(config, "companies", '')
if not company_list:
logging.info("Company list is empty")
return "Ok"
for row in company_list:
company_repository.save(row,original_data_source, actual_data_source)
return "OK"
Repository code
def save( dto, org_ds , act_dp):
try:
key = 'FIN/%s' % (dto['ticker'])
company = CompanyInfo(id=key)
company.stock_code = key
company.ticker = dto['ticker']
company.name = dto['name']
company.original_data_source = org_ds
company.actual_data_provider = act_dp
company.put()
return company
except Exception:
logging.exception("company_repository: error occurred saving the company
record ")
raise
RestClient
def load(config, resource, filter):
try:
username = config['xxxx']['xxxx']
password = config['xxxx']['xxxx']
headers = {"Authorization": "Basic %s" % base64.b64encode(username + ":"
+ password)}
if filter:
from_date = filter['from']
to_date = filter['to']
ticker = filter['ticker']
start_date = datetime.strptime(from_date, '%Y%m%d').strftime("%Y-%m-%d")
end_date = datetime.strptime(to_date, '%Y%m%d').strftime("%Y-%m-%d")
current_page = 1
data = []
while True:
if (filter):
url = config['xxxx']["endpoints"][resource] % (ticker, current_page, start_date, end_date)
else:
url = config['xxxx']["endpoints"][resource] % (current_page)
response = urlfetch.fetch(
url=url,
deadline=60,
method=urlfetch.GET,
headers=headers,
follow_redirects=False,
)
if response.status_code != 200:
logging.error("xxxx GET received status code %d!" % (response.status_code))
logging.error("error happend for url: %s with headers %s", url, headers)
return 'Sorry, xxxx API request failed', 500
db = json.loads(response.content)
if not db['data']:
break
data.extend(db['data'])
if db['total_pages'] == current_page:
break
current_page += 1
return data
except Exception:
logging.exception("Error occured with xxxx API request")
raise
I'm guessing this is the same question as this, but now with more code:
DeadlineExceededError: The overall deadline for responding to the HTTP request was exceeded
I modified your code to write to the database after each urlfetch. If there are more pages, then it relaunches itself in a deferred task, which should be well before the 10 minute timeout.
Uncaught exceptions in a deferred task cause it to retry, so be mindful of that.
It was unclear to me how actual_data_source & original_data_source worked, but I think you should be able to modify that part.
crontask
def run(current_page=0):
try:
config = cron.config
actual_data_source = config['xxx']['xxxx']
original_data_source = actual_data_source
data, more = cron.rest_client.load(config, "companies", '', current_page)
for row in data:
company_repository.save(row, original_data_source, actual_data_source)
# fetch the rest
if more:
deferred.defer(run, current_page + 1)
except Exception as e:
logging.exception("run() experienced an error: %s" % e)
RestClient
def load(config, resource, filter, current_page):
try:
username = config['xxxx']['xxxx']
password = config['xxxx']['xxxx']
headers = {"Authorization": "Basic %s" % base64.b64encode(username + ":"
+ password)}
if filter:
from_date = filter['from']
to_date = filter['to']
ticker = filter['ticker']
start_date = datetime.strptime(from_date, '%Y%m%d').strftime("%Y-%m-%d")
end_date = datetime.strptime(to_date, '%Y%m%d').strftime("%Y-%m-%d")
url = config['xxxx']["endpoints"][resource] % (ticker, current_page, start_date, end_date)
else:
url = config['xxxx']["endpoints"][resource] % (current_page)
response = urlfetch.fetch(
url=url,
deadline=60,
method=urlfetch.GET,
headers=headers,
follow_redirects=False,
)
if response.status_code != 200:
logging.error("xxxx GET received status code %d!" % (response.status_code))
logging.error("error happend for url: %s with headers %s", url, headers)
return [], False
db = json.loads(response.content)
return db['data'], (db['total_pages'] != current_page)
except Exception as e:
logging.exception("Error occured with xxxx API request: %s" % e)
return [], False
I would prefer to write this as a comment, but I need more reputation to do that.
What happens when you run the actual data fetch directly instead of
through the cron job?
Have you tried measuring a time delta from the start to the end of
the job?
Has the number of companies being retrieved increased dramatically?
You appear to be doing some form of stock quote aggregation - is it
possible that the provider has started blocking you?

JIRA-REST-JAVA - Error MalformedInputException on creating issue

I'm using jira-rest-java-client-core and jira-rest-java-client-api (2.0.0-m31) wih maven.
GET an issue works fine but i've a problem when I try to create one.
JiraRestClientFactory factory = new AsynchronousJiraRestClientFactory();
JiraRestClient jiraRestClient = factory.createWithBasicHttpAuthentication(URI.create("https://jira_host"), username, password);
IssueRestClient client = jiraRestClient.getIssueClient();
IssueInputBuilder issueBuilder = new IssueInputBuilder("TEST", 1L, "Summary Test");
IssueInput issueInput = issueBuilder.build();
BasicIssue issue = client.createIssue(issueInput).claim();
and the error :
org.apache.http.impl.nio.client.LoggingAsyncRequestExecutor exception
GRAVE: http-outgoing-1 [ACTIVE(5)] HTTP protocol exception: Input length = 1
java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:277)
at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:193)
at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:171)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:171)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:125)
at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:50)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:745)
Have you got an idea to resolve that ?
Tks,
Alexander

Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0] in solr

I am getting exception while adding batch to solrInstance. I have tested my code in different servers. It is working properly all other servers except one server.
my code is
SolrServer solrInstance = solrInstance1;
final Collection<SolrInputDocument> batch = new ArrayList<SolrInputDocument>();
for (Doctor doctor : doctorsList) {
SolrInputDocument doc = new SolrInputDocument();
String docID = doctor.getId();
doc.addField( "_id", docID, 1.0f );
batch.add(doc);
}
if (batch.size() > 0) {
UpdateResponse upres = solrInstance.add( batch );
upres = solrInstance.commit( true, true );
}
getting exception like`
` org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
Can any one please help me. Is there any server setting required to submit batch

Resources