I am creating a subscription in the cloud composer code with a dynamic subscription name. After that I have a PubSubPullSensor operator. Before this task starts I manually send a message to that topic with some attribute so that it can go that subscription. But after I publish a message in the topic, it takes a long time for the PubSubPullSensor, sometime more than 19 minutes.
Is it that attribute which is causing the issue of taking long time?
How can I resolve this?
Please let me know.
with DAG(
"pubsub_trigger_dy_sub_2",
schedule_interval=None,
catchup=False,
default_args=default_arguments,
) as dag:
def samplePythonFn():
print("Getting called from cloud composer")
def _createSubscription(**kwargs):
hook = PubSubHook()
subscriptionName = "airflow-test-" + str(uuid.uuid4())
application = "LumiPortal"
result = hook.create_subscription(
topic=topic,
subscription=subscriptionName,
expiration_policy={
"ttl": Duration(seconds=24*60*60)
},
labels={
"source": "dag"
},
filter_='attributes.correlid = "' + subscriptionName + '"'
#f'attributes.correlid = "{subscriptionName}"'
)
ti = kwargs['ti']
print(subscriptionName)
print(application)
ti.xcom_push(key='Subscription_1', value=subscriptionName)
return subscriptionName
pull_messages_1 = PubSubPullSensor(
task_id="pull_first_messages",
ack_messages=True,
max_messages=1,
return_immediately=True,
project_id=project_id,
subscription="{{ task_instance.xcom_pull(task_ids='CreateSubscription_1') }}",
dag=dag
)
start = DummyOperator(task_id='start')
create_subscription_task_1 = PythonOperator(
task_id='CreateSubscription_1',
python_callable=_createSubscription,
provide_context=True,
)
call_stored_procedure_load = BigQueryInsertJobOperator(
task_id="call_stored_procedure",
configuration={
"query": {
"query": "CALL `amexpoc1.bqTableFullRefresh`('{}','{}','{}')".format(dataset, stage_table,main_table),
"useLegacySql": False,
}
}
)
delete_subscription_task_1 = PubSubDeleteSubscriptionOperator(
task_id='DeleteSubscription_1',
subscription="{{ task_instance.xcom_pull(task_ids='CreateSubscription_1') }}"
)
def pullxcomm1(**kwargs):
ti = kwargs['ti']
fetched_first_message = ti.xcom_pull(key='return_value', task_ids=['pull_first_messages'])
print(fetched_first_message)
return fetched_first_message
pullxcomm_first_message = PythonOperator(
task_id='pull_First_xcomm',
python_callable=pullxcomm1,
provide_context=True,
dag=dag
)
end = DummyOperator(task_id='end')
Related
I am currently making a quiz project for school. The quiz is supposed to evaluate in the front-end, which works perfectly fine, and after that it should send the information to the backend. The backend should save the details to the database and after that it should send the information back to the front-end. The front-end should show a message with some details about how you performed compared to all the others, the information is gotten from the database.
So, the weird thing about all that is that it works sometimes and sometimes not. If I test it on localhost, the code works perfectly fine. The error only occurs on the Ionos server (I got a hosting contract so I do not have access to the console).
Here is the link to my website: https://tor-netzwerk-seminarfach2024.com/ . If you click on the upper left corner and then on the quiz button, you will get to the quiz. If I look at the console and network analytics, when the error occurs, the following message shows up:
Cross-Origin request blocked: The same source rule prohibits reading
the external resource on https://api. tor-netzwerk-seminarfach2024.
com/apache-test/. (Reason: CORS request failed).
https://api.tor-netzwerk-seminarfach2024.com/apache-test/ is my backend and https://tor-netzwerk-seminarfach2024.com/ is my front-end.
The settings file in Django is not the problem. To make it clear, the error does not occur always. Sometimes you have to go again to the website and try it one a few more times until you get the error.
If I look at the network analytics, then I see that the client request works as usual, but the answer field from the server is completely empty. Let's move on to the weirdest part of all: the Data actually gets saved in the Database, there is just sometimes no response?
It would be way to much for a minimum working product but here is the most important code. If you want to test it you can just visit the link to my Website:https://tor-netzwerk-seminarfach2024.com/
Server side with Django
from rest_framework.response import Response
from .models import Lead
import json
from .serializer import TestingSerializer
def updateDigits():
leadObj = Lead.objects.all()
currentScore = leadObj[0].sumScore; currentRequests = leadObj[0].sumRequests
valueBtnOne = leadObj[0].buttonOne; valueBtnTwo = leadObj[0].buttonTwo;
valueBtnThree = leadObj[0].buttonThree; valueBtnFour = leadObj[0].buttonFour;
valueBtnFive = leadObj[0].buttonFive; valueBtnSix = leadObj[0].buttonSix;
valueBtnSeven = leadObj[0].buttonSeven; valueBtnEight = leadObj[0].buttonEight;
valueBtnNine = leadObj[0].buttonNine;
return [valueBtnOne,valueBtnTwo,valueBtnThree,valueBtnFour,valueBtnFive,valueBtnSix,valueBtnSeven,valueBtnEight,valueBtnNine,currentScore,currentRequests]
#api_view(["POST"])
def home(request):
body_unicode = request.body.decode('utf-8')
body = json.loads(body_unicode)
result = body["Result"]
isItRight = body["whichOnesRight"]
ip_adress = get_client_ip(request)
queryset = Lead.objects.all()
if Lead.objects.all().exists():
currentValues = updateDigits()
queryset.update(
sumScore = result + currentValues[9],
sumRequests = 1 + currentValues[10],
buttonOne = currentValues[0] + isItRight[0],
buttonTwo = currentValues[1] + isItRight[1],
buttonThree = currentValues[2] + isItRight[2],
buttonFour = currentValues[3] + isItRight[3],
buttonFive = currentValues[4] + isItRight[4],
buttonSix = currentValues[5] + isItRight[5],
buttonSeven = currentValues[6] + isItRight[6],
buttonEight = currentValues[7] + isItRight[7],
buttonNine = currentValues[8] + isItRight[8],
)
currentValues = updateDigits()
else:
obj = Lead()
obj.save()
serializer = TestingSerializer(queryset[0], many =False)
return Response({**serializer.data, "myResult": result})
The Django model
from django.db import models
class Lead(models.Model):
sumScore = models.IntegerField(default=0)
sumRequests = models.IntegerField(default=0)
buttonOne = models.IntegerField(default=0)
buttonTwo = models.IntegerField(default=0)
buttonThree = models.IntegerField(default=0)
buttonFour = models.IntegerField(default=0)
buttonFive = models.IntegerField(default=0)
buttonSix = models.IntegerField(default=0)
buttonSeven = models.IntegerField(default=0)
buttonEight = models.IntegerField(default=0)
buttonNine = models.IntegerField(default=0)
serializer just serializes _ _ all _ _ ...
Front end with React
function evaluateAndCheckQuiz(){
let countNotChosen = howManyNotChosen()
if(countNotChosen){ answerQuestions(countNotChosen); return }
let points = evaluateQuiz()
document.getElementById("scoreText").style.display = "block"
document.getElementById("scoreNumber").textContent = points
return points
}
Calling sendAndGetQuizAverage with evaluateAndCheckQuiz as a parameter
function sendAndGetQuizAverage(points){
const myPostRequest = {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({"Result":points, "whichOnesRight": whichOnesRight}),
};
let data
fetch('https://api.tor-netzwerk-seminarfach2024.com/apache-test/', myPostRequest).then(
(response) => data = response.json()).then(
(data) => {showResultAnalyse(data)})
}
function showResultAnalyse(data){
console.log(data)
let averageScore = (data["sumScore"] / data["sumRequests"]).toFixed(2)
let myResult = data["myResult"]
changeLinksToStatistics([
data["buttonOne"],data["buttonTwo"],data["buttonThree"],
data["buttonFour"],data["buttonFive"],data["buttonSix"],
data["buttonSeven"],data["buttonEight"],data["buttonNine"]
],
data["sumRequests"]
)
swal(
{
text:`${checkWhichText(myResult,averageScore,data["sumRequests"] )}`,
icon: myResult > averageScore ?"success": "error",
button:"Oki doki",
}
);
}
Background
I have a function that makes a REST API call using UrlFetchApp in Google Scripts.
But the response only returns 2000 records at a time. If there are more records, there is, in the response, a key called nextRecordsUrl, which contains the endpoint and parameters needed to get the next batch of records.
I use a do...while loop to iterate through, pushing the records into a predesignated array, make the next api call. And when it reaches the last batch of records, it exists the do-while loop, then prints (not sure if that's the right term here) the entire to a Google Sheet.
The code
It looks like this:
function getCampaignAssociations() {
clearPage('CampaignAssociations');
var query = '?q=select+CampaignMember.FirstName,CampaignMember.LastName,CampaignMember.LeadId,CampaignMember.ContactId,CampaignMember.Name,CampaignMember.CampaignId,CampaignMember.SystemModstamp,CampaignMember.Email+from+CampaignMember+ORDER+BY+Email ASC,SystemModstamp+ASC';
try {
var arrCampAssociation = getInfoByQuery(query);
if (arrCampAssociation.records.length < 1) {
throw 'there are no records in this query';
}
var campaignAssoc = [];
do {
Logger.log(arrCampAssociation.nextRecordsUrl);
for (var i in arrCampAssociation.records) {
let data = arrCampAssociation.records[i];
let createDate = Utilities.formatDate(new Date(data.SystemModstamp), "GMT", "dd-MM-YYYY");
let a1 = "$A" + (parseInt(i) + 2);
let nameFormula = '=IFERROR(INDEX(Campaigns,MATCH(' + a1 + ',Campaigns!$A$2:A,0),2),"")';
let typeFormula = '=IFERROR(INDEX(Campaigns,MATCH(' + a1 + ',Campaigns!$A$2:A,0),3),"")';
campaignAssoc.push([data.CampaignId, nameFormula, typeFormula, data.Email, data.FirstName, data.LastName, data.LeadId, data.ContactId, createDate]);
}
var arrCampAssociation = getQueryWithFullEndPoint(arrCampAssociation.nextRecordsUrl);
} while (arrCampAssociation.nextRecordsUrl != null && arrCampAssociation.nextRecordsUrl != undefined);
let endRow = campAssocSheet.getLastRow(),
endColumn = campAssocSheet.getLastColumn(),
nameRange = campAssocSheet.getRange(2, 1, endRow, endColumn),
destRange = campAssocSheet.getRange(2, 1, campaignAssoc.length, campaignAssoc[0].length);
destRange.setValues(campaignAssoc);
sheet.setNamedRange('CampaignAssociation', nameRange);
} catch (e) {
Logger.log(e);
Logger.log(arrCampAssociation);
Logger.log(campaignAssoc);
Logger.log(i);
}
}
Issue
Everything works nicely until it comes to printing the array campaignAssoc to the Google Sheet.
See screenshot of the log below. It contains the endpoint for the next both. Notice the timestamp between the earlier logs and the timestamp between the last endPoint and the log where it timed out.
It seems to me that the issue is that when it comes to the printing of the data, it's having issues. If that's the case, have I overloaded the array? There are a total of over 36400 records.
Second attempt
I've tried resetting the array at each loop and printing the array to Google sheet. This is just 2000 records at each attempt and I've definitely done more rows at 1 time, but that didn't help.
Here's the code for that attempt.
function getCampaignAssociations() {
clearPage('CampaignAssociations');
var query = '?q=select+CampaignMember.FirstName,CampaignMember.LastName,CampaignMember.LeadId,CampaignMember.ContactId,CampaignMember.Name,CampaignMember.CampaignId,CampaignMember.SystemModstamp,CampaignMember.Email+from+CampaignMember+ORDER+BY+Email ASC,SystemModstamp+ASC';
try {
var arrCampAssociation = getInfoByQuery(query);
if (arrCampAssociation.records.length < 1) {
throw 'there are no records in this query';
}
do {
Logger.log(arrCampAssociation.nextRecordsUrl);
var campaignAssoc = [];
for (var i in arrCampAssociation.records) {
let data = arrCampAssociation.records[i];
let createDate = Utilities.formatDate(new Date(data.SystemModstamp), "GMT", "dd-MM-YYYY");
let a1 = "$A" + (parseInt(i) + 2);
let nameFormula = '=IFERROR(INDEX(Campaigns,MATCH(' + a1 + ',Campaigns!$A$2:A,0),2),"")';
let typeFormula = '=IFERROR(INDEX(Campaigns,MATCH(' + a1 + ',Campaigns!$A$2:A,0),3),"")';
campaignAssoc.push([data.CampaignId, nameFormula, typeFormula, data.Email, data.FirstName, data.LastName, data.LeadId, data.ContactId, createDate]);
}
let lastRow = campAssocSheet.getLastRow()+1;
campAssocSheet.getRange(lastRow,1,campaignAssoc.length,campaignAssoc[0].length).setValues(campaignAssoc);
var arrCampAssociation = getQueryWithFullEndPoint(arrCampAssociation.nextRecordsUrl);
} while (arrCampAssociation.nextRecordsUrl != null && arrCampAssociation.nextRecordsUrl != undefined);
let endRow = campAssocSheet.getLastRow(),
endColumn = campAssocSheet.getLastColumn(),
nameRange = campAssocSheet.getRange(2, 1, endRow, endColumn);
sheet.setNamedRange('CampaignAssociation', nameRange);
} catch (e) {
Logger.log(e);
Logger.log(arrCampAssociation);
Logger.log(campaignAssoc);
Logger.log(i);
}
}
So here, each loop took a lot longer. Instead of being 1-2 seconds between each loop, it took 45 seconds to a minute between each and timed out after the 4th loop. See the log below:
How do I fix this?
To establish an RPC connection in the community edition we need to specify the rpc username, password and permissions but when we are integrating external database like MySQL and change the datasource type from INMEMORY to "DB" it does not allows to give user properties.
these are the settings I am using in my node.conf
security = {
authService = {
dataSource = {
type = "DB"
passwordEncryption = "SHIRO_1_CRYPT"
connection = {
jdbcUrl = "jdbc:mysql://localhost:3306"
username = "root"
password = "password"
driverClassName = "com.mysql.jdbc.Driver"
}
}
options = {
cache = {
expireAfterSecs = 120
maxEntries = 10000
}
}
}
Maybe I didn't understand your question, but database setup in node.conf is separate from RPC user setup in node.conf:
Database (PostGres in my case)
extraConfig = [
'dataSourceProperties.dataSourceClassName' : 'org.postgresql.ds.PGSimpleDataSource',
'dataSourceProperties.dataSource.url' : 'jdbc:postgresql://localhost:5432/postgres',
'dataSourceProperties.dataSource.user' : 'db_user',
'dataSourceProperties.dataSource.password' : 'db_user_password',
'database.transactionIsolationLevel' : 'READ_COMMITTED',
'database.initialiseSchema' : 'true'
]
RPC User
rpcUsers = [[ user: "rpc_user", "password": "rpc_user_password", "permissions": ["ALL"]]]
Ok, I'm adding my node's node.config (it's part of Corda TestNet, and it's deployed on Google Cloud):
baseDirectory = "."
compatibilityZoneURL = "https://netmap.testnet.r3.com"
emailAddress = "xxx"
jarDirs = [ "plugins", "cordapps" ]
sshd { port = 2222 }
myLegalName = "OU=xxx, O=TESTNET_xxx, L=London, C=GB"
keyStorePassword = "xxx"
trustStorePassword = "xxx"
crlCheckSoftFail = true
database = {
transactionIsolationLevel = "READ_COMMITTED"
initialiseSchema = "true"
}
dataSourceProperties {
dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource"
dataSource.url = "jdbc:postgresql://xxx:xxx/postgres"
dataSource.user = xxx
dataSource.password = xxx
}
p2pAddress = "xxx:xxx"
rpcSettings {
useSsl = false
standAloneBroker = false
address = "0.0.0.0:xxx"
adminAddress = "0.0.0.0:xxx"
}
rpcUsers = [
{ username=cordazoneservice, password=xxx, permissions=[ ALL ] }
]
devMode = false
cordappSignerKeyFingerprintBlacklist = []
useTestClock = false
I'm trying to bind some arguments to my GQL query string.
Everything's fine when I run the query without binding anything:
$result = $DB->query_entities("SELECT * FROM kind WHERE fieldName = 4
LIMIT 1");
But I don't know how to bind some arguments. This is how I'm trying to do so:
function query_entities($query_string, $args=[]){
$gql_query = new Google_Service_Datastore_GqlQuery();
$gql_query->setQueryString($query_string);
$gql_query->setAllowLiteral(true);
if( !empty($args) ){
$binding_args = [];
foreach ($args as $key => $value) {
$binding_value = new Google_Service_Datastore_Value();
$binding_value->setIntegerValue($value);
$arg = new Google_Service_Datastore_GqlQueryArg();
$arg->setName($key);
$arg->setValue($binding_value);
$binding_args[] = $arg;
}
$gql_query->setNameArgs($binding_args);
}
$req = new Google_Service_Datastore_RunQueryRequest();
$req->setGqlQuery($gql_query);
return $this->dataset->runQuery($this->dataset_id, $req, []);
}
$exampleValue = 4;
$result = $DB->query_entities("SELECT * FROM kind WHERE fieldName = :fieldName LIMIT 1", ["fieldName" => $exampleValue]);
Or:
function query_entities($query_string, $args=[]){
$gql_query = new Google_Service_Datastore_GqlQuery();
$gql_query->setQueryString($query_string);
$gql_query->setAllowLiteral(true);
if( !empty($args) ){
$binding_args = [];
foreach ($args as $value) {
$binding_value = new Google_Service_Datastore_Value();
$binding_value->setIntegerValue($value);
$arg = new Google_Service_Datastore_GqlQueryArg();
$arg->setValue($binding_value);
$binding_args[] = $arg;
}
$gql_query->setNumberArgs($binding_args);
}
$req = new Google_Service_Datastore_RunQueryRequest();
$req->setGqlQuery($gql_query);
return $this->dataset->runQuery($this->dataset_id, $req, []);
}
$exampleValue = 4;
$result = $DB->query_entities("SELECT * FROM kind WHERE fieldName = :1 LIMIT 1", [$exampleValue]);
This is what I get:
'Error calling POST https://www.googleapis.com/datastore/v1beta2/datasets/age-of-deployment/runQuery: (400) Lexical error at line 1, column 52. Encountered: ":" (58), after : ""' in ...
Cloud Datastore GQL and Python GQL (App Engine) handle argument binding slightly differently.
In this case, you need to use a # instead of a :, for instance:
SELECT * FROM kind WHERE fieldName = #fieldName LIMIT 1
or
SELECT * FROM kind WHERE fieldName = #1 LIMIT 1
Here's the reference documentation for argument binding in Cloud Datastore GQL.
Is there a working example anywhere of how to create a meeting request using EWS for Exchange 2007 using C#? Which properties are required? I have added a web service reference and can connect to create and send various items but keep getting the error "Set action is invalid for property." on the response messages. It never says what property is invalid
var ews = new ExchangeServiceBinding {
Credentials = new NetworkCredential("user", "pass"),
Url = "https://servername/ews/exchange.asmx",
RequestServerVersionValue = new RequestServerVersion {
Version = ExchangeVersionType.Exchange2007}
};
var startDate = new DateTime(2010, 9, 18, 16, 00, 00);
var meeting = new CalendarItemType {
IsMeeting = true,
IsMeetingSpecified = true,
Subject = "test EWS",
Body = new BodyType {Value = "test body", BodyType1 = BodyTypeType.HTML},
Start = startDate,
StartSpecified = true,
End = startDate.AddHours(1),
EndSpecified = true,
MeetingTimeZone = new TimeZoneType{
TimeZoneName = TimeZone.CurrentTimeZone.StandardName, BaseOffset = "PT0H"},
Location = "Meeting",
RequiredAttendees = new [] {
new AttendeeType{Mailbox =new EmailAddressType{
EmailAddress ="test1#domain.com",RoutingType = "SMTP"}},
new AttendeeType{Mailbox =new EmailAddressType{
EmailAddress ="test2#domain.com",RoutingType = "SMTP"}}
}
};
var request = new CreateItemType {
SendMeetingInvitations =
CalendarItemCreateOrDeleteOperationType.SendToAllAndSaveCopy,
SendMeetingInvitationsSpecified = true,
SavedItemFolderId = new TargetFolderIdType{Item = new DistinguishedFolderIdType{
Id=DistinguishedFolderIdNameType.calendar}},
Items = new NonEmptyArrayOfAllItemsType {Items = new ItemType[] {meeting}}
};
CreateItemResponseType response = ews.CreateItem(request);
var responseMessage = response.ResponseMessages.Items[0];
Microsoft provides an XML example at http://msdn.microsoft.com/en-us/library/aa494190(EXCHG.140).aspx of what the message item should look like. Just setting these properties does not seem to be enough. Can someone tell me what I'm missing or point me to some better examples or documentation?
<CreateItem
xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
SendMeetingInvitations="SendToAllAndSaveCopy" >
<SavedItemFolderId>
<t:DistinguishedFolderId Id="calendar"/>
</SavedItemFolderId>
<Items>
<t:CalendarItem>
<t:Subject>Meeting with attendee0, attendee1, attendee2</t:Subject>
<t:Body BodyType="Text">CalendarItem:TextBody</t:Body>
<t:Start>2006-06-25T10:00:00Z</t:Start>
<t:End>2006-06-25T11:00:00Z</t:End>
<t:Location>CalendarItem:Location</t:Location>
<t:RequiredAttendees>
<t:Attendee>
<t:Mailbox>
<t:EmailAddress>attendee0#example.com</t:EmailAddress>
</t:Mailbox>
</t:Attendee>
<t:Attendee>
<t:Mailbox>
<t:EmailAddress>attendee1#example.com</t:EmailAddress>
</t:Mailbox>
</t:Attendee>
</t:RequiredAttendees>
<t:OptionalAttendees>
<t:Attendee>
<t:Mailbox>
<t:EmailAddress>attendee2#example.com</t:EmailAddress>
</t:Mailbox>
</t:Attendee>
</t:OptionalAttendees>
<t:Resources>
<t:Attendee>
<t:Mailbox>
<t:EmailAddress>room0#example.com</t:EmailAddress>
</t:Mailbox>
</t:Attendee>
</t:Resources>
</t:CalendarItem>
</Items>
</CreateItem>
This is probably too late for you, but this for anyone else trying this.
The issue seems to be with providing the Is-Specified params. I deleted the IsMeetingSpecified and the request worked. Here's the revised CalendarItemType.
var meeting = new CalendarItemType
{
IsMeeting = true,
Subject = "test EWS",
Body = new BodyType { Value = "test body", BodyType1 = BodyTypeType.HTML },
Start = startDate,
StartSpecified = true,
End = startDate.AddHours(1),
EndSpecified = true,
MeetingTimeZone = new TimeZoneType
{
TimeZoneName = TimeZone.CurrentTimeZone.StandardName,
BaseOffset = "PT0H"
},
Location = "Room 1",
RequiredAttendees = new[] {
new AttendeeType
{
Mailbox =new EmailAddressType
{
EmailAddress ="test#test.com"
}
},
}
};