I indeed got a relatively big dataset and my mixed effects logistic regression is like below. Is that normal to take that long to run? or I made some mistakes?
library(lme4)
glmer_EBRD_undersample_1 <- glmer(leave_happened ~
performance_rating_2016 + performance_rating_2017 + performance_rating_2018 + performance_rating_2019 + performance_rating_2020
+ gender
+ target_group
+ target_pmf_band
+ target_hq_or_ro
+ target_office_location_country_distilled
+ target_org_unit_cost_centre_code_distilled
+ target_ebrd_region_distilled
+ target_contract_group_distilled
+ target_position_tenure_group
+ target_length_of_service_group_distilled
+ leaves_to_date
+ moves_to_date
+ joins_to_date
+ applied_count_to_date
+ line_reviewed_to_date
+ interviewed_to_date
+ offered_to_date
+ hired_to_date
+ (1 | person_id)
,
data = train_undersample_1,
family = binomial,
control = glmerControl(optimizer = "bobyqa"),
nAGQ = 10
)
summary(glmer_EBRD_undersample_1)
Also gave a warning like this: Warning in commonArgs(par, fn, control, environment()) :
maxfun < 10 * length(par)^2 is not recommended.
My main target is to create a SAS url (without azure libraries)for file.
I tried to create azure storage SAS with blob and everything worked correctly. When I try to do the same in File i got an error. This is my code:
string azAccName = "AccountName";
string resource = "/upgfile/prt.png";
string endPoint = "https://" + azAccName + ".file.core.windows.net";
string uri = endPoint + resource;
string _now = DateTime.UtcNow.ToString("s") + "Z";
string _noww = DateTime.UtcNow.AddHours(3).AddMinutes(5).ToString("s") + "Z";
string StorageKey = "xxx";
string signedpermissions = "r";
string signedstart = _now;//"2017-02-14"; //yyyy-mm--dd
string signedexpiry = _noww;// "2017-02-14";
string canonicalizedresource = "/file/" + azAccName + resource; //"/blob/myaccount/music/intro.mp3"
string signedidentifier = ""; //YWJjZGVmZw==
string signedIP = "";
string signedProtocol = "https";
string signedversion = "2015-02-21";
string rscc = ""; //Cache-Control
string rscd = "file; attachment"; //Content-Disposition
string rsce = ""; //Content-Encoding
string rscl = ""; //Content-Language
string rsct = "binary"; //Content-Type binary
string StringToSign = signedpermissions + "\n" +
signedstart + "\n" +
signedexpiry + "\n" +
canonicalizedresource + "\n" +
signedidentifier + "\n" +
signedversion + "\n" +
rscc + "\n" +
rscd + "\n" +
rsce + "\n" +
rscl + "\n" +
rsct;
HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(StorageKey));
string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(StringToSign)));
string link = String.Format("{0}?sv={1}&st={2}&se={3}&sr={4}&sp={5}&rscd={8}&rsct={9}&spr={6}&sig={7}",
uri,
signedversion,
signedstart,
signedexpiry,
"c", //b for blob
signedpermissions,
"https",
signature.Replace("/", "%2"),
rscd,///////////////
rsct);
I am getting this error.
<Error>
<Code>AuthenticationFailed</Code>
<Message>
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:92eda75a-001a-0072-501d-1bb6fd000000 Time:2017-08-22T08:03:58.6115733Z
</Message>
<AuthenticationErrorDetail>Signature fields not well formed.</AuthenticationErrorDetail>
</Error>
I have almost the same code for blob(small differences) and it works correctly.
Any suggestions?
I believe the issue is missing parameters in your stringToSign. You must include all parameters as specified here:
StringToSign = signedpermissions + "\n" +
signedstart + "\n" +
signedexpiry + "\n" +
canonicalizedresource + "\n" +
signedidentifier + "\n" +
signedIP + "\n" +
signedProtocol + "\n" +
signedversion + "\n" +
rscc + "\n" +
rscd + "\n" +
rsce + "\n" +
rscl + "\n" +
rsct
If you're not using a parameter (for example signedIP in your case), you must specify an empty line.
Based on this, your StringToSign should be:
string StringToSign = signedpermissions + "\n" +
signedstart + "\n" +
signedexpiry + "\n" +
canonicalizedresource + "\n" +
signedidentifier + "\n" +
"\n" + //For signed IP
"\n" + //For signed Protocol
signedversion + "\n" +
rscc + "\n" +
rscd + "\n" +
rsce + "\n" +
rscl + "\n" +
rsct;
Furthermore sr (signed resource type) in the link should be f (for file) instead of c that you're using.
I am trying to convert the way time is written in the flat file source so it would actually look like time.(If that makes any sense).
Right now I have it as 1215, 630, 10, 1 ,.. etc. So it can be 4-1 character long, so I need to convert it depending on the length of the column.
I am doing this in Derived Column with an expression, however I can't seem to make it work, not sure if my syntax is wrong or what, the expression looks like this, however I'm getting errors:
(LEN([TIME OCC]) == 4) ? (SUBSTRING([TIME OCC],1,2) + ":" + SUBSTRING([TIME OCC],3,2)) : (LEN([TIME OCC]) == 3) ? (SUBSTRING([TIME OCC],1,1) + ":" + SUBSTRING([TIME OCC],2,2)) : (LEN([TIME OCC]) == 2) ? (SUBSTRING([TIME OCC],1,2) + ":00") : (LEN([TIME OCC]) == 1) ? (SUBSTRING([TIME OCC],1,1) + ":00")
When I do it with only two values like below it seems to work perfectly:
LEN([TIME OCC]) == 3 ? SUBSTRING([TIME OCC],1,1) + ":" + SUBSTRING([TIME OCC],2,2) : SUBSTRING([TIME OCC],1,2) + ":" + SUBSTRING([TIME OCC],3,2)
I would appreciate any help, thank you!
All these answers are too complicated. Way too complicated. To me this is far less complicated:
LEFT(RIGHT(("0000" + [TIME OCC]),4),2) +
":" +
RIGHT(RIGHT(("0000" + [TIME OCC]),4),2)
You are ending your expression with an incomplete ternary expression. I am adding some line breaks and indentation to make it more readable:
(LEN([TIME OCC]) == 4) ?
(SUBSTRING([TIME OCC],1,2) + ":" + SUBSTRING([TIME OCC],3,2)) :
(LEN([TIME OCC]) == 3) ?
(SUBSTRING([TIME OCC],1,1) + ":" + SUBSTRING([TIME OCC],2,2)) :
(LEN([TIME OCC]) == 2) ?
(SUBSTRING([TIME OCC],1,2) + ":00") :
(LEN([TIME OCC]) == 1) ?
(SUBSTRING([TIME OCC],1,1) + ":00") <-- there needs to be a : with an ELSE condition here
I don't know if it's necessary, but I would also put a set of parenthesis around each of the nested ternary expressions.
As Tab says, missing the final alternative, try this:
LEN([TIME OCC]) == 4 ? (DT_WSTR,2)SUBSTRING([TIME OCC],1,2) + ":" + (DT_WSTR,2)SUBSTRING([TIME OCC],3,2)
: LEN([TIME OCC]) == 3 ?(DT_WSTR,1)SUBSTRING([TIME OCC],1,1) + ":" + (DT_WSTR,2)SUBSTRING([TIME OCC],2,2)
: LEN([TIME OCC]) == 2 ? (DT_WSTR,2)SUBSTRING([TIME OCC],1,2) + ":00"
: (DT_WSTR,1)SUBSTRING([TIME OCC],1,1) + ":00"
assuming the last alternative with a length 1.
I hope this help.
I don't think that Derrived column is the best way to solve your issue (it may be more complicated). you can add a Script component and mark your column as input, create a new output Column (Type DT_STR) and use the following code inside the OutpoutBuffer0_ProcessInputRow method:
Assuming that TIMEOCC and outColumn are your input and output column
If Not row.TIMEOCC_IsNull AndAslo _
Not String.IsnullorEmpty(Row.TIMEOCC.trim) Then
Select Case Row.TIMEOCC.Trim.Length
Case 1
Row.OutColumn = Row.TIMEOCC & ":00"
Case 2
Row.OutColumn = Row.TIMEOCC & ":00"
Case 3
Row.OutColumn = Row.TIMEOCC.Substring(0,1) & ":" & Row.TIMEOCC.Substring(1,2)
Case 4
Row.OutColumn = Row.TIMEOCC.Substring(0,2) & ":" & Row.TIMEOCC.Substring(2,2)
Case Else
Row.OutColumn = "00:00"
End Select
Else
Row.OutColumn = "00:00"
End If
If you want your solution by derrived column only
i think this is what you are looking for:
ISNULL([TIME OCC]) ? "00:00"
: (LEN([TIME OCC]) == 4 ? ((DT_WSTR,2)SUBSTRING([TIME OCC],1,2) + ":" + (DT_WSTR,2)SUBSTRING([TIME OCC],3,2))
: (LEN([TIME OCC]) == 3 ?((DT_WSTR,2)SUBSTRING([TIME OCC],1,1) + ":" + (DT_WSTR,2)SUBSTRING([TIME OCC],2,2))
: (LEN([TIME OCC]) == 2 ? ((DT_WSTR,2)SUBSTRING([TIME OCC],1,2) + ":00")
: (LEN([TIME OCC]) == 1 ? ((DT_WSTR,2)SUBSTRING([TIME OCC],1,1) + ":00")
: "00:00"))))
References
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/48372e18-598b-4005-9ed8-7ccc8bba30ae/how-to-write-case-statement-in-ssis-expression-using-derived-column-transformation-available-in-bi?forum=sqlintegrationservices
https://www.sqlservercentral.com/Forums/Topic1456697-148-1.aspx
How to write SSIS switch/case expression?
Here is the indices code:
`
g = TitanFactory.build().set("storage.backend", "cassandra")
.set("storage.hostname", "127.0.0.1").open();
TitanManagement mgmt = g.getManagementSystem();
PropertyKey db_local_name = mgmt.makePropertyKey("db_local_name")
.dataType(String.class).make();
mgmt.buildIndex("byDb_local_name", Vertex.class).addKey(db_local_name)
.buildCompositeIndex();
PropertyKey db_schema = mgmt.makePropertyKey("db_schema")
.dataType(String.class).make();
mgmt.buildIndex("byDb_schema", Vertex.class).addKey(db_schema)
.buildCompositeIndex();
PropertyKey db_column = mgmt.makePropertyKey("db_column")
.dataType(String.class).make();
mgmt.buildIndex("byDb_column", Vertex.class).addKey(db_column)
.buildCompositeIndex();
PropertyKey type = mgmt.makePropertyKey("type").dataType(String.class)
.make();
mgmt.buildIndex("byType", Vertex.class).addKey(type)
.buildCompositeIndex();
PropertyKey value = mgmt.makePropertyKey("value")
.dataType(Object.class).make();
mgmt.buildIndex("byValue", Vertex.class).addKey(value)
.buildCompositeIndex();
PropertyKey index = mgmt.makePropertyKey("index")
.dataType(Integer.class).make();
mgmt.buildIndex("byIndex", Vertex.class).addKey(index)
.buildCompositeIndex();
mgmt.commit();`
Here is the search for vertices and then add vertex with 3 edges on 3GHz 2GB RAM pc. It does 830 vertices in 3 hours and I have 100,000 data its too slow. The code is below:
for (Object[] rowObj : list) {
// TXN_ID
Iterator<Vertex> iter = g.query()
.has("db_local_name", "Report Name 1")
.has("db_schema", "MPS").has("db_column", "txn_id")
.has("value", rowObj[0]).vertices().iterator();
if (iter.hasNext()) {
vertex1 = iter.next();
logger.debug("vertex1=" + vertex1.getId() + ","
+ vertex1.getProperty("db_local_name") + ","
+ vertex1.getProperty("db_schema") + ","
+ vertex1.getProperty("db_column") + ","
+ vertex1.getProperty("type") + ","
+ vertex1.getProperty("index") + ","
+ vertex1.getProperty("value"));
}
// TXN_TYPE
iter = g.query().has("db_local_name", "Report Name 1")
.has("db_schema", "MPS").has("db_column", "txn_type")
.has("value", rowObj[1]).vertices().iterator();
if (iter.hasNext()) {
vertex2 = iter.next();
logger.debug("vertex2=" + vertex2.getId() + ","
+ vertex2.getProperty("db_local_name") + ","
+ vertex2.getProperty("db_schema") + ","
+ vertex2.getProperty("db_column") + ","
+ vertex2.getProperty("type") + ","
+ vertex2.getProperty("index") + ","
+ vertex2.getProperty("value"));
}
// WALLET_ID
iter = g.query().has("db_local_name", "Report Name 1")
.has("db_schema", "MPS").has("db_column", "wallet_id")
.has("value", rowObj[2]).vertices().iterator();
if (iter.hasNext()) {
vertex3 = iter.next();
logger.debug("vertex3=" + vertex3.getId() + ","
+ vertex3.getProperty("db_local_name") + ","
+ vertex3.getProperty("db_schema") + ","
+ vertex3.getProperty("db_column") + ","
+ vertex3.getProperty("type") + ","
+ vertex3.getProperty("index") + ","
+ vertex3.getProperty("value"));
}
vertex4 = g.addVertex(null);
vertex4.setProperty("db_local_name", "Report Name 1");
vertex4.setProperty("db_schema", "MPS");
vertex4.setProperty("db_column", "amount");
vertex4.setProperty("type", "indivisual_0");
vertex4.setProperty("value", rowObj[3].toString());
vertex4.setProperty("index", i);
vertex1.addEdge("data", vertex4);
logger.debug("vertex1 added");
vertex2.addEdge("data", vertex4);
logger.debug("vertex2 added");
vertex3.addEdge("data", vertex4);
logger.debug("vertex3 added");
i++;
g.commit();
}
Is there anyway to optimize this code?
For completeness, this question was answered in the Aurelius Graphs mailing list:
https://groups.google.com/forum/#!topic/aureliusgraphs/XKT6aokRfFI
Basically:
build/use a real composite index:
mgmt.buildIndex("by_local_name_schema_value", Vertex.class).addKey(db_local_name).addKey(db_schema).addKey(value).buildComposite();
don't call g.commit() after each loop cycle, instead do something
like this: if (++1%10000 == 0) g.commit()
turn on storage.batch-loading if not already doing so
if all you can throw at cassandra is 2G of RAM consider using BerkleyDB. Cassandra prefers 4G of RAM minimum and would probably like "more"
I don't know the nature of your data, but can you pre-sort it and use BatchGraph as described in the Powers of Ten - Part I blog post and in the wiki - Using BatchGraph would prevent you from having to maintain the transaction described in number 2 above.
I am trying to implement Xmpp protocol in silverlight and trying to connect to Facebook, here I am getting everything correct until <challenge .. > obtained from server.
I am using X Facebook platform authentication.
I have done this with following code:
byte[] ch = Convert.FromBase64String(message.challenge);
string challenge = System.Text.Encoding.UTF8.GetString(ch, 0, ch.Length);
string response = "";
long callId = DateTime.UtcNow.Ticks;
MD5 md = new MD5();
String signature1 = "api_key=203366556348506"
+ "call_id=" + callId
+ "method=auth.xmpp_login"
+ param[2]
+ "session_key=" + messageClient.SessionKey
+ "v=1.0"
+ messageClient.AppSecret;
md.Value = signature1;
response = "method=auth.xmpp_login&api_key=203366556348506&session_key=bc6d6e00462cc2bb73a824bd.4-100001565408667&call_id=" + callId + "&sig=c47d741cb8f18c4e78b990f48e2f63aa&v=1.0&" + param[2];
message.Request = "<response xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\">" + Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(response)) + "</response>";
this.messageClient.SendMessageAsync(message);
But I am getting following message from server:
<failure xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><not-authorized/></failure>
Please let me know where I am going wrong.
Try following code:
String signature1 = "api_key=" + messageClient.ApiKey
+ "call_id=" + callId
+ "method=auth.xmpp_login"
+ param[2]
+ "session_key=" + messageClient.SessionKey
+ "v=1.0"
+ messageClient.AppSecret;
md.Value = signature1;
response = "method=auth.xmpp_login&api_key=" + messageClient.ApiKey + "&session_key=" + messageClient.SessionKey + "&call_id=" + callId + "&sig=" + md.FingerPrint.ToLower() + "&v=1.0&" + param[2];
I have changed response string to the one above.
This has returned success for me. Hope this will help you.