Optimizing my loop python for speed - loops

My function is trying to get a range of yearmonth strings in a list from two dates.
I've been working at it for awhile and managed to code a very poor solution. I would like some help how to optimize my algorithm.
my initial impression was to evaluate the algorithm with
while (beginMonth!=endMonth) and (beginYear!=endYear)
but this evaluated to false for the input values of
beginYear = 2016 and beginMonth = 10
endYear = 2017 and endMonth = 02
returning
["201610", "201611", "201612"]
it exited the loop earlier than i expected.
So given the input
fromDate = "201610"
toDate = "201702"
it should return
["201610", "201611", "201612", "201701", "201702"]
particularly i have concerns about this loop
Loop of inefficiency
while True:
dateRange.append(str(beginYear) + str(beginMonth).zfill(2))
if beginYear >= endYear:
if beginMonth >= endMonth:
break
beginMonth = beginMonth + 1
if (beginMonth > 12):
beginMonth = 1
beginYear = beginYear + 1
print("inc {0}{1}".format(beginYear,str(beginMonth).zfill(2)))
Full code
def getRange(fromDate, toDate):
dateRange = []
beginYear = int(fromDate[:4])
beginMonth = int(fromDate[-2:])
endYear = int(toDate[:4])
endMonth = int(toDate[-2:])
if endYear < beginYear:
print("End year must be less than begin year")
return
elif endYear - beginYear > 25:
print("Maximum number of years reached, can't analyse more than 25
years")
else:
#iterate from current date one month at a time to end date
while True:
dateRange.append(str(beginYear) + str(beginMonth).zfill(2))
if beginYear >= endYear:
if beginMonth >= endMonth:
break
beginMonth = beginMonth + 1
if (beginMonth > 12):
beginMonth = 1
beginYear = beginYear + 1
print("inc {0}{1}".format(beginYear,str(beginMonth).zfill(2)))
return dateRange

Related

Make strategy with arrays in Pine

I'm trying to make a strategy of an indicator, but I get the error: Line 73: Cannot call 'operator >' with argument 'expr0'='call 'alertcondition' (void)'. An argument of 'void' type was used but a 'const float' is expected. How can I change the code to get a correct boolean if statement for the trade entry? I'm very new to pine, hopefully someone can help me.
// This work is licensed under a Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) https://creativecommons.org/licenses/by-nc-sa/4.0/
// © LuxAlgo
//#version=5
// Umgeschrieben von JSt
strategy("Watson Strategie Nadaraya-Watson Envelope [JSt]",overlay=true,max_bars_back=1000,max_lines_count=500,max_labels_count=500)
length = input.float(500,'Window Size',maxval=500,minval=0)
h = input.float(8.,'Bandwidth')
mult = input.float(3.)
src = input.source(close,'Source')
up_col = input.color(#39ff14,'Colors',inline='col')
dn_col = input.color(#ff1100,'',inline='col')
//----
n = bar_index
var k = 2
var upper = array.new_line(0)
var lower = array.new_line(0)
lset(l,x1,y1,x2,y2,col)=>
line.set_xy1(l,x1,y1)
line.set_xy2(l,x2,y2)
line.set_color(l,col)
line.set_width(l,2)
if barstate.isfirst
for i = 0 to length/k-1
array.push(upper,line.new(na,na,na,na))
array.push(lower,line.new(na,na,na,na))
//----
line up = na
line dn = na
//----
cross_up = 0.
cross_dn = 0.
if barstate.islast
y = array.new_float(0)
sum_e = 0.
for i = 0 to length-1
sum = 0.
sumw = 0.
for j = 0 to length-1
w = math.exp(-(math.pow(i-j,2)/(h*h*2)))
sum += src[j]*w
sumw += w
y2 = sum/sumw
sum_e += math.abs(src[i] - y2)
array.push(y,y2)
mae = sum_e/length*mult
for i = 1 to length-1
y2 = array.get(y,i)
y1 = array.get(y,i-1)
up := array.get(upper,i/k)
dn := array.get(lower,i/k)
lset(up,n-i+1,y1 + mae,n-i,y2 + mae,up_col)
lset(dn,n-i+1,y1 - mae,n-i,y2 - mae,dn_col)
if src[i] > y1 + mae and src[i+1] < y1 + mae
label.new(n-i,src[i],'▼',color=#00000000,style=label.style_label_down,textcolor=dn_col,textalign=text.align_center)
if src[i] < y1 - mae and src[i+1] > y1 - mae
label.new(n-i,src[i],'▲',color=#00000000,style=label.style_label_up,textcolor=up_col,textalign=text.align_center)
cross_up := array.get(y,0) + mae
cross_dn := array.get(y,0) - mae
// TestUP = ta.crossover(src,cross_up) > 0
alertcondition(ta.crossover(src,cross_up),'Down','Down')
alertcondition(ta.crossunder(src,cross_dn),'Up','Up')
//---- Alarm für Webhook -----
plot(cross_up, color=#000000, transp=100) //Für den Alert, jedoch Darstellungsfehler → Transparent
plot(cross_dn, color=#000000, transp=100)
plotchar(cross_up, title="cross_up%", char="", location=location.top, color = color.green) // Damit der Wert in der Statusleiste dargestellt wird
plotchar(cross_dn, title="cross_dn%", char="", location=location.top, color = color.red)
//-------------------
// Start Date
// STEP 1. Create inputs that configure the backtest's date range
useDateFilter = input.bool(true, title="Begin Backtest at Start Date",
group="Backtest Time Period")
backtestStartDate = input.time(timestamp("1 Jan 2021"),
title="Start Date", group="Backtest Time Period",
tooltip="This start date is in the time zone of the exchange " +
"where the chart's instrument trades. It doesn't use the time " +
"zone of the chart or of your computer.")
// STEP 2. See if current bar happens on, or later than, the start date
inTradeWindow = not useDateFilter or time >= backtestStartDate
// ---------------
// Enter a long position when the entry rule is triggered
if inTradeWindow and ta.crossover(src,cross_up) > 0
strategy.entry('Long', strategy.long)
// Exit the Long position when the exit rule is triggered
if close > strategy.position_avg_price + 50
strategy.close("Long", comment = "TP")
else if close < strategy.position_avg_price - 50
strategy.close("Long", comment = "SL")
I tried the ta.crossover(src,cross_up) to compare it to zero, but it doesn't work.
ta.crossover() returns a bool value.
ta.crossover(source1, source2) → series bool
So, comparing its return value with some number does not make any sense and the compiler will complain: ta.crossover(src,cross_up) > 0.
You should just do:
if inTradeWindow and ta.crossover(src,cross_up)
strategy.entry('Long', strategy.long)

Manipulating character arrays quickly in R data.table [duplicate]

This question already has answers here:
Faster way to read fixed-width files
(4 answers)
Closed 4 years ago.
I have a huge datatset (14GB, 200 Mn rows) of character vector. I've fread it (took > 30 mins on 48 core 128 GB server). The string contains concatenated information on various fields. For instance, the first row of my table looks like:
2014120900000001091500bbbbcompany_name00032401
where the first 8 characters represent date in YYYYMMDD format, next 8 characters are id, next 6 the time in HHMMSS format and then next 16 are name (prefixed with b's) and the last 8 are price (2 decimal places).
I need to transfer the above 1 column data.table into 5 columns: date, id, time, name, price.
For the above character vector that will turn out to be: date = "2014-12-09", id = 1, time = "09:15:00", name = "company_name", price = 324.01
I am looking for a (very) fast and efficient dplyr / data.table solution. Right now I am doing it with using substr:
date = as.Date(substr(d, 1, 8), "%Y%m%d");
and it's taking forever to execute!
Update: With readr::read_fwf I am able to read the file in 5-10 mins. Apparently, the reading is faster than fread. Below is the code:
f = "file_name";
num_cols = 5;
col_widths = c(8,8,6,16,8);
col_classes = "ciccn";
col_names = c("date", "id", "time", "name", "price");
# takes 5-10 mins
data = readr::read_fwf(file = f, col_positions = readr::fwf_widths(col_widths, col_names), col_types = col_classes, progress = T);
setDT(data);
# object.size(data) / 2^30; # 17.5 GB
A possible solution:
library(data.table)
library(stringi)
widths <- c(8,8,6,16,8)
sp <- c(1, cumsum(widths[-length(widths)]) + 1)
ep <- cumsum(widths)
DT[, lapply(seq_along(sp), function(i) stri_sub(V1, sp[i], ep[i]))]
which gives:
V1 V2 V3 V4 V5
1: 20141209 00000001 091500 bbbbcompany_name 00032401
Including some additional processing to get the desired result:
DT[, lapply(seq_along(sp), function(i) stri_sub(V1, sp[i], ep[i]))
][, .(date = as.Date(V1, "%Y%m%d"),
id = as.integer(V2),
time = as.ITime(V3, "%H%M%S"),
name = sub("^(bbbb)","",V4),
price = as.numeric(V5)/100)]
which gives:
date id time name price
1: 2014-12-09 1 09:15:00 company_name 324.01
But you are actually reading a fixed width file. So could also consider read.fwf from base R or read_fwffrom readr or write your own fread.fwf-function like I did a while ago:
fread.fwf <- function(file, widths, enc = "UTF-8") {
sp <- c(1, cumsum(widths[-length(widths)]) + 1)
ep <- cumsum(widths)
fread(file = file, header = FALSE, sep = "\n", encoding = enc)[, lapply(seq_along(sp), function(i) stri_sub(V1, sp[i], ep[i]))]
}
Used data:
DT <- data.table(V1 = "2014120900000001091500bbbbcompany_name00032401")
Maybe your solution is not so bad.
I am using this data:
df <- data.table(text = rep("2014120900000001091500bbbbcompany_name00032401", 100000))
Your solution:
> system.time(df[, .(date = as.Date(substr(text, 1, 8), "%Y%m%d"),
+ id = as.integer(substr(text, 9, 16)),
+ time = substr(text, 17, 22),
+ name = substr(text, 23, 38),
+ price = as.numeric(substr(text, 39, 46))/100)])
user system elapsed
0.17 0.00 0.17
#Jaap solution:
> library(data.table)
> library(stringi)
>
> widths <- c(8,8,6,16,8)
> sp <- c(1, cumsum(widths[-length(widths)]) + 1)
> ep <- cumsum(widths)
>
> system.time(df[, lapply(seq_along(sp), function(i) stri_sub(text, sp[i], ep[i]))
+ ][, .(date = as.Date(V1, "%Y%m%d"),
+ id = as.integer(V2),
+ time = V3,
+ name = sub("^(bbbb)","",V4),
+ price = as.numeric(V5)/100)])
user system elapsed
0.20 0.00 0.21
An attempt with read.fwf:
> setClass("myDate")
> setAs("character","myDate", function(from) as.Date(from, format = "%Y%m%d"))
> setClass("myNumeric")
> setAs("character","myNumeric", function(from) as.numeric(from)/100)
>
> ff <- function(x) {
+ file <- textConnection(x)
+ read.fwf(file, c(8, 8, 6, 16, 8),
+ col.names = c("date", "id", "time", "name", "price"),
+ colClasses = c("myDate", "integer", "character", "character", "myNumeric"))
+ }
>
> system.time(df[, as.list(ff(text))])
user system elapsed
2.33 6.15 8.49
All outputs are the same.
Maybe try using matrix with numeric instead of data.frame. Aggregation should take less time.

daysBetween >= 23 HOURS means 1 more day and < 23 HOURS = the same day?

What is the rule to calculate daysBetween ??
Try a simple apex code and find something strange.
When two days are compare, if 23hours between them they are 1 day between. Why not 24 hours ?
The test code:
Datetime dt,dt0,dt1;
dt = Date.Today();
dt0 = dt.addHours(23);
dt0 = dt0.addMinutes(-1);
dt1 = dt.addHours(23);
dt1 = dt1.addMinutes(0);
Date d,d0,d1;
d=Date.ValueOf(dt);
d0=Date.ValueOf(dt0);
d1=Date.ValueOf(dt1);
System.debug(dt);
System.debug(d0+'|'+d.daysBetween(d0)); // 0
System.debug(d1+'|'+d.daysBetween(d1)); // 1

Calendar buttons aren't being removed correctly - Lua

I'm trying to make a calendar to be able to switch from month to month. My problem is in removing each day button when the next or previous buttons are touched.
Here is my code for the Previous Button to switch from the current month to the previous month. My Next button code is almost exactly the same. It works perfectly fine when I tap the button for the first time, but when I tap it again, I get an error at the child:removeSelf() line, and the print message tells me there are 61 elements in the table. It seems to add extra buttons to the table every time I go to a month that hasn't been seen yet.
This is really frustrating to me because I don't see any reason why the code is making extra buttons for every month, and even if it does, each one should still get removed when the button is tapped. Can someone please help me?
local prev_function = function(event)
if event.phase == "release" then
today.year = 2012
today.month = 3
today.day = 29
today.wday = 5
if monthNum == 1 then
monthNum = 12
yearNum = yearNum - 1
elseif monthNum ~= 1 then
monthNum = monthNum - 1
end
local month = ""
if monthNum == 1 then month = "January"
elseif monthNum == 2 then month = "February"
elseif monthNum == 3 then month = "March"
elseif monthNum == 4 then month = "April"
elseif monthNum == 5 then month = "May"
elseif monthNum == 6 then month = "June"
elseif monthNum == 7 then month = "July"
elseif monthNum == 8 then month = "August"
elseif monthNum == 9 then month = "September"
elseif monthNum == 10 then month = "October"
elseif monthNum == 11 then month = "November"
elseif monthNum == 12 then month = "December"
end
monthText.text = month .. " " .. yearNum
print("Table elements before button deletion: " .. #buttonTable)
for i = #buttonTable, 1, -1 do
--[[if button[i] ~= nil then
table.remove(buttonTable)
button[i]:removeSelf()
button[i] = nil
end--]]
local child = table.remove(buttonTable)
if child ~= nil then
child:removeSelf()
child = nil
end
end
print("Table elements after button deletion: " .. #buttonTable)
next_button.alpha = 1
for i = 1, math.floor(numYears * 365.25) do
dateTable[i] = calendar.getInfo(today) --calculate the next day's date
if dateTable[i].year == yearNum and dateTable[i].month == monthNum then -- create a button if the date's year and month match the desired month
button[i] = ui.newButton{
default = "images/day.png",
over = "images/dayover.png",
text = dateTable[i].day,
size = 30,
font = native.systemFontBold,
textColor = {0, 0, 0, 255},
onEvent = addExpense_function,
offset = -35 }
if dateTable[i].wday == 1 then button[i].x = math.floor(col/2)
elseif dateTable[i].wday == 2 then button[i].x = (col * 1) + math.floor(col/2)
elseif dateTable[i].wday == 3 then button[i].x = (col * 2) + math.floor(col/2)
elseif dateTable[i].wday == 4 then button[i].x = (col * 3) + math.floor(col/2)
elseif dateTable[i].wday == 5 then button[i].x = (col * 4) + math.floor(col/2)
elseif dateTable[i].wday == 6 then button[i].x = (col * 5) + math.floor(col/2)
elseif dateTable[i].wday == 7 then button[i].x = (col * 6) + math.floor(col/2)
end
if dateTable[i].day == 1 then button[i].y = wDayBar.y + wDayBar.height/2 + math.floor(row/2)
elseif dateTable[i].wday == 1 then button[i].y = button[i-1].y + row
else button[i].y = button[i-1].y
end
end
today = dateTable[i]
table.insert(buttonTable, button[i])
--button[i].id = "button_" .. i
end
print("Table elements after button creation: " .. #buttonTable)
end
return true
end
The reason why the code in the question doesn't work (and putting the table.insert inside the if...then does work) is due to the way you're using the button table.
The loop that starts for i = 1, math.floor(numYears * 365.25) do is going to create indices from 1 to a few hundred/thousand (depending on numYears).
However, as you're using button[i]= for filling the table and you're only filling 30 or so at a time, what you're creating is a sparse table with a few non-nil entries somewhere in the middle of the table.
Now with table.insert(buttonTable, button[i]) outside the if..then what actually happens is that most of the "inserts" are inserting nil. First time around this actually works ok for 2 reasons:
inserting nil on an empty table does nothing nor does inserting nil at the end of a table
there are only a month's worth of non-nil entries in button so only a month's worth will be inserted into buttonTable
With button and buttonTable setup like this, the first part of the call to the previous or next functions works as expected and removeSelf will be called on the month's worth of buttons. However this doesn't actually remove the button from the button table.
So when you come to the numYears loop again, not only will you add your freshly created buttons to both button and buttonTable, you'll also still have the buttons that you've just called removeSelf on in button so these will be added to buttonTable again. Crucially though, they are just tables now, so when you call removeSelf on them the second time it barfs. I presume this is the cause of the error you saw.
Moving table.insert(buttonTable, button[i]) to the if...then will resolve this problem as it will only ever add freshly minted buttons to buttonTable that are then removed each time next or previous are called.
A word of caution though. The button table is a bit of a mess. After a bit of navigating around, it'll have a bunch of dead buttons (that can't be garbage collected unless you've declared the values as weak) plus a month of working buttons. If you were to try to do anything with it, the chances are that you'll get problems.
It's not clear from the code if you even need the button table. If it's not required elsewhere it would probably be safer to just hold the button references in buttonTable.
Aside from making a new table for months as well as column coordinates like Nicol suggested, I found that I needed to put the table.insert() command inside the if ... then structure. I'm not sure exactly why putting it immediately after the if ... then structure caused it to not work properly, but I'm guessing it's because the buttons are being created inside the structure.
local prev_function = function(event)
if event.phase == "release" then
--set date to start from
today.year = 2012
today.month = 2
today.day = 29
today.wday = 4
--determine new month number
if monthNum == 1 then
monthNum = 12
yearNum = yearNum - 1
elseif monthNum ~= 1 then
monthNum = monthNum - 1
end
--set month string
local month = monthTable[monthNum]
monthText.text = month .. " " .. yearNum
print("Table elements before button deletion: " .. #buttonTable)
--remove all elements in buttonTable
for i = #buttonTable, 1, -1 do
local child = table.remove(buttonTable)
if child ~= nil then
child:removeSelf()
child = nil
end
end
print("Table elements after button deletion: " .. #buttonTable)
next_button.alpha = 1
--hide prev_button if we get to the month after the month we started from
if monthNum == today.month + 1 and yearNum == today.year then
prev_button.alpha = 0
end
--generate dates for specified number of years
for i = 1, math.floor(numYears * 365.25) do
dateTable[i] = calendar.getInfo(today)
--create a button for each date inside desired month
if dateTable[i].year == yearNum and dateTable[i].month == monthNum then
button[i] = ui.newButton{
default = "images/day.png",
over = "images/dayover.png",
text = dateTable[i].day,
size = 30,
font = native.systemFontBold,
textColor = {0, 0, 0, 255},
onEvent = addExpense_function,
offset = -35 }
--set x and y
button[i].x = colTable[dateTable[i].wday]
if dateTable[i].day == 1 then button[i].y = wDayBar.y + wDayBar.height/2 + math.floor(row/2)
elseif dateTable[i].wday == 1 then button[i].y = button[i-1].y + row
else button[i].y = button[i-1].y
end
table.insert(buttonTable, button[i])
end
--change 'today' for next iteration in the for loop
today = dateTable[i]
end
print("Table elements after button creation: " .. #buttonTable)
end
return true
end

Crystal Reports error: A subscript must be between 1 and size of array

The following function returns the "A subscript must be between 1 and size of array"-error when run in Crystal Reports XI. Any idea why and how to fix it?
Function (optional BooleanVar start := true)
DateVar Array reportdates := [
cDate(2012, 10, 22),
cDate(2012, 11, 15),
cDate(2013, 01, 23),
cDate(2013, 02, 20),
// some more lines of dates...
cDate(2014, 01, 02)
];
// Here is some code that sorts the array just to be sure.
// Removed from question
// Find index of last reportdate not later than today
NumberVar stopIndex;
for i := 1 to UBound(reportdates) do (
if CurrentDate >= reportdates[i] then
stopIndex := i
);
DateTimeVar returnDateTime;
if start = true then ( // return start date
NumberVar startIndex;
if stopIndex = 1 then
startIndex = 1
else
startIndex = stopIndex - 1;
//*** The error occurs here
returnDateTime := cDateTime(reportdates[startIndex], cTime(0,0,0));
//*** The error occurs here
)
else ( // return stop date
DateVar stopDate = reportdates[stopIndex];
returnDateTime := dateAdd("d", -1, cDateTime(reportdates[stopIndex], cTime(23,59,59)));
);
returnDateTime;
Note:
I found that the above function returns an earlier stop date than start date if run before the second date in the array. I rewrote the function to counter that and then I did not have a situation that produced the error in question, but I would still be interested in why the error occured in this function and how to handle it.
NumberVar startIndex;
if stopIndex = 1 then
startIndex = 1
else
startIndex = stopIndex - 1;
should of course be
NumberVar startIndex;
if stopIndex = 1 then
startIndex := 1
else
startIndex := stopIndex - 1;
and now it works...
NumberVar stopIndex;
should also be changed to
NumberVar stopIndex := 1;
to avoid errors if report is run before first report date.

Resources