I'm running a process, which uses a set of parameters defined in a file parameters.py
Now, I'd like to swept the value of one of the parameters over certain range. Thus, I wrote a batch file to automatise the procedure.
Problem
My problem is that I need to modify the 20th line of the file parameter, from value = 200 to value = $VAR, where $VAR sets the new value of the parameter.
Question
Is there a bash command that allows to change a specific target line of the file?
Thank you and cheers.
Try doing this with sed :
sed -i "20s/value = 200/value = $VAR/" file.txt
That will change value = 200 to value = $VAR on line 20. s/// is a skeleton for sed substitutions : s/before/after/
The -i switch edit the file in place
While it's certainly possible, I wouldn't recommend it. One day, someone will add a comment before this line or an empty line and your script will break.
Use sed instead:
sed -e 's/value = 200/value = $VAR/' < parameters.py
An even better solution would be to move everything in parameters.py into parameterdefaults.py like so:
parameterdefaults.py:
options = {
"value": 200,
}
parameters.py:
import parameterdefaults
options = dict( **parameterdefaults.options ) # copy defaults
options['value'] = 10 # Override the few values you need to change
or even better:
codethatneedsparameters.py:
import parameterdefaults
import parameters
options = dict( **parameterdefaults.options ) # copy defaults
options.update( parameters.options ) # merge with changes
Now you can write:
parameters.py:
options = {
"value": 10,
}
The easiest way (but not the best) to do this instead of hacking the script is to change it from
value = 200
to
import os # somewhere at the top if need be.
value = int(os.environ.get('VAR', 200))
That way it can be modified from the calling process what ever that may be.
Related
I'm working on a network using triplet mining for training. In order to make it work properly, I need my batches to contain several images of the same class. The problem I'm currently facing is that I have 751 classes, for a total of 12,937 pictures, and a batch size of 48 pictures. When shuffling the dataset using the command below, the odds to get pictures from the same class are really low, making the triplet mining inefficient.
dataset = dataset.shuffle(12937)
What I would need instead is a way of generating batches that contain a specific number of pictures for every class represented in this batch. As an example, let's say here that I want 12 classes per batch, there would be 4 pictures for each of them.
Another problem I'm facing is how would I shuffle this dataset at the end of every epoch so that I can have different batches that still follow the condition fixed above, that is 12 classes, 4 pictures for each one of them?
Is there any proper way to do it? I can't really find one. Please let me know if I'm unclear, and if you need further details.
================ EDIT ================
I've been trying a few things, and came up with something that would do what I want. The function would be the following:
counter = 0.
# Assuming a format such as (data, label)
def predicate(data, label):
global counter
allowed_labels = tf.constant([counter])
isallowed = tf.equal(allowed_labels, tf.cast(label, tf.float32))
reduced = tf.reduce_sum(tf.cast(isallowed, tf.float32))
counter += 1
return tf.greater(reduced, tf.constant(0.))
##tf.function
def custom_shuffle(train_dataset, batch_size, samples_per_class = 4, iterations_in_epoch = 100, database='market'):
assert batch_size%samples_per_class==0, F'batch size must be a {samples_per_class} multiple.'
if database == 'market':
class_nbr = 751
else:
raise Exception('Unsuported database yet')
all_datasets = [train_dataset.filter(predicate) for _ in range(class_nbr)] # Every element of this array is a dataset of one class
for i in range(iterations_in_epoch):
choice = tf.random.uniform(
shape=(batch_size//samples_per_class,),
minval=0,
maxval=class_nbr,
dtype=tf.dtypes.int64,
) # Which classes will be in batch
choice = tf.data.Dataset.from_tensor_slices(tf.concat([choice for _ in range(4)], axis=0)) # Exactly 4 picture from each class in the batch
batch = tf.data.experimental.choose_from_datasets(all_datasets, choice)
if i==0:
all_batches = batch
else:
all_batches = all_batches.concatenate(batch)
all_batches = all_batches.batch(batch_size)
return all_batches
It does what I want, however the returned dataset is extremely slow to iterate, making modele learning impossible. As per this thread, I understood that I needed to decorate custom_shuffle with #tf.function, as the one commented out. However, when doing so, it raises the following error:
Traceback (most recent call last):
File "training.py", line 137, in <module>
main()
File "training.py", line 80, in main
train_dataset = get_dataset(TRAINING_FILENAMES, IMG_SIZE, BATCH_SIZE, database=database, func_type='train')
File "E:\Morgan\TransReID_TF\tfr_to_dataset.py", line 260, in get_dataset
dataset = custom_shuffle(dataset, batch_size)
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
result = self._call(*args, **kwds)
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\def_function.py", line 846, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
return self._call_flat(
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
outputs = execute.execute(
File "D:\Programs\Anaconda3\envs\AlignedReID_TF\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: No unary variant device copy function found for direction: 1 and Variant type_index: class tensorflow::data::`anonymous namespace'::DatasetVariantWrapper
[[{{node BatchDatasetV2/_206}}]] [Op:__inference_custom_shuffle_11485]
Function call stack:
custom_shuffle
Which I don't understand, and don't see how to fix.
Is there something I'm doing wrong?
PS: I'm aware the lack of minimal code to reproduce this behavior makes it hard to debug, I'll try to provide some as soon as possible.
This is my Python Program that I have been having some issues with:
-- coding: cp1252 --
from time import gmtime, strftime
print("Welcome to the PiggyBank version 1.")
num_write = int(input("How much money would you like to store in your PiggyBank?"))
f = open("PiggyBanks_Records.txt", "w")
current_time = strftime("%Y-%m-%d %H:%M:%S", gmtime())
convert_1 = str(current_time)
convert_2 = str(int(num_write))
add_1 = ("\n" + convert_1 + " £" + convert_2)
add_2 = ("\n" + add_1) #Tried to make it so new line is added every time the program is run
final_record = str(add_2)
print("Final file written to the PiggyBank: " + final_record)
#Write to File
f.write(final_record)
f.close()
Right now whenever the program writes to the file it over-writes. I would preferably would like to keep, like a history of the amounts added. If anyone can help so the string that needs to be written to the .txt file goes down by one line and essentially keeps going for ever. I am also open to any suggestion on how I can shorten this code.
You need to open your file with append mode :
f = open("PiggyBanks_Records.txt", "a")
Using the 'w' write option with open automatically looks for the specified file, and deletes its contents if it already exists (which you can read about here) or creates it if it doesn't. Use 'a' instead to add / append to the file.
Python 3 program allows people to choose from list of employee names.
Data held on text file look like this: ('larry', 3, 100)
(being the persons name, weeks worked and payment)
I need a way to assign each part of the text file to a new variable,
so that the user can enter a new amount of weeks and the program calculates the new payment.
Below is my code and attempt at figuring it out.
import os
choices = [f for f in os.listdir(os.curdir) if f.endswith(".txt")]
print (choices)
emp_choice = input("choose an employee:")
file = open(emp_choice + ".txt")
data = file.readlines()
name = data[0]
weeks_worked = data[1]
weekly_payment= data[2]
new_weeks = int(input ("Enter new number of weeks"))
new_payment = new_weeks * weekly_payment
print (name + "will now be paid" + str(new_payment))
currently you are assigning the first three lines form the file to name, weeks_worked and weekly_payment. but what you want (i think) is to separate a single line, formatted as ('larry', 3, 100) (does each file have only one line?).
so you probably want code like:
from re import compile
# your code to choose file
line_format = compile(r"\s*\(\s*'([^']*)'\s*,\s*(\d+)\s*,\s*(\d+)\s*\)")
file = open(emp_choice + ".txt")
line = file.readline() # read the first line only
match = line_format.match(line)
if match:
name, weeks_worked, weekly_payment = match.groups()
else:
raise Exception('Could not match %s' % line)
# your code to update information
the regular expression looks complicated, but is really quite simple:
\(...\) matches the parentheses in the line
\s* matches optional spaces (it's not clear to me if you have spaces or not
in various places between words, so this matches just in case)
\d+ matches a number (1 or more digits)
[^']* matches anything except a quote (so matches the name)
(...) (without the \ backslashes) indicates a group that you want to read
afterwards by calling .groups()
and these are built from simpler parts (like * and + and \d) which are described at http://docs.python.org/2/library/re.html
if you want to repeat this for many lines, you probably want something like:
name, weeks_worked, weekly_payment = [], [], []
for line in file.readlines():
match = line_format.match(line)
if match:
name.append(match.group(1))
weeks_worked.append(match.group(2))
weekly_payment.append(match.group(3))
else:
raise ...
Is there a way to assign file names to set varibles using a GUI? Say I have 6 file sets which contain 4 colors each (blue, green, nir, red). There are 24 files in total, so i'd need 24 variables. And I want the set varialbes to be something like
blue1
green1
nir1
red1
blue2
green2
nir2
red2
etc...
Currently I'm trying to use GUIDE to creat a custom GUI that will allow the user to select the files they wish and have them assigned to certain variables. I am thinking something along the lines of having 24 popupmenus that are attached to a file directory and allows the user to select which file they want, and then it will assign that file and it's path to a variable (blue1 for example) I also want 24 check boxes to associate with an if statement
Let's say popupmenu1 is associated with the variable blue1 and checkbox1
if checkbox1 == checked
do import
elseif checkbox1 == unchecked
fill with zeros
I have the basic frame of the GUI created, I am just unclear on how to apply the file select and then associate the if statements, etc...
If you know the variable files in advance, it's bad practice (look also here and here) to use string defined variable names like this:
var1name = 'blue';
var2name = 'red';
% etc.
% load data
datablue=rand(4,1);
datared =rand(4,1);
% assign
eval([var1name '1 = datablue(1);']);
eval([var2name '1 = datared (1);']);
% etc.
eval([var1name '2 = datablue(2);']);
eval([var2name '1 = datared (2);']);
% etc
It's much easier and better to just use an ordinary array, given the variable name is not changing or application dependend, which in my example I already have as datablue and datared.
Another option if you'd like user defined variable names is to use an array of structs:
var1name = 'blue';
var2name = 'red';
sample(1).(var1name) = datablue(1);
sample(1).(var2name) = datared (1);
% ...
sample(2).(var1name) = datablue(2);
sample(2).(var2name) = datared (2);
Try some of these out, and only if you have a very good reason, resort to eval!
for k = 1:6
blue(k) = sprintf('blue%d', k);
green(k) = sprintf('green%d', k);
nir(k) = sprintf('nir%d', k);
red(k) = sprintf('red%d', k);
end
This will create the variable names for you. Then you can use assignin (i believe) or eval to set the values to the variable names.
Hi I am trying to represent a file location as a variable because the finial script will be run on another machine. This is the code I have tried followed by the error I get. It seems to me that some how python is adding "\" and that is causing the problem. If that is the case how do I get it not to insert the "\"? Thank you
F = 'C:\Documents and Settings\myfile.txt','r'
f = open(F)
and the error
TypeError: invalid file: ('C:\\Documents and Settings\\myfile.txt', 'r')
From the docs:
http://docs.python.org/tutorial/inputoutput.html#reading-and-writing-files
Try this:
F = r'C:\Documents and Settings\myfile.txt'
f = open(F, 'r')
About the "double backslashes" - you need to escape backslashes in your strings or use r'string', see this:
http://docs.python.org/release/2.5.2/ref/strings.html
E.g. try this:
>>> a = 'a\nb'
>>> print a
a
b
To get what you expect, you need this:
>>> a = r'a\nb'
>>> print a
a\nb
or this:
>>> a = 'a\\nb'
>>> print a
a\nb
Try
f=open('C:\Documents and Settings\myfile.txt','r')
Instead of using the variable F. the way you have it 'r' is part of the file name, which it is not.
Instead of writing / or \, you should do this:
import os
F = os.path.join(
"C:",
os.path.join(
"Documents and Settings", "myfile.txt"
)
)
f = open(F, 'r')`
so that it uses / or \ according to your os.
(Although if you write C:/ it must mean you want to run your code on Windows... Hum...)