Open policy agent - explicit logical AND in CI pipeline - eval

I am trying to write a policy which governs administrator username compatibility, which consists of three rules: alphanumeric value, not part of disallowed names (admin, administrator etc.), and longer than 5 characters.
I have found that when using OPA as part of a CI pipeline (which is my use case), the most comfortable solution is to create an object (dictionary) which contains policy results so that I can query them directly. My line inside the CI pipeline will look like this:
for file in rego_directory:
opa eval -i file -d data.json "package_name.policy"
Which doesn't print all of the variables and temporary resources I use inside the rego file (which saves a lot of logs and outputs). In order to make this "policy" object, I have inside my rego file the following pattern:
policy[policy_name] = result {
policy_name :=
...
computations...
...
result := <logical condition>
}
Now, my questions are as following: This doesn't seem like a best practice to me. Is there another way to simply omit the variables from the output? previously I've worked with separate values (i.e. something like this:
default policy_1 = false
default policy_2 = false
policy_1 = {
<logical condition>
}
policy_2 = {
<logical condition>
}
Second question: How do I create an output dictionary (since JSON output of a dictionary is a nice format to work with, after all) which can satisfy multiple conditions? looking back at my example, I can't write something like this:
policy[policy_name] = result {
policy_name :=
...
computations...
...
result := <logical condition>
result := <logical condition 2>
}
Since this is double assignment, which is invalid. Even if I use = instead of :=, it creates conflicts if one term is true and the other is false, and errors are not what I'm looking for (I need boolean answers). How do I create a complex rule whose output I can put inside this dictionary?
Thanks in advance.

TLDR; to answer your questions explicitly:
Now, my questions are as following: This doesn't seem like a best practice to me. Is there another way to simply omit the variables from the output?
There's nothing wrong with having your software query for the value generated by some rule. In fact, rules are the fundamental way of defining abstractions in your policies so that you can decouple policy decision-making (i.e., the logic that produces the decision) from policy enforcement (i.e., the logic/actions taken based on the policy decision.)
The only real alternative is to query for a set of rules inside one or more packages like you showed.
Second question: How do I create an output dictionary (since JSON output of a dictionary is a nice format to work with, after all) which can satisfy multiple conditions? looking back at my example, I can't write something like this:
You're asking how to express Logical OR. In this case, you would create multiple definitions (we call them "incremental definitions" in some places) of the policy rule:
policy[policy_name] = result {
policy_name :=
...
computations...
...
result := <logical condition>
}
policy[policy_name2] = result2 {
policy_name2 :=
...
some other computations...
...
result2 := <some other logical condition>
}
This snippet defines an object (named policy) that maps policy_name to result and policy_name2 to result2. We call this kind of rule a Partial Object. You can also define Partial Sets. When you define a Partial Object, you need to ensure that each key maps to at-most-one value (otherwise you'll get a runtime error.)
Another way of structuring your policy would be to (essentially) define a deny-list using partial sets:
package usernames
deny["username must use alphanumeric characters"] {
re_match("[a-zA-Z0-9]", input.username)
}
deny["username must be at least 5 characters long"] {
count(input.username) < 5
}
deny["username is reserved"] {
reserved_usernames[input.username]
}
reserved_usernames := {"admin", "root"}
Then your CI pipeline can simply query for deny:
opa eval -i input.json 'data.usernames.deny'
The result will contain reasons why the username should not be allowed.

Related

Counting array elements in Ruby (unexpected results by the count( ) function)

In my understanding the following ruby expressions should produce the same result.
Apparently I am missing something, this is a way too serious bug to go unnoticed...
# returns the number of ALL elements in the array
count = #quotation.quotation_items.count { |x| x.placement == current_placement}
# Does what I expect
count = (#quotation.quotation_items.find_all { |x| x.placement == current_placement }).length
quotation_items above is an ActiveRecord has_many association
#count does not take a block like that.
If you want to use conditions on a count, you would do:
#quotation.quotation_items.count(:conditions => "placement = #{current_placement}")
http://apidock.com/rails/ActiveRecord/Calculations/ClassMethods/count
If you're using ActiveRecord you need to keep in mind that there's a point where it's compiling conditions and clauses for a query and a point where you have a result set. Certain things only work in one mode or the other, though it tries to keep things pretty consistent regardless.
In your case you are expecting count to work like Enumerable, but that's still a database-level operator.
To fix that:
#quotation.quotation_items.where(placement: current_placement).count
That composes a query that counts only the things you need, something approximating:
SELECT COUNT(*) FROM quotation_items WHERE quotation_id=? AND placement=?
That's something that yields a single number and is considerably different than selecting every record, instantiating into models, then counting those using Enumerable.
Your usage of #count is incorrect.
I believe it doesn't accept a block. I'm not sure why it didn't return an error though.
you can use it like this :
count = #quotation.quotation_items.map{ |x| x.placement == current_placement}.count(true)

Generate sql query by anorm, with all nulls except one

I developing web application with play framework 2.3.8 and scala, with complex architecture on backend and front-end side. As backend we use MS SQL, with many stored procedures, and called it by anorm. And here one of the problems.
I need to update some fields in database. The front end calls play framework, and recive name of the field, and value. Then I parse, field name, and then I need to generate SQL Query for update field. I need assign null, for all parameters, except recived parameter. I try to do it like that:
def updateCensusPaperXX(name: String, value: String, user: User) = {
DB.withConnection { implicit c =>
try {
var sqlstring = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}"
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
for (p <- params){
sqlstring += ", "
if (name.endsWith(p))
sqlstring += value
else
sqlstring += "null"
}
SQL(sqlstring)
.on(
"login" -> user.login,
"domain" -> user.domain,
).execute()
} catch {
case e: Throwable => Logger.error("update CensusPaper04 error", e)
}
}
}
But actually that doesn't work in all cases. For example, when I try to save string, it give's me an error like:
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'some phrase'
What is the best way to generate sql query using anorm with all nulls except one?
The reason this is happening is because when you write the string value directly into the SQL statement, it needs to be quoted. One way to solve this would be to determine which of the fields are strings and add conditional logic to determine whether to quote the value. This is probably not the best way to go about it. As a general rule, you should be using named parameters rather than building a string to with the parameter values. This has a few of benefits:
It will probably be easier for you to diagnose issues because you will get more sensible error messages back at runtime.
It protects against the possibility of SQL injection.
You get the usual performance benefit of reusing the prepared statement although this might not amount to much in the case of stored procedure invocation.
What this means is that you should treat your list of fields as named parameters as you do with user and domain. This can be accomplished with some minor changes to your code above. First, you can build your SQL statement as follows:
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
val sqlString = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}," +
params.map("{" + _ + "}").mkString{","}
What is happening above is that you don't need to insert the values directly, so you can just build the string by adding the list of parameters to the end of your query string.
Then you can go ahead and start building your parameter list. Note, the parameters to the on method of SQL is a vararg list of NamedParameter. Basically, we need to create Seq of NamedParameters that covers "login", "domain" and the list of fields you are populating. Something like the following should work:
val userDomainParams: Seq[NamedParameter] = (("login",user.login),("domain",user.domain))
val additionalParams = params.map(p =>
if (name.endsWith(p))
NamedParameter(p, value)
else
NamedParameter(p, None)
).toSeq
val fullParams = userDomainParams ++ additionalParams
// At this point you can execute as follows
SQL(sqlString).on(fullParams:_*).execute()
What is happening here is that you building the list of parameters and then using the splat operator :_* to expand the sequence into the varargs needed as arguments for the on method. Note that the None used in the NamedParameter above is converted into a jdbc NULL by Anorm.
This takes care of the issue related to strings because you are no longer writing the string directly into the query and it has the added benefit eliminating other issues related with writing the SQL string rather than using parameters.

Advice on architecture for storing some trivial values in Django database?

I'd like to store some trivial values for each user in the database, like if the user can see the new comers' banner, the instruction on how to use each of the features etc. The number of values can increase as we come across new ideas.
So I've thought about two solutions for storing these data. Either having a field for each of these values (So the structure of the table will change a few times at least), or have one field for all these types of data, so they're stored as a dictionary in the field (In this case, I'm worried about if it's hurting db performance, I also need to write more logics for parsing the dictionary in string and the way around, and if storing dictionaries in db contradicts with what db does).
models.py
class Instruction(models.Model):
user=models.ForeignKey('auth.User')
can_see_feature_foo_instruction=models.BooleanField()
can_see_feature_bar_instruction=models.BooleanField()
...
or
class Instruction(models.Model):
user=models.ForeignKey('auth.User')
instruction_prefs=models.CharField() #Value will be "{'can_see_foo_inst':True, 'can_see_bar_inst':False, ...}"
Which will be the best solution?
It depends if you need to be able to search on these fields. If so, the text field option is not really suitable, as the individual flags won't be indexed. But if not, then this is a perfectly good way of going about it. You might want to consider storing it as JSON, which is useful as a method of serializing dicts objects to text and getting them back. There are plenty of implementations around of "JSONField" in Django that will take of serializing/deserializing the JSON for you.
Django has a built-in permission system. Try reading this link https://docs.djangoproject.com/en/dev/topics/auth/#permissions
Update
I think if you really want to use an Instruction model. You can use something like a JSONField and use it to store instructions. This way you can do something like instruction.key to access a value. You can try using this. https://github.com/derek-schaefer/django-json-field
You can create model for key value pair of instructions/permissions per user.
E.g.
class Instruction(models.Model):
user=models.ForeignKey('auth.User')
key = models.CharField(max_length=20)
value = models.BooleanField()
Then you can create multiple instances of this for each user depending upon permissions he has.
>>>> instr1 = Instruction()
>>>> instr1.user = user1
>>>> instr1.key = 'can_see_feature_foo'
>>>> instr1.value = True
>>>> instr1.save()
>>>> instr2 = Instruction()
>>>> instr2.user = user1
>>>> instr2.key = 'can_see_feature_bar'
>>>> instr2.value = True
>>>> instr2.save()
....
#To query
>>>> Instructions.filter(user=user1, key='can_see_feature_bar')
If you use a Model with a CharField to store the instruction and a ManyToManyField to the users you can create and assign any number of instructions to any number of users.
class Instruction(models.Model):
user = models.ManyToManyField('auth.User')
instruction = models.CharField() # Value will be a single instruction

Search entries in Go GAE datastore using partial string as a filter

I have a set of entries in the datastore and I would like to search/retrieve them as user types query. If I have full string it's easy:
q := datastore.NewQuery("Products").Filter("Name =", name).Limit(20)
but I have no idea how to do it with partial string, please help.
q := datastore.NewQuery("Products").Filter("Name >", name).Limit(20)
There is no like operation on app engine but instead you can use '<' and '>'
example:
'moguz' > 'moguzalp'
EDIT: GAH! I just realized that your question is Go-specific. My code below is for Python. Apologies. I'm also familiar with the Go runtime, and I can work on translating to Python to Go later on. However, if the principles described are enough to get you moving in the right direction, let me know and I wont' bother.
Such an operation is not directly supported on the AppEngine datastore, so you'll have to roll your own functionality to meet this need. Here's a quick, off-the-top-of-my-head possible solution:
class StringIndex(db.Model):
matches = db.StringListProperty()
#classmathod
def GetMatchesFor(cls, query):
found_index = cls.get_by_key_name(query[:3])
if found_index is not None:
if query in found_index.matches:
# Since we only query on the first the characters,
# we have to roll through the result set to find all
# of the strings that matach query. We keep the
# list sorted, so this is not hard.
all_matches = []
looking_at = found_index.matches.index(query)
matches_len = len(foundIndex.matches)
while start_at < matches_len and found_index.matches[looking_at].startswith(query):
all_matches.append(found_index.matches[looking_at])
looking_at += 1
return all_matches
return None
#classmethod
def AddMatch(cls, match) {
# We index off of the first 3 characters only
index_key = match[:3]
index = cls.get_or_insert(index_key, list(match))
if match not in index.matches:
# The index entity was not newly created, so
# we will have to add the match and save the entity.
index.matches.append(match).sort()
index.put()
To use this model, you would need to call the AddMatch method every time that you add an entity that would potentially be searched on. In your example, you have a Product model and users will be searching on it's Name. In your Product class, you might have a method AddNewProduct that creates a new entity and puts it into the datastore. You would add to that method StringIndex.AddMatch(new_product_name).
Then, in your request handler that gets called from your AJAXy search box, you would use StringIndex.GetMatchesFor(name) to see all of the stored products that begin with the string in name, and you return those values as JSON or whatever.
What's happening inside the code is that the first three characters of the name are used for the key_name of an entity that contains a list of strings, all of the stored names that begin with those three characters. Using three (as opposed to some other number) is absolutely arbitrary. The correct number for your system is dependent on the amount of data that you are indexing. There is a limit to the number of strings that can be stored in a StringListProperty, but you also want to balance the number of StringIndex entities that are in your datastore. A little bit of math with give you a reasonable number of characters to work with.
If the number of keywords is limited you could consider adding an indexed list property of partial search strings.
Note that you are limited to 5000 indexes per entity, and 1MB for the total entity size.
But you could also wait for Cloud SQL and Full Text Search API to be avaiable for the Go runtime.

How to ignore errors in datastore.Query.GetAll()?

I just started developing a GAE app with the Go runtime, so far it's been a pleasure. However, I have encountered the following setback:
I am taking advantage of the flexibility that the datastore provides by having several different structs with different properties being saved with the same entity name ("Item"). The Go language datastore reference states that "the actual types passed do not have to match between Get and Put calls or even across different App Engine requests", since entities are actually just a series of properties, and can therefore be stored in an appropriate container type that can support them.
I need to query all of the entities stored under the entity name "Item" and encode them as JSON all at once. Using that entity property flexibility to my advantage, it is possible to store queried entities into an arbitrary datastore.PropertyList, however, the Get and GetAll functions return ErrFieldMismatch as an error when a property of the queried entities cannot be properly represented (that is to say, incompatible types, or simply a missing value). All of these structs I'm saving are user generated and most values are optional, therefore saving empty values into the datastore. There are no problems while saving these structs with empty values (datastore flexibility again), but there are when retrieving them.
It is also stated in the datastore Go documentation, that it is up to the caller of the Get methods to decide if the errors returned due to empty values are ignorable, recoverable, or fatal. I would like to know how to properly do this, since just ignoring the errors won't suffice, as the destination structs (datastore.PropertyList) of my queries are not filled at all when a query results in this error.
Thank you in advance, and sorry for the lengthy question.
Update: Here is some code
query := datastore.NewQuery("Item") // here I use some Filter calls, as well as a Limit call and an Order call
items := make([]datastore.PropertyList, 0)
_, err := query.GetAll(context, &items) // context has been obviously defined before
if err != nil {
// something to handle the error, which in my case, it's printing it and setting the server status as 500
}
Update 2: Here is some output
If I use make([]datastore.PropertyList, 0), I get this:
datastore: invalid entity type
And if I use make(datastore.PropertyList, 0), I get this:
datastore: cannot load field "Foo" into a "datastore.Property": no such struct field
And in both cases (the first one I assume can be discarded) in items I get this:
[]
According to the following post the go datastore module doesn't support PropertyList yet.
Use a pointer to a slice of datastore.Map instead.

Resources