Duplicate tables created in django migrations when using SQL Server schemas - sql-server

I want to place all Django specific tables and my custom Auth models into the default dbo schema, and have all my different app specific tables in a schema named after the app. Something to note is that all of my app tables will foreign key back to my auth model (I have _created_by and _last_updated_by fields on a base model that all apps inherit from). Basically I want the DB structure to be something like this:
DBO
- my_custom_auth_table
- django_migrations
- django_session
- django_content_type
- etc...
APP1
-table1
-table2
APP2
-table1
-table2
In order to achieve this, I tried creating a Login/User pair on the DB server for each app and implemented a DB router.
my allow_migrate method:
def allow_migrate(self, db, app_label, model_name=None, **hints):
if app_label == db:
return True
else:
return False
my database settings (I will use my doglicense app as an example):
IP = xxx
default_db_settings = {
'ENGINE': 'mssql',
'NAME': 'DB',
'USER': 'some_user',
'PASSWORD':'***',
'HOST': IP,
'PORT':'1433',
'OPTIONS':{'driver':'ODBC Driver 18 for SQL Server', 'extra_params': 'trustServerCertificate=yes'},
}
doglicense = {
'ENGINE': 'mssql',
'NAME': 'DB',
'USER': 'DogLicense',
'PASSWORD':'***',
'HOST': IP,
'PORT':'1433',
'OPTIONS':{'driver':'ODBC Driver 18 for SQL Server', 'extra_params': 'trustServerCertificate=yes'},
}
I have successfully migrated the custom auth app and all of djangos apps into dbo, however this is where the fun begins.
If I run:
python manage.py migrate DogLicense --plan
we can see that it only tries to create the new tables:
Planned operations:
DogLicense.0001_initial
Create model Breed
Create model Color
Create model Dog
Create model ZipCode
Create model Veterinarian
Create model Street
Create model Registration
Create model Owner
Create model DogType
Add field owners to dog
Add field type to dog
However when I try to specify the database connection in order to dump theses files into the doglicense schema:
python manage.py migrate DogLicense --plan --database=doglicense
I get:
Planned operations:
contenttypes.0001_initial
Create model ContentType
Alter unique_together for contenttype (1 constraint(s))
contenttypes.0002_remove_content_type_name
Change Meta options on contenttype
Alter field name on contenttype
Raw Python operation
Remove field name from contenttype
auth.0001_initial
Create model Permission
Create model Group
Create model User
auth.0002_alter_permission_name_max_length
Alter field name on permission
auth.0003_alter_user_email_max_length
Alter field email on user
auth.0004_alter_user_username_opts
Alter field username on user
auth.0005_alter_user_last_login_null
Alter field last_login on user
auth.0006_require_contenttypes_0002
auth.0007_alter_validators_add_error_messages
Alter field username on user
auth.0008_alter_user_username_max_length
Alter field username on user
auth.0009_alter_user_last_name_max_length
Alter field last_name on user
auth.0010_alter_group_name_max_length
Alter field name on group
auth.0011_update_proxy_permissions
Raw Python operation -> Update the content_type of prox…
auth.0012_alter_user_first_name_max_length
Alter field first_name on user
MSSAuth.0001_initial
Create model FailedLoginAttempt
Create model MSSUser
Create model AuthProfile
DogLicense.0001_initial
Create model Breed
Create model Color
Create model Dog
Create model ZipCode
Create model Veterinarian
Create model Street
Create model Registration
Create model Owner
Create model DogType
Add field owners to dog
Add field type to dog
You can see it wants to create every table all over again. And obviously running this migration without the --plan flag does indeed result in tables like doglicense.django_migrations being created.
How can I prevent these duplicate tables from being created? Is this a problem with my SQL Server user permissions? Perhaps my router is poorly implemented?
Any help will be appreciated.

Related

why my new table cannot be added to my database

I decide to add new Model called Project to my project:
When I run python manage.py migrate, it shows me the below error:
class Project(models.Model):
statut_juridique=[
('per', 'personne physique' ),
('sarl', 'SARL'),
('sual', 'SUARL'),
('anony', 'SA'),
]
type_du_projet = [
('ind', 'industrie'),
('agr', 'agronome'),
('ser', 'service'),
('art', 'artisanat'),
('com', 'commerce'),
]
name = models.CharField(max_length=50)
produit = ArrayField(
ArrayField(
models.CharField(max_length=20, blank=True),
size=8,
),
size=8,
)
stat_jur = models.CharField(max_length=50, choices=statut_juridique)
type_projet = models.CharField(max_length=50, choices=type_du_projet)
Nomination = models.CharField(max_length=50)
adresse = models.CharField(max_length=200)
user = models.ForeignKey(User, related_name='projet', on_delete=models.CASCADE)
def __str__(self):
return self.name
Operations to perform: Apply all migrations: admin, auth,
businesplan, contenttypes, sessions Running migrations: Applying
contenttypes.0001_initial...Traceback (most recent call last): File
"/home/abdallah/projectdjango/oasis/venv/lib/python3.8/site-packages/django/db/backends/utils.py",
line 87, in _execute
return self.cursor.execute(sql) psycopg2.errors.DuplicateTable: relation "django_content_type" already exists
And also I can't see the new table in my Database, Can you help me please!
You probably use a database that already has some tables with their migrations.
For this case you can Try using a new database, or reset your existing database to remove duplicate tables or sometimes you can troubleshot using --fake-initial as one of django-admin cmd option:
$ python manage.py migrate --fake-initial
From Django-Doc:
Allows Django to skip an app’s initial migration if all database tables with the names of all models created by all CreateModel operations in that migration already exist. This option is intended for use when first running migrations against a database that preexisted the use of migrations. This option does not, however, check for matching database schema beyond matching table names and so is only safe to use if you are confident that your existing schema matches what is recorded in your initial migration.
Source: django-admin#cmdoption-migrate-fake-initial [django-doc]

Django migrate column is not the same data type as referencing column

I have the below model which is an existing DB model and through Django's inspectdb management command the below model is created.
class ExistingLegacyModel(models.Model):
period = models.TextField(db_column="Period", blank=True, null=True)
key = models.AutoField(db_column="OutlookKey", primary_key=True)
class Meta:
managed = False
db_table = "table_name"
and currently, I'm trying to create a model with a field foreign key reference to the existing legacy DB model
class TestModel(models.Model):
period = models.ForeignKey(
ExistingLegacyModel,
on_delete=models.CASCADE,
db_column="OutlookKey",
)
so when I run the makemigrations command the migration file is successfully getting created with no issue. below is the migration file content.
class Migration(migrations.Migration):
initial = True
dependencies = [
('historical', '0011_phoenixcontractprice'),
]
operations = [
migrations.CreateModel(
name='TestModel',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('period', models.ForeignKey(db_column='OutlookKey', on_delete=django.db.models.deletion.CASCADE, to='app.ExistingLegacyModel')),
],
),
]
so now when i run the migrate command now, it is failing and giving the below error.
django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Column 'table_name.OutlookKey' is not the same data type as referencing column 'version_testmodel.OutlookKey' in foreign key 'version_testmodel_OutlookKey_eb16c31c_fk_table_name_OutlookKey'. (1778) (SQLExecDirectW); [42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Could not create constraint or index. See previous errors. (1750)")
I'm stuck with this issue for the past couple of days and I searched all over the internet but didn't get any resolution. I found a couple of StackOverflow questions that are very similar to my issue, but those questions are also unanswered.
Django - Migration foreign key field type not matching current type
Django 3.2 update AutoField to BigAutoField backward compatibility with foreign key relations
I'm currently using Django 3.2.13 and mssql-django to connect to the MSSQL database.
Any help on this will be highly appreciated! Thank you in advance.
UPDATE 1
I ran the sqlmigrate command for the initial migration. So for the period column, it is creating the table with a foreign key field with big int [OutlookKey] bigint NOT NULL whereas the existing legacy model has a normal integer field.
ALTER TABLE [<app>_<model>] ADD CONSTRAINT [<app>_<model>_OutlookKey_3505d410_fk_<existing_legacy_table>_OutlookKey] FOREIGN KEY ([OutlookKey]) REFERENCES [<existing_legacy_table>] ([OutlookKey]);

How to make the 'public' schema default in a Scala Play project that uses PostgreSQL?

I am not sure if my issue connecting to the Scala Play 2.5.x Framework or to PostgreSQL so I am going to describe my setup.
I am using the Play 2.5.6 with Scala and PostgreSQL 9.5.4-2 from the BigSQL Sandboxes. I use the Play Framework default evolution package to manage the DB versions.
I created a new database in BigSQL Sandbox's PGSQL and PGSQL created a default schema called public. I use this schema for development.
I would like to create a table with the following script (1.sql in DB evolution config):
# Initialize the database
# --- !Ups
CREATE TABLE user (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT NOT NULL,
creation_date TIMESTAMP NOT NULL
);
# --- !Downs
DROP TABLE user;
Besides that I would like to read the table with a code like this:
val resultSet = statement.executeQuery("SELECT id, name, email FROM public.user WHERE id=" + id.toString)
I got an error if I would like to execute any of the mentioned code or even if I use the CREATE TABLE... code in pgadmin. The issue is with the user table name. If I prefix it with public (i.e. public.user) everything works fine.
My questions are:
Is it normal to prefix the table name with the schema name every time? It seems to odd to me.
How can I make the public schema a default option so I do not have to qualify the table name? (e.g. CREATE TABLE user (...); will not throw an error)
I tried the following:
I set the search_path for my user: ALTER USER my_user SET search_path to public;
I set the search_path for my database: ALTER database "my_database" SET search_path TO my_schema;
search_path correctly shows this: "$user",public
I got the following errors:
In Play: p.a.d.e.DefaultEvolutionsApi - ERROR: syntax error at or near "user"
In pgadmin:
ERROR: syntax error at or near "user"
LINE 1: CREATE TABLE user (
********** Error **********
ERROR: syntax error at or near "user"
SQL state: 42601
Character: 14
This has nothing to do with the default schema. user is a reserved word.
You need to use double quotes to be able to create such a table:
CREATE TABLE "user" (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT NOT NULL,
creation_date TIMESTAMP NOT NULL
);
But I strongly recommend not doing that. Find a different name that does not require a quoted identifier.

Entity Framework 6 Code First From Database context performs CREATE TABLE on existing an VIEW

We have a production Oracle database server maintained by our ERP partner.
For some custom development I need to connect to this Oracle database using Entity Framework 6. I have a user that can SELECT any table on the ERP schema and I create views in the schema/user used in my EF context.
The view itself is pretty straightforward, a few joins but all referencing tables on another schema ofcourse.
i.e.:
CREATE TABLE ERP.M_GROUP
(
FILE VARCHAR2(3 BYTE)
, MATFAM VARCHAR2(1 BYTE) NOT NULL
, GROUP VARCHAR2(20 BYTE) NOT NULL
, OMS1 VARCHAR2(60 BYTE)
, OMS2 VARCHAR2(60 BYTE)
, RESTW_FACTOR1_I NUMBER
)
CREATE VIEW EF6CTX.GROUPS AS
SELECT
GROUP Id,
MAX(OMS1) Name
FROM
M_GROUP
WHERE
FILE = 'BAT'
AND MATFAM IN ('B','C','I', 'K')
GROUP BY GROEP
When I connect to my database using Visual Studio's Entity Framework 6 Code First from Database identifing as user EF6CTX I can select this view and my model is created as it should.
But when I try to read these groups..
var ctx = new TestContext();
ctx.Database.Log = Console.WriteLine;
foreach (var group in ctx.GROUPS)
{
Console.WriteLine("Group: {0}", group.NAME);
}
I get this result:
Opened connection at 21/11/2014 15:29:05 +01:00
Started transaction at 21/11/2014 15:29:05 +01:00
create table "EF6CTX"."GROUPS"
(
"ID" varchar2(20 CHAR) not null,
"NAME" varchar2(60 CHAR) null,
constraint "PK_GROUPS" primary key ("ID")
)
-- Executing at 21/11/2014 15:29:05 +01:00
-- Failed in 217 ms with error: ORA-01031: insufficient privileges
The user EF6CTX has no permissions to create a table.. ofcourse. But why is it trying to create a table? It should USE the existing view!
Fixed when migrations are disabled:
System.Data.Entity.Database.SetInitializer<TestContext>(null);

Different Permissions in Apache Shiro for every User?

I built a database with the entity user and permission
user (id, email, password, permission)
permission (id, create_user, delete_user, user_fk)
create_user and delete_user is BOOLEAN.
Relationship: One-One
Now every user can have it's own permissions.
My question is: How can I use shiro to read the permissions from the database?
If you really only wish to assign permissions on user level, you can "fake" the roles table to make Shiro happy.
As Wouter mentioned, use the JdbcRealm and specify the 3 queries for your table setup.
You should modify your permission table to have this structure:
permission (id, permissionname, user_fk)
Then you insert rows for the create_user/delete_user rights as needed.
This way it's very simple to add another permission (reset_password for example) to your setup, without the need to modify the db schema.
In the shiro.ini (or how you call the your shiro config file):
jdbcRealm = org.apache.shiro.realm.jdbc.JdbcRealm
For the queries use then this:
jdbcRealm.authenticationQuery = select password from user where email=?
jdbcRealm.userRolesQuery = select id from user where email=?
jdbcRealm.authenticationQuery = select permissionname from permission where user_fk=?
The small trick in your setup is: you don't have roles at all, so we just return the id of the user as the role name.
When the lookup in the permission table is done, it then uses the role name (=user pk) and returns the associated permissions.
You should configure a JdbcReam in your .ini file:
jdbcRealm=org.apache.shiro.realm.jdbc.JdbcRealm
If you change your schema to adhere to the shiro queries, no extra config is needed. You need tables users, user_roles and roles_permissions.
See the source code how the exact column names should be:
https://svn.apache.org/repos/asf/shiro/trunk/core/src/main/java/org/apache/shiro/realm/jdbc/JdbcRealm.java
Alternatively you can configure your own queries to match your schema in the .ini file like so:
jdbcRealm.authenticationQuery=<your password select statement>
jdbcRealm.userRolesQuery=<your role names for username select statement>
jdbcRealm.authenticationQuery=<your permissions for role name select statement>

Resources