Change sa.Column nullable property with Alembic in Batch mode - database

I have a database upgrade migration I want to apply to Database column:
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('details') as batch_op:
batch_op.alter_column('details', 'non_essential_cookies',
existing_type=sa.BOOLEAN(),
nullable=False)
# ### end Alembic commands ###
I am implementing batch mode sice ALTER is unsupported and previously I received this error : sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "ALTER":. However, I hoped batch mode would work but now instead I receive the new error:
TypeError: <flask_script.commands.Command object at 0x1149bb278>: alter_column() got multiple values for argument 'nullable'.
I only have one tuple in the table and the relevant attribute is not NULL so the database migration is valid. I just don't understand why there are multiple values

From docs:
The method is used as a context manager, which returns an instance of
BatchOperations; this object is the same as Operations except that
table names and schema names are omitted.
Key point here being that you don’t have to provide the table name when calling ops on the BatchOperations instance.
The signature for alter_column is:
alter_column(table_name, column_name, nullable=None, server_default=False, new_column_name=None, type_=None, existing_type=None, existing_server_default=False, existing_nullable=None, schema=None, **kw)
So from your code:
with op.batch_alter_table('details') as batch_op:
batch_op.alter_column('details', 'non_essential_cookies',
existing_type=sa.BOOLEAN(),
nullable=False)
'details' is being passed to column_name, and 'non_essential_cookies' is getting passed to nullable as a positional argument. The issue is caused later when you specify the value of nullable again with the keyword arg,nullable=False.

Related

How to make the 'public' schema default in a Scala Play project that uses PostgreSQL?

I am not sure if my issue connecting to the Scala Play 2.5.x Framework or to PostgreSQL so I am going to describe my setup.
I am using the Play 2.5.6 with Scala and PostgreSQL 9.5.4-2 from the BigSQL Sandboxes. I use the Play Framework default evolution package to manage the DB versions.
I created a new database in BigSQL Sandbox's PGSQL and PGSQL created a default schema called public. I use this schema for development.
I would like to create a table with the following script (1.sql in DB evolution config):
# Initialize the database
# --- !Ups
CREATE TABLE user (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT NOT NULL,
creation_date TIMESTAMP NOT NULL
);
# --- !Downs
DROP TABLE user;
Besides that I would like to read the table with a code like this:
val resultSet = statement.executeQuery("SELECT id, name, email FROM public.user WHERE id=" + id.toString)
I got an error if I would like to execute any of the mentioned code or even if I use the CREATE TABLE... code in pgadmin. The issue is with the user table name. If I prefix it with public (i.e. public.user) everything works fine.
My questions are:
Is it normal to prefix the table name with the schema name every time? It seems to odd to me.
How can I make the public schema a default option so I do not have to qualify the table name? (e.g. CREATE TABLE user (...); will not throw an error)
I tried the following:
I set the search_path for my user: ALTER USER my_user SET search_path to public;
I set the search_path for my database: ALTER database "my_database" SET search_path TO my_schema;
search_path correctly shows this: "$user",public
I got the following errors:
In Play: p.a.d.e.DefaultEvolutionsApi - ERROR: syntax error at or near "user"
In pgadmin:
ERROR: syntax error at or near "user"
LINE 1: CREATE TABLE user (
********** Error **********
ERROR: syntax error at or near "user"
SQL state: 42601
Character: 14
This has nothing to do with the default schema. user is a reserved word.
You need to use double quotes to be able to create such a table:
CREATE TABLE "user" (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT NOT NULL,
creation_date TIMESTAMP NOT NULL
);
But I strongly recommend not doing that. Find a different name that does not require a quoted identifier.

Weird behavior with web2py DAL on table definition

Hi getting behaviour I don't understand with web2py
In [50]: db = DAL('sqlite://deposit/sample.sqlite')
In [51]: db.define_table('customer',Field('name','string',required=True),
Field('nric','string',required=True),
Field('address','string'),
Field('phone','integer'),
primarykey=['name'])
Out[51]: <Table customer (name,nric,address,phone)>
works as expected.
I then do
In [53]: db.define_table('check',
Field('nric', db.customer.nric, required=True),
Field('clear','string'))
which gets me the message
AttributeError: 'DAL' object has no attribute 'customer.nric'
So thinking this may be an issue of not having committed customer to the database
so I do a db.commit() and then try again
In [56]: db.define_table('check',Field('nric', db.customer.nric, required=True), Field('clear','string'))
File "<string>", line unknown
SyntaxError: table already defined: check
Not sure why .. but anyway I try and drop the table
In [59]: db['check'].drop()
and get the following weird traceback
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-59-998297b798f5> in <module>()
----> 1 db['check'].drop()
/home/tahnoon/.dropbox-cyn/Dropbox (TIS Group)/Cynopsis/Builds/Apollo/Code Src/web2py/gluon/dal.pyc in drop(self, mode)
9225
9226 def drop(self, mode=''):
-> 9227 return self._db._adapter.drop(self, mode)
9228
9229 def _listify(self, fields, update=False):
/home/tahnoon/.dropbox-cyn/Dropbox (TIS Group)/Cynopsis/Builds/Apollo/Code Src/web2py/gluon/dal.pyc in drop(self, table, mode)
1328 queries = self._drop(table, mode)
1329 for query in queries:
-> 1330 if table._dbt:
1331 self.log(query + '\n', table)
1332 self.execute(query)
/home/tahnoon/.dropbox-cyn/Dropbox (TIS Group)/Cynopsis/Builds/Apollo/Code Src/web2py/gluon/dal.pyc in __getitem__(self, key)
9108 return self._db(self._id == key).select(limitby=(0, 1), orderby_on_limitby=False).first()
9109 elif key:
-> 9110 return ogetattr(self, str(key))
9111
9112 def __call__(self, key=DEFAULT, **kwargs):
AttributeError: 'Table' object has no attribute '_dbt'
Checking tables shows
In [61]: db.tables()
Out[61]: ['customer']
Is this expected behaviour? If so how do I drop/create a table after a syntax error? thanks
Since db.customer is a keyed table (i.e., you have defined a primarykey attribute rather than relying on the default autoincrement integer ID field as the primary key), it can only be referenced by other keyed tables.
Also, when creating reference fields for keyed tables, use the following syntax:
Field('nric', 'reference customer.nric', required=True)
However, I don't think keyed tables are supported for SQLite (the docs say only DB2, MS-SQL, Ingres, and Informix are supported). Anyway, if you are creating a new table in SQLite, there is no reason to use a keyed table (that functionality was added primarily to enable access to legacy databases that lack autoincrement integer primary key fields).
Finally, dropping a table does not remove the model from the db DAL instance -- rather, that operation drops the table from the database itself. If you want to redefine a model within a shell session, you should use the "redefine" argument:
db.define_table(..., redefine=True)

Hive Serde errors with Array<Struct<>> org.json.JSONArray cannot be cast to [Ljava.lang.Object;

I have created a table :
add jar /../xlibs/hive-json-serde-0.2.jar;
CREATE EXTERNAL TABLE SerdeTest
(Unique_ID STRING
,MemberID STRING
,Data ARRAY>
)
PARTITIONED BY (Pyear INT, Pmonth INT)
ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde";
ALTER TABLE SerdeTest ADD
PARTITION (Pyear = 2014, Pmonth =03) LOCATION '../Test2';
The data in the file :
{"Unique_ID":"ABC6800650654751","MemberID":"KHH966375835","Data":[{"SerialNo":1,"VariableName":"Var1","VariableValue":"A_49"},{"SerialNo":2,"VariableName":"Var2","VariableValue":"B_89"},{""SerialNo":3,"VariableName":"Var3","VariableValue":"A_99"}]}
Select query that I am using:
select Data[0].SerialNo from SerdeTest where Unique_ID = 'ABC6800650654751';
however, when I run this query I get the following error:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.ClassCastException: org.json.JSONArray cannot be cast to [Ljava.lang.Object;
at org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector.getList(StandardListObjectInspector.java:98)
at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:330)
at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:386)
at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:237)
at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:223)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
]
Can anyone please suggest me what am I doing wrong
Few suggestions:
Make sure that all the packages of hive and hive-json-serde-0.2.jar have execute permission for hadoop user.
Hive creates a file called derby.log and metastore_db in the hive directory. It should be allowed to the user invoking the hive query to create files and directories.
Location for data should have / at the end. e.g. LOCATION '../Test2/';
In short, the working JAR is json-serde-1.3-jar-with-dependencies.jar which can be found here. This one is working with 'STRUCT' and can even ignore some malformed JSON. During the creation of the table, include the following code:
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("ignore.malformed.json" = "true")
LOCATION ...
If needed, it is possible to recompile it from here or here. I tried the first repository and it is compiling fine for me, after adding the necessary libs. The repository has also been updated recently.
Check for more details here.

'do_replace()' not working?

while trying ATK4 I've found a problem:
$this->api->db->dsql()->table('person')->set('id', 1)->set('name', 'Test user')->do_replace();
This is not working. Then I looked a little bit deeper in ATK4 source and found in /opt/ipism/www/atk4/lib/DB/dsql.php the lines
public $sql_templates=array(
'select'=>"select [options] [field] [from] [table] [join] [where] [group] [having] [order] [limit]",
'insert'=>"insert [options_insert] into [table_noalias] ([set_fields]) values ([set_values])",
'replace'=>"replace [options_replace] into [table_noalias] ([set_fields]) values ([set_values])",
'update'=>"update [table_noalias] set [set] [where]",
'delete'=>"delete from [table_noalias] [where]",
'truncate'=>'truncate table [table_noalias]',
'describe'=>'desc [table_noalias]',
);
After changing the 'replace'-line into
'replace'=>"replace into [table_noalias] ([set_fields]) values ([set_values])",
it worked for me (removing the options_replace and appending a 's' to set_value). I'm using latest version from git with a MySQL database connection.
But I'm not sure, if I'm using 'do-replace()' in the wrong way?
ByE...
By the way: Is there a way to send fixes, without creating an account on GitHub or somewhere?
Edit: Here is the output if the options_replace isn't removed from the template:
replace [options_replace] into `person` (`id`,`name`) values ("1","John Doe") [:a_2, :a]Application Error: Database Query Failed
Exception_DB, code: 0Additional information: pdo_error: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[options_replace] into `person` (`id`,`name`) values ('1' at line 1 mode: replace params: :a: 1 :a_2: John Doe query: replace [options_replace] into `person` (`id`,`name`) values (:a,:a_2) template: replace [options_replace] into [table_noalias] ([set_fields]) values ([set_values])/opt/ipism/www/atk4/lib/DB/dsql.php:1519
Stack trace:
File Object NameStack Trace/opt/ipism/www/atk4/lib/BaseException.php:63 Exception_DBException_DB->collectBasicData(Null)
/opt/ipism/www/atk4/lib/AbstractObject.php:545 Exception_DBException_DB->__construct("Database Query Failed", Null)
/opt/ipism/www/atk4/lib/DB/dsql.php:1519 sample_project_db_db_dsql_mysqlDB_dsql_mysql->exception("Database Query Failed")
/opt/ipism/www/atk4/lib/DB/dsql.php:1586 sample_project_db_db_dsql_mysqlDB_dsql_mysql->execute()
/opt/ipism/www/atk4/lib/DB/dsql.php:1624 sample_project_db_db_dsql_mysqlDB_dsql_mysql->replace()
/opt/ipism/www/page/test.php:40 sample_project_db_db_dsql_mysqlDB_dsql_mysql->do_replace()
/opt/ipism/www/atk4/lib/AbstractObject.php:306 sample_project_testpage_test->init()
/opt/ipism/www/atk4/lib/ApiFrontend.php:130 sample_projectFrontend->add("page_test", "test", "Content")
/opt/ipism/www/atk4/lib/ApiWeb.php:428 sample_projectFrontend->layout_Content()
/opt/ipism/www/atk4/lib/ApiFrontend.php:39 sample_projectFrontend->addLayout("Content")
/opt/ipism/www/atk4/lib/ApiWeb.php:275 sample_projectFrontend->initLayout()
/opt/ipism/www/index.php:15 sample_projectFrontend->main()
Note: To hide this information from your users, add $config['logger']['web_output']=false to your config.php file. Refer to documentation on 'Logger' for alternative logging options
Replace is similar to "insert" by it's nature, but instead of failing when primary key is duplicated, it replaces the value.
Please add ->debug() to your line before do_replace and give me the output, which would help me understand why that parameter needs removing.
set_value seems to be a typo, I have changed and committed it into master: https://github.com/atk4/atk4/commit/24b20865b9e3345a8e7504dfb68b7ef96335009e
the best way to submit changes is by creating a pull request. The best way to report issues is through "issues" in github currently.

Is it possible to change name of the system table

I want to change name of the system table in my database is it possible? Probably I shouldn't change it but I'm curious.
When I execute sp_rename I get the following error:
Msg 15001, Level 16, State 1, Procedure sp_rename, Line 404
Object 'cdc.[dbo_CdcTest_CT]' does not exist or is not a valid object for this operation.
Edit:
I want to change name of tables created by Change Data Capture because I want to disable CDC mechanism for table and still have data - I know that I can create additional table and move there data from CDC table but it's easier to change name of the CDC and then disable cdc for specified table.
No you cannot change the name of the system tables. However you can refer it with a different name.
You can use synonyms for that:
CREATE SYNONYM [ schema_name_1. ] synonym_name FOR <object>
<object> :: =
{
[ server_name.[ database_name ] . [ schema_name_2 ].| database_name . [ schema_name_2 ].| schema_name_2. ] object_name
}
Also to mention that sp_rename
Changes the name of a user-created object in the current database.
This object can be a table, index, column, alias data type, or
Microsoft .NET Framework common language runtime

Resources