I use tchdb in a server to store some data. And use another program to clean some old and useless data. But after executing this cleaner the file size didn't change and keep going bigger while the server still running. Did I do things wrong? Should I do something else in the server after executing the cleaner?
Open mode for tchdbopen is HDBOREADER | HDBOWRITER | HDBOCREAT | HDBONOLCK.
Related
I am relatively new to SQlite3. I am currently using DB Browser. I have two very large .db files that I want to combine such that a all the tables (all different) all reside together in one .db (ie- I don't want to just attach them).
I am trying to use a simple .dump code in order to achieve this, but keep getting errors. I first define the path/directory and then:
for db in OctoberData, OctoberData2; { sqlite3 ${db}.db .dump | sqlite3 OctoberDataAll.db }
and get
Result: near "for": syntax error
At line 1:
for
Im sure it is something simple, but what am I doing wrong?
Thank you for your help!
I'm using an sqlite3 database to record data every second. The interface to it is provided by Flask-SQLAlchemy.
This can work fine for a couple of months, but eventually (as the .db file approaches 8 GB), an error prevents any more data from being written to the database:
Failed to commit: (sqlite3.OperationalError) disk I/O error
The journal file does not seem to be the issue here - if I restart the application and use the pragma journal_mode=TRUNCATE, the journal file is created but the disk I/O error persists.
Here's the .dbinfo (obtained from sqlite3.exe):
database page size: 1024
write format: 1
read format: 1
reserved bytes: 0
file change counter: 5200490
database page count: 7927331
freelist page count: 0
schema cookie: 12
schema format: 4
default cache size: 0
autovacuum top root: 0
incremental vacuum: 0
text encoding: 1 (utf8)
user version: 0
application id: 0
software version: 3008011
number of tables: 6
number of indexes: 7
number of triggers: 0
number of views: 0
schema size: 5630
data version 2
However this worked:
I made a copy of the .db file (call app.db and copy.db).
I renamed app.db to orig.db
I renamed copy.db to app.db (so effectively, I swapped it so that the copy becomes the app).
When I started my application again, it was able to write to the app.db file once more! So I could write to a copy I made of the database.
The drive is an SSD (Samsung 850 EVO mSATA)> I wonder if that's got something to do with it? Does anyone have any ideas on how I can prevent it from happening again?
EDIT: I've used the sqlite3.exe CLI to execute an INSERT INTO command manually, and this actually completed successfully (and wrote to the disk). However, when I re-ran my Flask-SQLAlchemy interface to write to it, it still came up with the disk I/O error.
UPDATE:
A colleague pointed out that this might be related to another question: https://stackoverflow.com/a/49506243/3274353
I strongly suspect now that this is a filesystem issue - in my system, the database file is being updated constantly alongside some other files which are also being written to.
So to reduce the amount of fragmentation, I'm getting the database to pre-allocate some disk space now using the answer provided in aforementioned question: https://stackoverflow.com/a/49506243/3274353
Something like this:
CREATE TABLE t(x);
INSERT INTO t VALUES(zeroblob(500*1024*1024)); -- 500 MB
DROP TABLE t;
To know whether this needs to be done, I use a call to the freelist_count pragma:
PRAGMA schema.freelist_count;
Return the number of unused pages in the database file.
I have 2 files that do not have the same columns. I want to be able to concatenate them, but I can't seem to find a way to do this. I thought about using a Script Component, but that still wants you to declare output columns.
The first file has columns and data like this:
A|B|C|D|E|F|G
Adata1|Bdata1|Cdata1|Ddata1|Edata1|Fdata1|Gdata1
Adata2|Bdata2|Cdata2|Ddata2|Edata2|Fdata2|Gdata2
Adata3|Bdata3|Cdata3|Ddata3|Edata3|Fdata3|Gdata3
Adata4|Bdata4|Cdata4|Ddata4|Edata4|Fdata4|Gdata4
The second file has columns and data like this:
H|I|J|K|L|M|N
Hdata1|Idata1|Jdata1|Kdata1|Ldata1|Mdata1|Ndata1
Hdata2|Idata2|Jdata2|Kdata2|Ldata2|Mdata2|Ndata2
Hdata3|Idata3|Jdata3|Kdata3|Ldata3|Mdata3|Ndata3
Hdata4|Idata4|Jdata4|Kdata4|Ldata4|Mdata4|Ndata4
I need the overall output file to look like this:
A|B|C|D|E|F|G
Adata1|Bdata1|Cdata1|Ddata1|Edata1|Fdata1|Gdata1
Adata2|Bdata2|Cdata2|Ddata2|Edata2|Fdata2|Gdata2
Adata3|Bdata3|Cdata3|Ddata3|Edata3|Fdata3|Gdata3
Adata4|Bdata4|Cdata4|Ddata4|Edata4|Fdata4|Gdata4
H|I|J|K|L|M|N
Hdata1|Idata1|Jdata1|Kdata1|Ldata1|Mdata1|Ndata1
Hdata2|Idata2|Jdata2|Kdata2|Ldata2|Mdata2|Ndata2
Hdata3|Idata3|Jdata3|Kdata3|Ldata3|Mdata3|Ndata3
Hdata4|Idata4|Jdata4|Kdata4|Ldata4|Mdata4|Ndata4
I know it's out of the ordinary, but that's the requirements...
Any suggestions?
Go with a script task. If you have connection managers for your 3 files, you don't need any inputs to the task.
Access the connection strings with:
var fileAString = Dts.Connections["fileAConnectionManagerName"].ConnectionString;
Workflow can be, open fileC for write. Open fileA for read. Read data and write to fileC. Close fileA. Open fileB for read. Read data and write to fileC. Close files and return success.
So my question is how do I take a table that populates QC(quality control) SQL scripts and then have them run within SSIS, so that I can have flags set regarding each SQL Script.
There are 15 total scripts that check different things, so I will need to loop through them each.
--example--
Id | SQL_Statment | Bypass
1 |Select count() from "" where firstName is null; | 0
2 |Select count() from "" where lastName is null; | 1
Depending on if those scripts are true or false I will need to set flags to later place back in another table.
Is this even possible and how would I approach it.
#Liqudid715
Not sure if I understand your question. Also, for SO, what you should be doing is showing what you have already tried and why it didn't work.
There is an execute SQL task that you can use in control flow items to run the scripts. Set the result set to single row. Create a user variable, and set the user variable based on the result (Result set, with result name being 0). Then you can use the variable in another execute sql task to update another table.
I dont know what you mean by loop through, but you can connect all of the execute sql tasks, but if you are using them to set variable, I don't think you would need to do that.
I have a select query producing a big output and I want to execute it in sqldeveloper, and get all the results into a file.
Sql-developer does not allow a result bigger than 5000 lines, and I have 100 000 lines to fetch...
I know i could use SQL+, but let's assume I want to do this in sqldeveloper.
Instead of using Run Script (F5), use Run Statement (Ctrl+Enter). Run Statement fetches 50 records at a time and displays them as you scroll through the results...but you can save the entire output to a file by right-clicking over the results and selecting Export Data -> csv/html/etc.
I'm a newbie SQLDeveloper user, so if there is a better way please let me know.
This question is really old, but posting this so it might help someone with a similar issue.
You can store your query in a query.sql file and and run it as a script. Here is a sample query.sql:
spool "C:\path\query_result.txt";
select * from my_table;
spool off;
In oracle sql developer you can just run this script like this and you should be able to get the result in your query_result.txt file.
#"C:\Path\to\script.sql"
Yes you can increase the size of the Worksheet by change the setting Tool-->Preferences - >Database -> Worksheet -> Max rows to print in a script(depends on you).
Mike G answer will work if you only want the output of a single statement.
However, if you want the output of a whole sql script with several statements, SQL*Plus reports, and some other output formats, you can use the spool command the same way as it is used in SQL*Plus.