I am trying to access a SQLite database (that basically reads info about phones connected to network) , retrieve info and store them into my domain variable's to be stored in the webservice (that view the only of connected phone) database. The thing is I retrieve all the info every 5 secs, and want to add to my database the new records only(new connected phones). what is the proper method to do that?
There is two approach that I could think of:
1.Have a variable that saves the last rows id and every time you want to retrieve data from your database, just read data with bigger id than what you have in your variable.(this could only work if your id is incremental)
2.Add another boolean column to your database and make it true when you read that row, and every time you read from database just query rows which that column in them are false.
I personally prefer the first approach.
Related
In my Access application's data entry form I'm using Me.Refresh to show the user the new value of a calculated field based on data they entered in a non-calculated field. The purpose is to give them the deadline for doing a job next.
This Access app is using linked SQL Server 2012 tables via ODBC. The calculated field is in the underlying SQL Server table, not part of the Access Record Source query because I want to store the calculated value in the actual data, not just as an interface element.
The calculation is very simple:
nextjob = jobdate + 79
So I have the field for jobdate set to run Me.Refresh after update. All well and good.
The problem is that if the user updates jobdate, triggers the refresh by moving to another field, then returns to the jobdate field and changes the date they entered Access throws a "The data has been changed by another user" error.
I tested the method using native Access tables and the problem does not occur. However the data needs to stay on the server, so moving to native tables is not a solution for me.
There are several possible solutions.
1- If it's always jobdate + 79, don't store it at all, use a view that has the calculated field.
2- Use Me.Requery instead of Me.Refresh. If the form shows multiple records, you must navigate back to the current record, you can use Me.Bookmark for that.
3- Move the calculation into the Access frontend - make nextjob an ordinary column and set it in the form, so it isn't another user (the server) that updates the data.
Do you know how to transfer only new records between two different databases (ie. Oracle and MSSQL) using SSIS? There is no problem transfering new data only between two tables in the same database and server, but is this possible to do such operation between completely different servers and databases?
Ps. I know about solution using Lookup but it is not very efficient if anybody needs to check and add a lot of records (50k and more) several times per day. I would like to operate with new data only.
You have several options:
Timestamp based solution
If you have a column which stores the insertation time in the source system, you can select only the new records created since the last load. With the same logic, you can transfer modified records too, just mark the records with the timestamp value when it change.
Sequence based solution
If there is a sequence in the source table, you can load the new records based on that sequence. Query the last value from the destination system, then load avarything which is larger than that value.
CDC based solution
If you have CDC (Change Data Capture) in your source system, you can track the changes and you can load them based on the CDC entries.
Full load
This is the most resource hungry solution: you have to copy all data from the source to the destination. If you do not have any column which marks the new records, you should use this solution.
You have several options to achieve this:
TRUNCATE the destination table and reload it from source
Use a Lookup component to determine which records are missing
Load all data from source to a temporary table and write a query which retrieves the new/changed records.
Summary
If you have at least one column, which marks the new/modified records, you can use it to implement a differential/incremental load with SSIS. If you do not have any clue, which columns/rows are changed, you have to load (or at least query) all of them.
There is no solution which enables a one-query (INSERT .. SELECT) solution using multiple servers without transferring all data. (Please note, that a multi-server query using Linked Servers are transfers the data from the source system).
What about variables? Is it possible to use the same variable between different databases and servers in SSIS?
I would like to transfer last id number from a destination table and transfer it to the source table (different server!).
I can set a variable in a database scope like this:
DECLARE #Last int
SET #Last = (SELECT TOP 1 Id FROM dbo.Table_1 ORDER BY Id DESC)
SELECT *
FROM dbo.Table_2
WHERE ID > #Last;
However it works between two tables in the same database (as a SQL command) only. I can create a variable for a entire SSIS package in Variables --> Add variable, but I don't know it is possible to use the variable in a similar way as above - to keep an information about last id in a destination table and pass it to another table on a source server as data limit.
I have a SQL Server database with table containing 300.000 data rows. There is an Index on the Primary Key and another key. I am using the following query in my standalone WCF server to fetch the data using an SQLConnection and SQLDataReader.
SELECT * FROM Users WHERE UserTypeId = #UserTypeId ORDER BY Users.Id OFFSET
#OFFSET ROWS FETCH NEXT #NUMBER ROWS ONLY
The Data returned by the DataReader is pushed into my own Class/Model and than returned by the function of the WCF server.
The WPF Client connects to the server and starts the command and only wants 500 data rows. However the time needed for this task is about 3-4 seconds. (Not mentioning the time for all data...)
The returned List is then used as the DataContext for the WPF Datagrid.
My question is, what can I check or what might be wrong. If you need more Information,CodeSamples,etc. please let me know!
First, don't use select *, instead specify what fields you want from the table. Now you are getting data that you don't need, for example the UserTypeId field which you already know for all the records that you get.
Then you can create a covering index that contains UserTypeId and Id, and has any other fields that you want to return from the query as included fields. That way the database can run the query against the index alone, and doesn't have to read anything from the actual table.
I'm currently creating my first real life project in Pervasive. The task is to map a certain XML structure containing orders (as in shops and products) to 3 tables I created myself. These tables rest inside a MS-SQL-Server instance.
All of the tables have a unique key called "id", an automatically incremented column. I've dropped this column from all mappings so that Pervasive will not try to fill it itself.
For certain calculations, for a split key in one of the tables and for references to the created records in other tables, I will need the id that the database has just created. For that, I have googled the answer. I can use "select ##identity;" as a statement, and this returns the id that has most recently been created for the current connection. This means that in Pervasive, I will have to execute this statement using the already existing target connection object.
But how to do that? I am quite sure that I will need a JDImport or DJExport object, but how to get one associated with the current connection that Pervasive inserts the records by?
Or is there any other way to handle this auto increment when I need to reference the id in other tables?
Not sure how things work in Pervasive, but you may run into issues with ##identity,. Scope_identity() would probably be safer but may still not work in Pervasive.
Hopefully your tables have a natural key in addition to the generated id, in which case you can select your id based on the natural key. This will avoid any issues you may have with disparate sessions and scope.
If there is anyone looking this post up and wonders about the answer, it's "You can't". Pervasive does not allow access to their very own connection object, the one they use to query the database. Without access to it, you cannot guaranteed fetch the right id. The solution for us was this: We used a stored procedure which we called in the Before-Transformation event that created the header record and returned the id and an optional error message as a table. We executed it and it returns the id we then save and use throughout our mapping.
Currently, I would like provide this as an option to the user when storing data to the database.
Save the data to a file and use a background thread to read data from the textfile to SQL server.
Flow of my program:
- A stream of data coming from a server constantly (100 per second).
- want to store the data in a textfile and use background thread to copy data from the textfile back to the SQL database constantly as another user option.
Has this been done before?
Cheers.
Your question is indeed a bit confusing.
I'm guessing you mean that:
100 rows per second come from a certain source or server (eg. log entries)
One option for the user is textfile caching: the rows are stored in a textfile and periodically an incremental copy of the contents of the textfile into (an) SQL Server table(s) is performed.
Another option for the user is direct insert: the data is stored directly in the database as it comes in, with no textfile in between.
Am I right?
If yes, then you should do something in the lines of:
Create a trigger on an INSERT action to the table
In that trigger, check which user is inserting. If the user has textfile caching disabled, then the insert can go on. Otherwise, the data is redirected to a textfile (or a caching table)
Create a stored procedure that checks the caching table or text file for new data, copies the new data into the real table, and deletes the cached data.
Create an SQL Server Agent job that runs above stored procedure every minute, hour, day...
Since the interface from T-SQL to textfiles is not very flexible, I would recommend using a caching table instead. Why a textfile?
And for that matter, why cache the data before inserting it into the table? Perhaps we can suggest a better solution, if you explain the context of your question.