Any easy way to get mysql server to query from an iseries (as/400 db2)? I have the odbc installed so I can query and export the data manually to my desktop and then import it to mysql.
The problem is the as400 database is so huge the performance is poor. I need to run a query every 1/2 hour or so on mysql to pull the new updated information on the iseries database.
Basically how do you use odbc on the mysql server to query from the iseries odbc?
I haven't worked on an iSeries for over 10 years but - here is what I know/remember.
You create physical files and then logicals(sort sequences) over them.
To help make it as efficient as possible the FIRST logical that will be executed during a "reorg" should contain ALL the fields you will use in any subsequent select/sequence logicals. Then the following logicals will use the first logical to built themselves - it is now ONLY using an index instead of a physical file.
Second when you use open query it looks for a logical that is "pre-built". If it can't find one at least "near" what it needs it has to build one of its own every time.
My next point is the file you are reading and selecting from. When a record is added does it update physical/logicals immediately? On open? On close?
If you are looking for speed for your query then you don't want to be busy updating the records which have been added.
Note that if these are order entry type of records the update may be deliberately delayed to enhance the data entry process.
Hope this helps - An "updated" and "appropriate" keyed and sequenced logical will make a huge difference.
If you don't know the iSeries you need someone who does that can check that side. Cheers Ted
Data replication. One method is using a row update timestamp and using column to drive the replication.
alter table mylib.mytable add column
UPDATETS TIMESTAMP GENERATED ALWAYS FOR EACH ROW
ON UPDATE AS ROW CHANGE TIMESTAMP NOT NULL
Now your replication would use the updatets column and pull rows with a updatets greater than the current max(updatets) in the mysql database.
Related
Right now I've got Logstash importing a miniature version of my MSSQL database, using the JDBC plugin. I've got each JDBC input scheduled to run every minute to update Elasticsearch. To update, I'm currently just re-importing every single table and row in my database, and adding all rows to Elasticsearch. When I begin using the full database though, this will be very inefficient as it will take more than a minute to go through the entire database. Is there another way to keep Elasticsearch in sync with my database? I've tried to use the 'sql_last_value' parameter to only import new rows into the database, but this will only work when the 'id' for my database table is a number, and each new entry in the table has a greater number than the last. Some tables in the database have an 'id' which can be completely random (ie. "43f4-f43ef-e44454r"), which cannot be used with 'sql_last_value' as they are impossible to compare. I'm also unable modify the actual database at all, which cuts out a lot of my potential solutions. I feel as if i am out of options here, so can anyone suggest anything that I can try?
I am trying to copy data from views on a trusted SQL Server 2012 to tables on a local instance of SQL Server on a scheduled transfer. What would be the best practice for this situation?
Here are the options I have come up with so far:
Write an executable program in C# or VB to delete existing local table, query the data from remote database and then write results to tables in the local database. The executable would run on a scheduled task.
Use BCP to copy data to a file and then upload into local table.
Use SSIS
Note: The connection between local and remote SQL Server is very slow.
Since the transfers are scheduled, so I suppose you want this data to be up-to-date.
My recommendation would be to use SSIS and schedule it using SQL Agent. If you wrote a C# program, I think the best outcome you will gain is a program imitating SSIS. Moreover, SSIS will be a very easy to amend the workflow anytime.
Either way, to make such program/package up-to-date, you will have to answer an important question: Is the source table updatable or is it like a log (inserts only)?
This question is so important because it will determine how you will fetch the new updates from the source table. For example, if the table represents logs, you will most probably use the Primary Key to detect new records, if not, you might want to seek a column representing update date/time. If you have the authority to alter the source table, you might want to add timestamp column which represent the row version (timestamp differs than datetime)
For building an SSIS package, it will mainly contain the following components:
Execute SQL Task to get the maximum value from source table.
Execute SQL Task to get the last value where it should start from at the destination table. You can get this value either by selecting the maximum value from the destination table or if the table is pretty large you can store that value in another table (configuration table for example).
Data Flow which moves the data from source table starting after the value fetched in step 2 to the value fetched in step 1.
Execute SQL Task for updating the new maximum value back to the configuration table if you chose this technique.
BCP can be used to export the data compress and transfer over network which can be then imported into local instance of SQL.
Also with BCP data exports can be contained with smaller batches of data for easier management of data.
https://msdn.microsoft.com/en-us/library/ms191232.aspx
https://technet.microsoft.com/en-us/library/ms190923(v=sql.105).aspx
I have a table for bio-metric devices which capture the data as soon as the employees punch their fingers and uses SQL Server 2014 Standard Edition.
However, our legacy devices were exporting log files and we used a vb engine to push to our Oracle table and used to generate the attendance details.
I managed to export the data from SQL Server and built the first set of records. I would like to schedule a JOB with SQL Server with a condition that the Oracle table should receive ONLY the rows those are NOT already inserted from the SQL Server table.
I checked the append possibilities, which dumps the entire SQL Server table data when the job is executed thus duplicating the rows within the Oracle target table, forcing me to discard the job and to build a new one that deletes the Oracle table and recreates when the job is executed. I feel this is a kind of overkill...
Any known methods available to append only the rows those are NOT existing in the Oracle target table? Unfortunately the SQL Server tables doesn't have any unique id column for the transaction.
Please suggest
Thanks in advance
I think the best way is to use sal server replication with Oracle database as subscriber.
You can read about this solution on MSDN site:
Oracle Subscribers
Regards
Giova
Since you're talking about attendance data for something like an electronic time card, you could just send the data where the punch time is > the last time stamp synced. You would need to maintain that value some where, and it doesn't take into account retro actively entered records. If there's a record creation date in addition to the punch time you could use the created date. Further if there is a modified date in the record you could look into using the merge statement as Alex Pool suggested so you could get both new records and modifications synced to oracle.
I want to use merge statement in SSIS. I have one source (Oracle) and one destination (SQL Server). Both the tables and structure are same.
I need to insert, update and delete the data based on some date criteria. My question is should I use Merge Join or Lookup Table as I have more than 40 million records in Oracle.
If need more clarification let me know. I will provide you with more info. I am not good in posting though so forgive me.
Personally i would transfer the oracle table to SQL Server and perform any operations locally. I use this approach almost always (nothing quite to the size of your data) but its also useful when dealing with cloud based databases (latency, etc). Its worth noting that if you don't have a datetime column in your source you can use the ORA_ROWSCN pseudo column which gives you a crude change set to load locally.
I have read lots of tales about Merge join not performing accurate joins - i would expect with data of your size it could be an issue.
Lookup could be an issue also due to the size as it has to cache everything (this would attempt to load all oracle records into SSIS anyway so better to transfer it locally).
Hope this helps :)
I moved an Access database backend to sql server and am still using an Access frontend. I am now getting overwrite errors when entering records. I found a solution (thank you) to add a Timestamp (rowversion) column to the table which I did. However, the timestamp column does not populate for new records nor did it update on the existing records. Your help is much appreciated.
The Timestamp field is for SQL Server's use only. Access neither reads or writes it, but the ODBC driver does.
Its sole purpose is for the ODBC driver to see, before updating a record, if this has changed on the server since the last reading by the driver. If it hasn't, the update is safe - if it has, the record is re-read and the update cancelled.