In our production job, we come across duplicate records in our Datastage dataset file which is causing our job to fail. Is there a way to manually delete duplicate records from a Datastage dataset file i.e., in the unix environment command line? We're thinking of this as a quick-fix solution while we investigate the cause and potential solution. Thanks.
Related
I am working on project where i am getting logs or data feed from outside in CSV and DAT file we have SSIS Package configured.i just wanted to create Trigger on table to reconcile table count with File count if i create triggers is it lead to performance issue?
AVG CSV/DAT FILE COUNT IS 2.5 M Records
Yes, triggers will reduce your system performance by locking the tables for longer time. It is always better to look for other alternate options like CDC or handling the it manually thru' stored procedures or some other way.
I am working with oracle database and I have a lot of queries in a file. My problem is that this file execute very slow.
Some queries in file are delete and I know that, in this moment, database is empty.
So, what happens if I delete "delete queries"? Is better? Talking about performance.
I am working on a project that manages many databases on a handful of servers, and we are using liquibase to keep our database structures current.
As I understand it, liquibase is meant to generate the Databasechangelog and databasechangelogLock tables the first time that it recognizes a Changelog file is executed for a specific database.
The problem is that in one of our environments a couple of the databases seem to try to generate those tables with every run, so that the script that we are getting from liquibase continues to have the initial table creation every time a script is generated. The actual db changes are generated as expected, along with their databasechangelogtable entries, and I can continue generating and running scripts as long as the create table statements for the initial tables is ignored with every run.
This is happening on Databases that work correctly on another server. The majority of the databases on the 'problem' server work correctly as well, so it is only a few of them that are experiencing this problem.
If anyone knows what part of liquibase does the check to decide if the tables need to be generated, or how I can manipulate the run to not try to create them every time it would be very appreciated.
Thanks
Pretty much every Liquibase command checks for the tables and creates them if needed. If you want to know 'which part of Liquibase' is doing this, it is typically the StandardChangeLogHistoryService and the StandardLockService.
I mistakenly inserted few test records using PROD GUI, which got written to PROD Database. Is there a way to find what tables and column did those record touched ?
Thanks
I suppose you don't have a running trace, CDC or other tracking mechanism enabled. So it seems like the following steps would be a reasonable solution:
Make sure that you can't find and drop that data from the
Application GUI
Run SQL Profiler Trace using Tuning Template (it will give you enough information). Include ApplicationName and HostName columns to identify your connection.
Insert one more test record using UI (try to do the same operations as you did before)
Stop the trace and find the data you've inserted in it.
Identify other modification which was done from your application using ApplicationName, HostName, and SPID.
Create a SQL Script to delete those records.
Identify records which you had inserted before (probably they were inserted into the same tables)
Write a query to delete them too
Open transaction
Delete those records
Check that you have deleted only needed records
Commit transaction
UPD: according to the comment to this answer (with which I completely agree), if you have DEV or TEST environment on which you can do the same operation, do it and find modified records there. After that find modified records in the same tables on PROD.
P.S. I can not guarantee that following these steps you will be able to clean the data you've inserted, but probably you will be able to do this. I also recommend creating a full backup before deleting data.
If you have proper transaction logging enabled and using SQL Server 2008 and above. You can try using the Change Data Capture Stored Procedures (Transact-SQL) and check the changes happened to the tables. Hope it helps
Well you could track through the code to see what tables it touches. Run Profiler on dev to see what code it sends or procs it calls when enter a new records the same way that you did on prod.
If you have formal PK and FK relationships, you likely will find out by trial and error because it won't let you delete the parent records until all the children are deleted. And test with some other record on the dev environment to figure out what tables might be involved. Or you could script the FKs to see what other tables are related to the parent table.
If you have auditing (as every Enterprise solution should have, but I digress), you can often find out by looking in the audit tables for transactions at that time. Our audit tables have both the datetime of the transaction and the user which makes it easier to filter for these things.
Of course if you know your data model, you should have a pretty good idea before you start. Or if you have a particular id that is in all the child tables and you do not have nice convenient FKs, then you could check out the system tables to find what tables have that column name. That assumes a fairly standard naming convention though. If you call the same column different things in different tables, yo u might miss some.
If you are using an ORM, there should be some way to check what tables are in the object related to the particular task you did. So if you inserted a test order of instance, check out what is contained in the order object.
I am using a couple bash script that writes to multiple tables "never the same" in the same sqlite database.
I am getting database locked errors, I did read around, but I am not sure that I should be getting this error given that I am writing to different tables in the same database, never to the same table in the same database.
Did I misunderstand this ?
Regards
SQLite uses one file for all the tables. It locks the whole file, not single tables.
As a workaround, you might need to repeat the operation later after a time interval several times, until it succeeds.