I have a very simple SQL query in my SSIS (VS 2017) Data Flow. It connects to Oracle via Native OLE DB\Oracle Provider for OLE DB and uses SQL Command to query the Oracle view. The destination table is a SQL Server 2017 table. If I query only the first 20 columns or so (I am querying 57 columns), I get all 1,060,000ish records. As I start to add more columns, the rowcount drops. I have already removed any date fields from both tables, and have done quite a few data conversions (source table has several varchar2(4000) fields that need to be SUBSTR to reasonable lengths in the SQL destination table. All fields in the destination table are nullable. When I pull the SQL out of SSIS and run it in SQL Developer, I get the right row count. When I run it in SSIS, it drops from 1.06 M rows to around 28k. I already tried the SQLChick hack (https://www.sqlchick.com/entries/2012/9/2/resolving-missing-records-in-ssis-from-oracle-source.html) doesn't work and causes connection errors (I had to use VS Code to add that property to my Oracle connection, then when I went back to VS, the connection was broken. When opening it back up to re-enter connection credentials, the extra property gets dropped.) I have reduced and increased the Rows per Batch and Maximum insert commit size values to zero avail. I have also set the RetainSameConnection property to True for all the Connection Managers. I'm at a loss! (As you can see from the pics, both jobs finish "successfully".)
This code returns all records:
SELECT
PIDM,
STUDENT_ID,
LAST_NAME,
FIRST_NAME,
MIDDLE_NAME,
LFM_NAME,
FML_NAME,
SORT_NAME,
GENDER,
ETHNIC_CODE,
ETHNIC_CODE_DESC,
LEGACY_CODE,
LEGACY_CODE_DESC,
ADDR_STR_LINE1,
ADDR_STR_LINE2,
ADDR_STR_LINE3,
ADDR_CITY,
ADDR_COUNTY,
ADDR_STATE,
ADDR_NATION,
ADDR_ZIPCODE,
ADDR_AREA_CODE,
ADDR_PHONE
FROM <TABLE_NAME>
This code returns only 28k:
SELECT
PIDM,
STUDENT_ID,
LAST_NAME,
FIRST_NAME,
MIDDLE_NAME,
LFM_NAME,
FML_NAME,
SORT_NAME,
GENDER,
ETHNIC_CODE,
ETHNIC_CODE_DESC,
LEGACY_CODE,
LEGACY_CODE_DESC,
ADDR_STR_LINE1,
ADDR_STR_LINE2,
ADDR_STR_LINE3,
ADDR_CITY,
ADDR_COUNTY,
ADDR_STATE,
ADDR_NATION,
ADDR_ZIPCODE,
ADDR_AREA_CODE,
ADDR_PHONE,
ORIGIN_STR_LINE1,
ORIGIN_STR_LINE2,
ORIGIN_STR_LINE3,
ORIGIN_CITY,
ORIGIN_COUNTY,
ORIGIN_NATION,
ORIGIN_STATE,
ORIGIN_ZIPCODE,
EMAIL,
HIGH_SCHOOL_CODE,
HIGH_SCHOOL_CODE_DESC,
HIGH_SCHOOL_CITY,
HIGH_SCHOOL_STATE,
HIGH_SCHOOL_GPA,
HIGH_SCHOOL_RANK,
PRIOR_COLLEGE_CODE,
PRIOR_COLLEGE_CODE_DESC,
PRIOR_COLLEGE_DEGREE_CODE,
PRIOR_COLLEGE_DEGREE_CODE_DESC,
PRIOR_COLLEGE_CITY,
PRIOR_COLLEGE_STATE,
ADMIT_FLAG,
GENERAL_STUDENT_FLAG,
CURRENT_ENROLLMENT_FLAG,
LETTER_CODES,
CONTACT_CODES,
COMMENT_CODES,
DIRECTORY_EMAIL,
ADDR_DIVISION_CODE,
HIGH_SCHOOL_CLASS_SIZE,
ETHNICITY,
RACE_CODE,
REGULATORY_RACE,
INT_LANG
FROM <TABLE_NAME>
Troubleshooting steps from the comments
If you run the all column version of the query in sql developer (whatever the Oracle query tool is) using the same credentials as the SSIS package, do you get 28k rows or 1M?
1M records are returned in SQL Developer when I use the same credentials SSIS is using. –
As painful as it may be, I would add 1 column, run, observe results. The first time you see a drop in row count, interrogate the heck of the source data (data type, collation, whether some permission thing is at play). If nothing seems out of place, edit the question to include the full table definition and identify what the first source column is that is throwing the results off.
I've done that. Column by column. I've even added a column that already existed (ADDR_STR_LINE1) as ORIGIN_STR_LINE1 and just aliased it, knowing that ADDRR_STR_LINE1 had already worked and both fields shared the exact datatypes/lengths etc. I just ran it with this code:SELECT PIDM, ORIGIN_STR_LINE1, ORIGIN_STR_LINE2, ORIGIN_STR_LINE3, ORIGIN_CITY, ORIGIN_COUNTY, ORIGIN_NATION, ORIGIN_STATE, ORIGIN_ZIPCODE FROM ODSMGR.RECRUIT_PERSON_OSU and it returned 1m records.
While little, consolation, you hitting all the troubleshooting steps I'd employ. I suppose the next item I would try to rule out is some bizarre row width issue/bug. Add a new data flow. As your source query, take one of your varchar2(4000) fields and duplicate it 60 times i.e. SELECT ADDR_STR_LINE1 AS Col0, ADDR_STR_LINE1 AS Col1, ..., ADDR_STR_LINE1 As Col59 FROM Owner.Table and connect that to a Derived Column task (it doesn't need to do anything, just serve as an anchor point) and run it. Do you get 1M or 28k?
Adding more of my troubleshooting steps. 1) Created a view off the original table, casting all of the fields that would need to be truncated as VARCHAR(proper length based on dest table). 2) Added/substracted fields piecemeal, until I thought I had a stable query, knowing that if I added <this fields>, <this many rows> would be dropped. But, for instance, I added PRIOR_COLLEGE_CITY and the first time, my counts dropped from 1063202 to 952755, but then later, I ran it again, and the counts dropped from 1063202 to 953989, so even if it was a data issue (it's not) it's not a consistent one.
Once I got my 953989 rows into the destination table, I compared which PRIOR_COLLEGE_CITY records were missing. In the Source Data Flow, I explicitly queried for those records, and they loaded fine, so again, not a data issue.
According to the picture you provided, when Source component output the records, some records have lost, so we could determine that this problem occurs in Source component.
In this case, please try to check the following thing in your case.
1.Run the query(not the views but the query inside the view) in your Oracle environment when execute the query in Source component, then check whether the number of records(returned from Oracle environment)is equal to the number of records(returned from SSIS Source component). Do this on a separate data task.
2. Check if there are some changes on the source table.
3. If the returned results is correct when running the query in Oracle environment, please try to compare the correct results with the SSIS Source returned results, and analyze the missing data.
I had a similar problem, mostly with odbc driver for oracle!
The problem not only lies on the volume of or rows that returns but in my case, for some reason it grouped the values of the first column also.
The only solution I ve found is to use another driver besides odbc and oledb.
Using the native Oracle Destination and Oracle Source in VS2017 it worked perfect and also the performance was better than odbc and ole db.
enter image description here
I was having a similar issue: 1,470,491 rows in the Oracle view that I was querying, all would come across when I run the package in Visual Studio, but only 377,257 rows would be read when I ran the package from SQL Agent. I tried the SQLChick "UseSessionFormat" hack that you mentioned. While editing the connection string used by the job (it comes in from configuration) I noticed that the connection string in the package had a "USERNAME" parameter as well as a "user id" paramter, but the configuration used by SQL Agent only had "USERNAME". I added "user id" parameter to the configuration used by SQL Agent and after that, the job retrieves all 1,470,491 rows.
This question already has answers here:
Any way to edit data on MS-Access form using SQL View with two tables
(3 answers)
Closed 2 years ago.
I'm using MS Access 2016 and SQL Server 2012.
I have single table with a schemabound view over the table. Our design was to allow our users to insert data through the view so we can do some id lookups for them and audit who inserted the record using an instead of insert trigger.
No matter how I configure it, Access doesn't want to let me add records to the view. If I use the table, it works as expected and it lets me insert new records.
I've tried the following:
Opening the external table and try to add new record at bottom - no luck
Creating an "Update" query - no luck
Creating an "Append" query - no luck
Creating a form, setting the view as a source, and setting Data Entry = Yes - no luck
Is what I'm trying to do even possible? I come from a SQL/C# background, and have only basic MSAccess skills. Any help is appreciated!
EDIT: I found out the View had a group by when it shouldn't have. I corrected that, but now it is complaining that it is not updatable because there are multiple underlying tables.
EDIT2: I've made progress but it's not ideal.
More info:
This view represents a datawarehouse fact, and joins to two dimension tables to get business keys.
I was able to get this to work by creating a "dummy" table with all the same fields, then creating a view over that. I was able to insert, but the data comes back "deleted" immediately because my instead of trigger fires.
Data went to the right place, but obviously the view doesn't show any data since it's not actually querying the correct tables anymore.
Clearly the view is irrelevant now, and I'll likely write the trigger over the dummy table.
It's not ideal, but I think my solution will be to have them insert through this dummy table, and then have a second view that does the joins so they can view the results.
I appreciate the detailed answer that was given, but unfortunately it didn't help this situation.
If the view is up-datable from SSMS, then it will be so in Access. So, before you try anything in Access, you have to first ensure that the view is up-datable in SSMS. So not all SQL views are up-datable.
You would do beyond well to test/check this in SSMS.
Once you done this, then you have to link the table from access. (and if you messed around with modify to the view, then delete the linked view in access and re-link). YOU MUST delete the access link and re-link.
And the linking process needs extra CARE. WHEN you link, you will be asked for a primary key for the view. Because a view can have multiple PK's (as a result of several tables), then Access can't know or guess. And far worse is that a SQL view does not in fact have a defined PK, and there is no command or means that Access can use to determine this. So you are PROMPTED during that link process. I note this issue, since if you are using some VBA re-link code, then if you re-link and change the database (or server), then during that re-link process, the PK setting you had will be lost. You ONLY get this prompt during a link of a new table - not a re-link. So, keep this important detail in mind.
You can after the fact (after linking a table) execute the following command to re-enable or "set" which column is to be the PK with this command in Access:
I in fact use this routine:
Sub createPK(strTable As String, strPK As String)
CurrentDb.Execute "CREATE UNIQUE INDEX " & strPK & _
" ON " & strTable & " (" & strPK & ") WITH PRIMARY"
End Sub
So, to set a PK for a linked view, then I use this:
Call createPK("dbo_tblHotels","ID")
As a FYI:
The above command DOES NOT create a index in access for that linked view, but is a MEANS to ONLY TELL Access what column to use for the PK. So, in this context, the create index command is not creating a index, but is the means/approach/how/process in which you can tell/set in Access what column is to be used for the PK view. As noted, you only need to do the above if you using code to re-link (or create) a table (view in this case).
So, if you using the Access UI, and you link to a view? Then Access will prompt/ask you to choose a column for the PK. You can as noted after linking use the above routine/command in Access to set which column is to be used as the PK if you missed the prompt, or as noted are using some VBA code to re-link.
A re-link (refresh) with the Access UI to the SAME database will preserve the PK setting. But if you change the connection string (database or server name), then the PK setting will be lost.
First, this is a really good answer to a similar question. Try adding an INSTEAD OF trigger to the view in SQL Server. In order to insert into a view the key columns must all be present and each table must be UNION'ed together. INSTEAD OF trigger can be made to do exactly as you wish.
Summary
In a Microsoft Access 2010 database (accdb), I have a form that dynamically loads other forms into a subform object on the main form. The forms used in the subform object are bound to ODBC pass-through queries that execute stored procedures to return recordsets. I can't figure out why I can use one sproc and it works perfectly fine, but if I bind the form to another sproc, it fails to load the subform.
Technical Walkthrough
I have two pass-through ODBC queries. qryGood and qryBad. They use identical ODBC connection strings (ODBC;DRIVER=SQL Server;SERVER=MyServer;UID=MyUser;Trusted_Connection=Yes;DATABASE=MyDatabase), and the SQL behind them is identical, but pointing to two different SQL stored procedures on the SQL 2012 database server.
qryGood source: exec spGoodProc 123456
qryBad source: exec spBadProc 123456
The SQL behind the sproc is very simple. Return records from a single table, filtering by the ID passed as a parameter. (Some will do more complex things, but I am just focusing on a simplified example here that demonstrates the problem.)
The RecordSource property of frmMySubform is set to qryBad.
The subform SourceObject is set via VBA code: sfrmMain.SourceObject = "frmMySubform" No errors are thrown at this point. While the SourceObject property now returns frmMySubform, the .Form object does not seem to be set.
I then try to reference a property on the subform: Debug.Print sfrmMain.Form.Name This fails with error 2467: The expression you entered refers to an object that is closed or doesn't exist.
I can then open frmMySubform in design view, change the RecordSource property to qryGood and it works just fine. This seems to point to a problem with spBadProc that only manifests itself when used as the RecordSource on a subform.
What I Have Tried
In an effort to troubleshoot this problem, I have used the process of elimination to narrow this down as far as I can, but I am still not understanding why the one sproc works and the other doesn't. Both return records just fine in SQL and when running the pass-through query directly. Both work fine when opening the form directly. It only becomes a problem when the form is set as a SourceObject in a subform control.
I have used sp_procedure_params_rowset to compare the parameters in the sprocs, and they are identical. I have compared the data types of the columns in SQL and there is nothing new or different in tblBad that isn't in tblGood. I have also tried profiling the SQL server while setting the form, and it seems to call the sproc just fine. I didn't see any clues when comparing the trace between the bad and the good calls.
Setting the RecordSet directly to an ODBC link to tblBad works just fine (and I presume a view would be fine as well) but having the simple stored procedure wrapper somehow triggers the error.
I have also compared the security, properties and extended properties for spGoodProc and spBadProc and they are identical.
My Question
What can I do on the troubleshooting side to reduce this down further? Has anyone out there encountered similar issues with bound sprocs on subforms? I am working on a very complex database with hundreds of forms, tables and queries, so I would really like to understand why this is occurring before I go too far down this path.
Thanks in advance for any insight you are able to share on this perplexing problem. :-)
Found it!
After tracing it back to something with the specific table, I removed all constraints, keys, and then columns from a copy of the table, systematically testing to see if I could pinpoint the problem. Sure enough, it was a specific column name in the stored procedure!
Simply aliasing this column to a different name solved the problem. (See below for expanded details)
Update after Further Testing
After additional testing to further pinpoint the issue, I think I now understand why this was occurring. When you link an ODBC table and specify a unique (key) column, Access will automatically attempt to set the LinkMasterFields and LinkChildFields to the key column name when a subform is loaded and the subform has a column with the same name. While this works fine with linked tables or views, it does not work when the RecordSource of the subform is set to a stored procedure.
If you attempt to do this by manually adding the subform, you will see the following notification:
However, if you set the subform target through VBA code, you don't get any warning or error message. It simply doesn't (fully) load the subform. #Albert D. Kallal, you were right on about this being related to a master/child fields issue!
I was able to consistently reproduce the issue in a test database file in both Access 2010 and Access 2016. If you would like to see this for yourself, you can use the following steps to reproduce it:
Create a SQL table with a PrimaryID column.
CREATE TABLE [dbo].[tblBugTest](
[PrimaryID] [int] NOT NULL,
[TestColumn] [nchar](10) NULL
) ON [PRIMARY]
Add a couple test records to the table you just created.
Create a Stored Procedure to return the records.
CREATE PROCEDURE [dbo].[spBugExample]
AS SELECT * FROM tblBugTest
Create a blank Microsoft Access database (accdb).
Using ODBC, create a linked table to tblBugTest.
Important: Select PrimaryID as the unique column.
Create a pass-through query named qryPassThrough using ODBC to the same database, and set the SQL to exec spBugExample.
Open the query to verify that it returns records.
Create three blank forms in the database. frmMain, frmSubForm, and frmBlank.
Add frmBlank as a subform to frmMain. Name the subform sfrmSubform.
Set the RecordSource of frmMain to the linked table.
Add a button to frmMain to switch the subform from frmBlank to frmSubForm.
Private Sub cmdShowBug_Click()
With Me.sfrmSubform
.SourceObject = "frmSubForm"
Debug.Print .LinkMasterFields
Debug.Print .LinkChildFields
Debug.Print .Form.Name
.SourceObject = "frmBlank"
End With
End Sub
Set the RecordSource of frmSubForm to qryPassThrough.
Drop a couple bound controls onto frmSubForm.
Test frmSubForm by itself. It should load a record from tblBugTest.
Open frmMain and click the button. It should throw an error.
If you step through the code, you will see that before setting the SourceObject, the LinkMasterFields property is blank. After setting SourceObject, you can hover over LinkMasterFields and see that it is now set to the PrimaryID column.
Workarounds
Changing any of the following will work around the error by avoiding the problematic auto-linking of the master/child fields.
Delete and relink the linked table, this time not specifying a unique column.
Alias the column in the Stored Procedure to a different name than the unique column.
Clear the RecordSource property of the parent form.
Clear the subform RecordSource property and set the RecordSet after loading the subform.
Use a view or linked table instead of a stored procedure in the subform.
Keep in mind that the subform data source will be attempted to be loaded BEFORE the main form loads. What this suggests is that on the main form's load event, you will
First setup the PT query.
Then set the OBJECT source of the sub form.
In other words, the source object of the sub form control should be blank.
You code then to set up the PT query will be:
With currentdb.queryDefs("qryGood")
.SQL = "EXEC spGoodProc " & 123456
end with
Of course you can replace the 123456 with a varible, or even a value from a text box (from the main form).
Now that the PT query is setup, you THEN are to set the form that the sub-form is to load.
So, after above code, we then have:
me.mySubForm.SourceObject = "name of subform goes here"
So, it should be about a total of 4 lines of code. And as above shows, you don't even need any connection string stuff in your VBA code.
So, just keep in mind:
Setup the PT query as per above. You can then launch a report, or even a form, or in this case set the form that the sub-form control is to load. This also suggests/hints that you need to remove the source object of the sub form control (leave it blank).
You can have the sub-from source object set, but then this would suggest that you setup the PT query source as per above BEFORE you launch the main form with the sub form based on the PT query. As noted, this set of steps is required since the sub form actually load and resolves it data source BEFORE the main form displays and renders. So, by leaving the source-object blank for the sub form, then you the developer re-gains complete control over the order of loading.