I imported a table into access via vba as dao. Now I need to run a couple of replacements and updates of some table values.
This includes:
- mapping/replacement of Strings (i.e., replace all "ABB" with "Read-Only", "AAB" with "Write", etc.)
- remove leading Strings (i.e. cut leading "_customname")
As I have to work with a HUGE table, I am in need of the fastest approach:
Should I run simple update queries within VBA?
Should I go for a line-by-line approach with an "if then apprach"?
Should I already change the value during the import process?
Or is there a better solution after all?
Related
Right now, I run a stored procedure whose output feeds a "Create CSV Table" Data Operations component. This component, not surprisingly, outputs a comma-delimited list of fields, which is not supported by our remote system. The fields need to be tab-delimited. One would think that the Data Operations component would have a tab (or other character-delimited option). But no, only commas are available, and no other Data Operations component outputs a tab-delimited table.
Using any mechanism for which we'd have to write code is completely the last option, as there's no need for code to use CSV. Also, any mechanism which requires paying for 3rd party components is categorically out, as is using any solution which is in preview mode.
The only option we've thought of is to revamp the stored procedure which outputs a single "column" containing the tab-delimited columns, and then output to a file - ostensibly, a comma-delimited file, but one without commas embedded inside (which is allowed for my system) so that the single column isn't itself enquoted.
Otherwise, I guess Function Apps is the solution. Anyone with ideas?
The easiest way is to use string function and replace comma with other delimiter. If you could accept this way, after creating the csv table I initiate a string variable with this input replace(body('Create_CSV_table_2'),',',' ').
And this is the result.
And if you don't want this way, yes you have to solve it with code and the Function is a choice.
The process is one where I would get 28 fixed width files and combine it into one table. In the past, this was done via FoxPro. As I have learned today, there were duplicates for which FoxPro did not reject or have any issues with. I have discovered that I need to write a merge statement in order to import the 28 and not get tripped up by duplicate primary key errors when I try to import each one separately using the Import Wizard.
I use Management Studio with a SQL Server Express front end and therefore can't create SSIS packages.
I am going to break this up into two questions so as to not make this too convoluted. First, I have since converted the fixed width files into tab-delimited text files by using Excel.
First question: Can one construct a merge statement that brings the files (tab-delimited) into SQL Server from the C drive? I could import each using the import wizard but that is cumbersome. I know how to write a merge statement but it demands that the data already exist in SQL Server. Below is an example. The question is how would I bring it in from outside.
Merge Industry as TARGET
Using Table1 as SOURCE
On (TARGET.Primary keys 1-9 = SOURCE.Primary keys 1-9)
No, you can't import data during or as part of a MERGE statement. The MERGE operation is purely for the 'upsert' situation; constructing logic on combining two result sets with criteria for matches and mismatches.
To get data into SQL Server you can either work via the UI (which is pretty boring and error prone when you have 28 files), or you can use some of the built in commands such as BULK INSERT.
Perhaps you could BULK INSERT the files one by one, and merge after each import.
If you wanted to continue using Foxpro but eliminate the duplicate records the first piece of advice would be to quit using the Import Wizard.
Wizards may be convenient to use, but they come with their own set of 'baggage' which can be problematic.
Aside from saying that they are in fixed field length format, you don't indicate which format(s) the 28 import files are in (CSV, SDF, TXT, ect.). Regardless you can farily easily write Foxpro code to handle all of the importing without the use of a 'Wizard'.
Then once all of the records have been imported you can readily eliminate the duplicates with something like the following:
SELECT ImportDBF && Assuming it is used EXCLUSIVELY
DELETE ALL
INDEX ON <primary key> UNIQUE TAG Uniq && Create an Index on only UNIQUE instances of your Primary key field
RECALL ALL && Recall only those UNIQUE records
DELETE TAG Uniq && Eliminate the temporary Index
PACK && PACK out the duplicate records
Now your Foxpro data table should be ready to go.
Good Luck
I'm looking for a way to remove special characters from a field within my Access database. The field has both text and numbers along with dashes, underscores and periods. I want to keep the letters and numbers but remove everything else. There are multiple examples of VB scripts, and some in SQL, but the SQL examples I've seen are very lengthy and do not seem very efficient.
Is there a better way to write a SQL script to remove these characters without having to list each of the special characters such as the below example?
SELECT REPLACE([PolicyID],'-','')
FROM RT_PastDue_Current;
If you are actually manipulating the data and executing code from the context of the MS Access application, then SQL calls can call any public function inside the modules in the MDB. You could write a cleanup function, then
UPDATE Mytable SET MyField=Cleanup(MyField)
Other than that, I have yet to encounter any RDBMS database engine that has much advanced string manipulation features beyond the simple Replace you've mentioned.
I am using SSIS (SQL Server 2008 R2) to transform an input CSV file into an SQL table. Five columns in the input file (reals - e.g. 19.54271) occasionally have a bad value (strings - e.g. "NAN") that cause the package to fail.
What is the simplest way to check these 5 columns for the bad value "NAN", convert that into either a NULL value or known bad numeric (-9999), and write the corrected values into the same final SQL table?
I have the following mess going so far, and finally decided to ask if there is a simpler way...
My current conditional logic:
My Case1 Derived Column Conversion:
Note: Still not sure if I can combine the other derived columns into one instance, but since my destination can have only one input, I suspect I will need to...
TIA
SSIS expressions get hairy and hard to read when the logic is complex or if there are multiple evaluations. In your case you're going to wind up with a bunch of tasks that, individually, do very little.
I'd bundle this up into a script component. That way you could use basic vb or c# functions to evaulate if all of your columns properly convert to numeric and assign defaults when they don't. Additionally, you can implement a try/catch scenario and gracefully send errors to a different output buffer.
Here's some examples of how to use the script component as a transformation:
http://www.bimonkey.com/2009/09/the-script-transformation-part-1-a-simple-transformation/
http://www.sqlis.com/sqlis/post/The-Script-Component-as-a-Transformation.aspx
http://www.codeproject.com/Articles/193855/An-indespensible-SSIS-transformation-component-Scr
It looks like using a script component would be the best way to proceed if my logic had been more complex than simply converting bad values into nulls.
However, the logic with transformation objects is fairly straightforward, so hopefully this can help someone else:
The package (note that I redirect rows for suspect columns in the datasource):
The conditional split logic:
[EDIT: I found that every case condition requires a separate processing path. If you are evaluating multiple expressions, you can do so in one case by appending them with the || operator.]
The derived column logic:
Is it possible in SQL Server to define a String constant? I am rewriting some queries to use stored procedures and each has the same long string as part of an IN statement [a], [b], [c] etc.
It isn't expected to change, but could at some point in future. It is also a very long string (a few hundred characters) so if there is a way to define a global constant for this that would be much easier to work with.
If this is possible I would also be interested to know if it works in this scenario. I had tried to pass this String as a parameter, so I could control it from a single point within my application but the Stored Procedure didn't like it.
You can create a table with a single column and row and disallow writes on it.
Use that as you global string constant (or additional constants, if you wish).
You are asking for one thing (a string constant in MS SQL), but appear to maybe need something else. The reason I say this is because you have given a few hints at your ultimate objective, which appears to be using the same IN clause in multiple stored procedures.
The biggest clue is in the last sentence:
I had tried to pass this String as a
parameter, so I could control it from
a single point within my application
but the Stored Procedure didn't like
it.
Without details of your SQL scripts, I am going to attempt to use some psychic debugging techniques to see if I can get you to what I believe is your actual goal, and not necessarily your stated goal.
Given your Stored Procedure "didn't like that" when you tried to pass in a string as a parameter, I am guessing the composition of the string was simply a delimited list of values, something like "10293, 105968, 501940" or "Juice, Milk, Donuts" (pay no attention to the actual list values - the important part is the delimited list itself). And your SQL may have looked something like this (again, ignore the specific names and focus on the general concept):
SELECT Column1, Column2, Column3
FROM UnknownTable
WHERE Column1 IN (#parameterString);
If this approximately describes the path you tried to take, then you will need to reconsider your approach. Using a regular T-SQL statement, you will not be able to pass a string of parameter values to an IN clause - it just doesn't know what to do with them.
There are alternatives, however:
Dynamic SQL - you can build up the
whole SQL statement, parameters and
all, then execute that in the SQL
database. This probably is not what
you are trying to achieve, since you
are moving script to stored
procedures. But it is listed here
for completeness.
Table of values -
you can create a single-column table
that holds the specific values you
are interested in. Then your Stored
Procedure can simply use the column
from this table for the IN clause).
This way, there is no Dynamic SQL
required. Since you indicate that
the values are not likely to change,
you may just need to populate the
table once, and use it wherever
appropriate.
String Parsing to
derive the list of values - You can
pass the list of values as a string,
then implement code to parse the
list into a table structure on the
fly. An alternative form of this
technique is to pass an XML
structure containing the values, and
use MS SQL Server's XML
functionality to derive the table.
Define a table-value function that
returns the values to use - I have
not tried this one, so I may be
missing something, but you should be
able to define the values in a
table-value function (possibly using
a bunch of UNION statements or
something), and call that function
in the IN clause. Again - this is an
untested suggestion and would need
to be worked through to determine
it's feasibility.
I hope that helps (assuming I have guessed your underlying quandary).
For future reference, it would be extremely helpful if you could include SQL script showing
your table structure and stored procedure logic so we can see what you have actually attempted. This will considerably improve the effectiveness of the answers you receive. Thanks.
P.S. The link for String Parsing actually includes a large variety of techniques for passing arrays (i.e. lists) of information to Stored Procedures - it is a very good resource for this kind of thing.
In addition to string-constants tables as Oded suggests, I have used scalar functions to encapsulate some constants. That would be better for fewer constants, of course, but their use is simple.
Perhaps a combination - string constants table with a function that takes a key and returns the string. You could even use that for localization by having the function take a 'region' and combine that with a key to return a different string!