We have survey data for each survey on a server. The data is separated with one method I can download the column names (via pooling) and with another I can download the data (via SSE). I have accomplished to write a procedure that creates a datatable dynamically for each survey I choose to download. Now I have to get the data into that table. For this I have streamed the data via SSE into a List(of String) where each element comprises a comma separated string.
This looks like
For Each x As String In stringList
Console.WriteLine(x)
Next
would give me
(1)data:[1,2,1,5,2,6,John,Winchester,234]
(2)data:[5,3,2,4,1,6,Mike,Lenchester,555]
...
Each Element of the List is a dataset I have to put into one column of my datatable. So I guess I have to loop through the List Object and the pick each element between the comma and write them into the columns.
So my problem now is to get the data into the database.
Usually I provide an approach but this time I have no clue how to start.
I tried to experiment with this
.Parameters.Add("#id", SqlDbType.varchar(max)).Value = x
but ended up in frustration.
Could anyone give me something to start with? The data size is up to 500MB if I store it in a .txt file.
Related
Below I have linked my CSV for a store (woocommerce), I have columns C and D pulling data from other columns. So when I export as CSV and upload to woocommerce the data from those cells isn't represented correctly, what I need to do is make those cells actually contain the text that is displayed using the concentrate function.
Is there an easy way to go about this, or am I bound to copy paste from two columns and add text that repeats?
https://docs.google.com/spreadsheets/d/189DoiV2LwV5JrXqPVAZTL9ww5hAq2Ws7_IdWqsd7mqo/edit?usp=sharing
I have figured it out. I have selected the whole column that used the formula to generate the value (text) and copied it, then I have pasted it somewhere else in the same document, deleted the original column and pasted the value column in its place.
I have user variable of type string(MyNames) which contains a list of comma separate values. I have a conditional splits which outputs around 96 names. In the next step, I only want to put only those names in a Recordset Destination which match with the names in the comma separated list string variable MyNames. The Recordset Destination is a uservariable of type Object.
I am new to SSIS and I dont have any scripting experience. Your help will be appreciated. Thank You
You can use a lookup transformation with full cache (it will require storing the list of names first)
https://blogs.msdn.microsoft.com/mattm/2008/10/18/lookup-cache-modes/
Depending on the comma separated values source, I can give you more alternatives.
I am using SQL Server 2008 and I have a column in a table, which has values like below. It basically shows departure and arrival information.
-->Heathrow/Dublin*Dublin/Heathrow
-->Gatwick/Liverpool*Liverpool/Carlisle *Carlisle/Gatwick
-->Heathrow/Dublin*Liverpool/Heathrow
(The 3rd example shown above is slightly different where the person did not depart from Dublin, instead departed from a Liverpool).
This makes the column too lengthy, and I want to remove only the adjacent duplicates, so the information can be shown like below:
-->Heathrow/Dublin/Heathrow
-->Gatwick/Liverpool/Carlisle/Gatwick
-->Heathrow/Dublin***Liverpool/Heathrow
So, this would still show the correct travel route, but omits only the contiguous duplicates. Also, in the 3rd case, since the departure and arrival information location is not the same, Iwould like to show it as ***.
I found a post here that removes all duplicates (Find and Remove Repeated Substrings) but this is slightly different from the solution that I need.
Could someone share any thoughts please?
The first step is to adapt the process defined in the following link so that it splits based on /:
T-SQL split string
This returns a table which you would then loop through checking if the value contains an *. In that case you would get the text values before and after the * and compare them. Use CHARINDEX to get the position of the *, and SUBSTRING to get the values before and after. Once you have those check both values and append to your output string accordingly.
So you have a database column that contains this text string? Is your concern to display the data to the user in a new format, or to update the data in your database table with a new value?
Do you have access to the original data from which this text string was built? It would probably be easier to re-create the string in the format you desire than it would be to edit the existing string programmatically.
If you don't have access to this data, it would probably be a lot simpler to update your data (or reformat it for display) if you do the string manipulation in a high-level language such as c# or java.
If you're reformatting it for display, write the string manipulation code in whatever language is appropriate, right before displaying it. If you're updating your table, you could write a program to process the table, reading each record, building the replacement string, and updating the record before moving on to the next one.
The bottom line is that T-SQL is just not a good language for doing this sort of string examination and manipulation. If you can build a fresh string from the original data, or do your manipulation in a high-level language, you'll have an easier job of it and end up with more maintainable code.
I wrote a code for the first example you gave. You still need to
improve it for the rest ...
DECLARE #STR VARCHAR(50)='Heathrow/Dublin*Dublin/Heathrow'
IF (SELECT SUBSTRING(#STR,CHARINDEX('/',#STR)+1,CHARINDEX('*',#STR)-CHARINDEX('/',#STR)-1)) =
(SELECT SUBSTRING(#STR,CHARINDEX('*',#STR)+1,LEN(SUBSTRING(#STR,CHARINDEX('/',#STR)+1,CHARINDEX('*',#STR)-CHARINDEX('/',#STR)-1))))
BEGIN
SELECT STUFF(#STR,CHARINDEX('*',#STR),LEN(SUBSTRING(#STR,CHARINDEX('/',#STR)+1,CHARINDEX('*',#STR)-CHARINDEX('/',#STR)-1))+1,'')
END
ELSE
BEGIN
SELECT STUFF(#STR,CHARINDEX('*',#STR),LEN(SUBSTRING(#STR,CHARINDEX('*',#STR)+1,LEN(SUBSTRING(#STR,CHARINDEX('/',#STR)+1,CHARINDEX('*',#STR)-CHARINDEX('/',#STR)-1)))),'***')
END
I have a csv file read into a TextFieldParser.
Before I place the data rows into a DataTable, I want to add a couple of extra fields that are not in the csv.
This line writes all the csv data into the table ok -
tempTable.Rows.Add(parser.ReadFields())
If I do something like this -
tempTable.Rows.Add(parser.ReadFields(), stationID, sMaxSpeedDecimal, sqlFormattedDate)
the Row.Add seems to treat all the Parser data as one field and then appends the new columns. Basically the parser data is lost to the database.
How can I add additional columns so the tempTable.Rows.Add includes all the parser data plus new column data in one write?
You must either pass one array containing all the field values or else pass all the field values individually. Because you are passing multiple arguments, it is assumed to be the latter and the array is treated as one field value. You must either break up the array and pass each field individually, e.g.
Dim fields = parser.ReadFields()
tempTable.Rows.Add(fields(0), fields(1), stationID, sMaxSpeedDecimal, sqlFormattedDate)
or else combine the extra field values with the original array to create a new array, e.g.
Dim fields = parser.ReadFields().Concat({stationID, sMaxSpeedDecimal, sqlFormattedDate})
tempTable.Rows.Add(fields)
I have a database formatting problem in which I am trying to concatenate column "B" rows based on column "A" rows. Like So:
https://docs.google.com/spreadsheet/ccc?key=0Am8J-Fv99YModE5Va3hLSFdnU0RibmQwNVFNelJCWHc
Sorry I couldn't post a picture. I don't have enough reputation points YET. I'LL Get them eventually though
So I'd like to solve this problem within Excel or Access. Its currently an access database, but I can export it to excel easily. As you can see, I want to find "userid" in column A and where there are multiple column A's such as "shawn" I'd like to combine the multiple instances of shawn and concatenate property num as such.
Even though there are multiple instances of column A still, I could just filter all unique instances of the table later. My concern is how to concatenate column B with a "|" in the middle if column A has multiple instances.
This is just a segment of my data (There is a lot more), so I would be very thankful for your help.
The pseudo code in my head so far is:
If( Column A has more than one instance)
Then Concatenate(Column B with "#"+ "|" +"#")
I'm also wondering if there is a way to do this on access with grouping.
Well Anyways, PLEASE HELP.
In excel we can achieve it easily by custom function in vba module. Hopefully using vba(Macros) is not an issue for you.
Here is the code for the function which can be added in vba. (Press Alt+F11, this will take you to visual editor, right click the project and add a module. Add the below code in module)
Public Function ConcatenatePipe(ByVal lst As Range, ByVal values As Range, ByVal name As Range) As String
ConcatenatePipe = ""
Dim i As Integer
For i = 1 To lst.Count
If name.Value = lst.Item(i).Value Then ConcatenatePipe = ConcatenatePipe & "|" & values.Item(i).Value
Next
ConcatenatePipe = Mid(ConcatenatePipe, 2)
End Function
This function you can use in excel in F Column of your example. Copy the below formulla in F2 and the copy paste the cell to rest of F column. =ConcatenatePipe($A$2:$A$20,$B$2:$B$20,E2)
I believe you can solve this with an SQL GROUP BY function. At least, here's how I'd do it in MySQL or similar:
SELECT userid, GROUP_CONCAT(propertynum SEPARATOR '|') FROM Names GROUP BY userid
as described in this stack overflow post: How to use GROUP BY to concatenate strings in MySQL?
Here's a link on how to use SQL in MS Access: http://www.igetit.net/newsletters/Y03_10/SQLInAccess.aspx
Unfortunately there is not a GROUP_CONCAT function in MSAccess, but this other SO post explains some ways round that: is there a group_concat function in ms-access?