I need help with something I'm trying to do and cannot find help anywhere.
I'm trying to upload a file to Host via ISPF (ISPF -> Command -> "Send File to Host"). And the problem I'm having is that the file have variable length (it was exported from a DB2 database via a SH script) and it's not working well.
What I mean is:
In windows, the file looks like this:
This is line one
This is the second line
And this is the third
But in Host it always ends being like this:
This is line one This is
the second line and this
is the third
Or similar, depending on the "Record length" I set when allocating the data set.
I don't know if the problem is how I'm creating the file on Host. If the problem is with the send parameters.. or maybe is with the TXT file.
I tried creating the dataset with different Record Formats (F, FB, V, VB) and with all was the same.
And also tried modifing the Send parameters in here:
Send parameters
And checked the txt file, but it seems to be ok.
Well, thanks in advance for the help! and sorry for my the poor english.
UPDATE 03/18
Hi! I'm still trying to solve this. But now I have a more info!
It seems that problem is within the file exported, not the configuration of the terminal.
I'm using a linux script to export the file from a DB2 database, and I'm trying to upload it from a Windows PC (that have the E3270 terminal).
I read a lot, and noticed that the file exported from DB2 to linux only use the "New Line" code to mark an End of Line (0A in hex), while Windows use "Carriage Return + New Line" (which are "0D 0A" in hex).
Could the problem be there?
I tried creating a new txt file with Windows (which end each line with 0D 0A).. and it worked great! But I tried to modify the exported file.. adding an "space" at the end, and then changing that space hex (20) with the 0D (so I had 0D 0A.. it didn't let me "add" a new hexa).. but it didn't work. That.. throw me away the whole theory haha, but maybe I'm doing something wrong.
well, thanks!
From the Host output the file (dataset) is being considered as fixed length of 24. It needs to be specified as Variable (VB) in the send.
From here Personal Communications 6.0.0>Product Documentation>Books>Emulator User's Reference>Transferring Files it appears that you can specify this as per :-
Record Format
Valid only for VM/CMS and MVS/TSO when APPEND is not specified for
file transmission. You can select any of the following:
Default
Fixed (fixed length)
Variable (variable length)
Undefined (undefined mode for MVS/TSO only)
If you select the Default value, the record format is selected
automatically by the host system.
Specifying Variable for VM file transfer enables host disk space to be
used efficiently. Logical Record Length (LRECL)
Valid only for VM/CMS and MVS/TSO when APPEND is not specified for
file transmission.
Enter the logical record length to be used (host record byte count) in
the LRECL text box. If Variable and Undefined Mode are specified as
the record format, the logical record length is the maximum record
length within a file. The maximum value is 32767.
The record length of a file sent from a workstation to the host system
might exceed the logical record length specified here. If so, the host
file transfer program divides the file by the logical record length.
When sending a text file from a workstation to a host, if the text
file contains 2-byte workstation codes (such as kanji codes), the
record length of the file is changed because SO and SI have been
inserted.
To send a file containing long records to the host system, specify a
sufficiently long logical record length.
Because the record length of a workstation file exceeds the logical
record length, a message does not appear normally if each record is
divided. To display a message, add the following specification to the
[Transfer] item of the workstation profile:
DisplayTruncateMessage = Y
As I don't have access I can't actually look into this further but I do recall that it can be a little confusing to use the file transfer.
I'd suggest using the 32767 as the LRECL, along with variable, and perhaps having a look at the whole page that has been linked. Something on the PC side will have to know how to convert the file (ie at LF determine the length of the record and prefix the record with that record length (if I recall correctly 2 bytes/a word)) so you might have to use variable in conjunction with another selectable parameter.
If you follow the link, you will see that Record Format is part of the Defining Transfer Types, you may have to define a transfer type as per :-
Click Edit -> Preferences -> Transfer from the session window.
Click the tab for your host type or modem protocol.
The property page for the selected host or modem protocol opens. The items that appear depend on the selected host system.
Enter transfer-type names in the Transfer Type box, or select them from the drop-down list.
Select or enter the required items (see Items to Be Specified).
To add or replace a transfer type, click Save. To delete a transfer type, click Delete.
A dialog box displays, asking for confirmation. Click OK.
Related
I'm going to create a simple database bank account in C-language but I haven't quite figured out how I'm gonna fetch data for a specific account already created and sent to a file. I was thinking of doing a search from the beginning of the file using fseek for an account number specified since all account numbers will be unique. Is there a way to read the the amount of lines specified below that account number once it is found? For e.g in my file accounts.txt there will be the accounts
Account # : 13398
First Name : Eric
Last Name : Walters
Parish : St.tofu
Year of Birth : 1980
Age : 34
Savings Period : 5 year(s)
Password : Eric1
Account # : 13398
Account balance: $0.00
====================================
I want to search through the file for the account number and fetch it along with everything else 10 lines below it and display it on the screen if this is possible then say 'aye' and point me to a certain area I should study to achieve this and when I'm successful i'll post my coding here to show what I have done.
fseek() allows you to skip a certain number of bytes in each file. If your lines are not always the same length, you will have to read the entire file, not just to search for the account numbers, but also to find the ten newlines that delimit each account. To do this, you are better off using fgets().
The steps would be something like this
foreach line in file
if line starts with "Account Number"
if the number is the one you want
print the next 10 lines
else
skip the next 10 lines
else
keep looking
Firstly, fseek is used to move the file pointer not for searching. For search text, i.e. account id in your case, there is some examples Trying to find and replace a string from file in C. To write your own code, learning the basic use of file handling functions is enough. Furthermore, since your data is structured (every 11 lines represent one account), you code can be accelarated. At last, what you are trying to do is what database software offers and it is hard too implement your own database as fast as commercial software.
You could search in the file, but that would be a bit tedious. Even more tedious if you wanted to modify the account details.
Why don't you use SQLite:
It is designed to replace fopen().
?
I'm doing an Excel loop through fifty or more Excel files. The loop goes through each Excel file, grabs all the data and inputs it into the database without error. This is the typical process of setting delay validation to true, and making sure that the expression for the Excel Connection is a string variable called EFile that is set to nothing (in the loop).
What is not working: trying to input the name of the Excel file into the database.
What's been tried (edit; SO changed my 2 to 1 - don't know why):
Add a derived column between the Excel file and database input, and add a column using the EFile expression (so under Expression in the Derived Column it would be #[User::EFile]). and add the empty. However, this inputs nothing a blank (nothing).
One suggestion was to add ANOTHER string variable and set its properties EvaluateAsExpression to True and set the Expression to the EFile variable (#[User::EFile]). The funny thing is that this does the same thing - inputs a blank into the database.
Numerous people on blogs claim they can do this, yet I haven't seen one actually address this (I have a blog and I will definitely be showing people how to do this when I get an answer because, so far, these others have fallen short). How do I grab an Excel file's name and input it in a database during a loop?
Added: Forgot to add, no scripts; the claim is that it can be done without them, so I want to see the solution without them.
Note: I already have the ability to import the data from the Excel files - that's easy (see my GitHub account, as I have two different projects for importing all sorts of txt, csv, xls, xlsx data). I am trying to also get the actual name of the file being imported also into the database. So, if there are fifty Excel files, along with the data in each file, the database will have the fifty file names alongside that data (so if each file has 1000 rows of data, each 1000 rows would also have the name of the file they came from next to them as an additional column). This point seems to cause a lot of confusion, as people assume I'm having trouble importing data in files - NOPE, see my GitHub; again that's easy. It's the FILENAME that needs to also be imported.
Test package: https://github.com/tmmtsmith/SSISLoopWithFileName
Solution: #jaimet pointed out that the Derived Column needed to be the #[User::CurrentFile] (see the test package). When I first ran the package, I still got a blank value in my database. But when we originally set up the connection, we do point it to an actual file (I call this "fooling the package"), then change the expression on the connecting later to the #[User::CurrentFile], which is blank. The Derived Column, using the variable #[User::CurrentFile], showed a string of 0. So, I removed the Derived Column, put the full file path and name in the variable, then added the variable to the Derived Column (which made it think the string was 91 characters long), then went back and set the variable to nothing (English teacher would hate the THENs about right now). When I ran the package, it inputted the full file path. Maybe, like the connection, it needs to initially think that a file exists in order for it to input the full amount of characters?
Appreciate all the help.
The issue is because of blank value in the variable #[User::FileNameInput] and this caused the SSIS package to assume that the value of this variable will always be of zero length in the Derived Column transformation.
Change the expression on the Derived column transformation from #[User::FileNameInput] to (DT_STR, 2000, 1252)#[User::FileNameInput].
Type casting the derived column to 2000 sets the column length to that maximum value. The value 1252 represents the code page. I assumed that you are using ANSI code page. I took the value 2000 from your table definition because the FilePath column had variable VARCHAR(2000). If the column data type had been NVARCHAR(2000), then the expression would be (DT_WSTR, 2000)#[User::FileNameInput]
Tim,
You're using the wrong variable in your Derived Column component. You are storing the filename in #[User::CurrentFile] but the variable that you're using in your Derived Column component is #[User::FileNameInput]
Change your Derived Column component to use #[User::CurrentFile] and you'll be good.
Hope that helps.
JT
If you are using a ForEach loop to process the files in a folder then I have have used the technique described in SSIS Junkie's blog to get the filename in to an SSIS variable: SSIS: Enumerating files in a Foreach loop
You can use the variable later in your flow to write it to the database.
TO all intents and purposes your method #1 should work. That's exactly how I would attempt to do it. I am baffled as to why it is not working. Could you perhaps share your package?
Tony, thanks very much for the link. Much appreciated.
Regards
Jamie
I ran a query on a MS SQL database using SQL Server Management Studio, and some the fields contained new lines. I selected to save the result as a csv, and apparently MS SQL isn't smart enough to give me a correctly formatted CSV file.
Some of these fields with new lines are wrapped in quotes, but some aren't, I'm not sure why (it seems to quote fields if they contain more than one new line, but not if they only contain one new line, thanks Microsoft, that's useful).
When I try to open this CSV in Excel, some of the rows are wrong because of the new lines, it thinks that one row is two rows.
How can I fix this?
I was thinking I could use a regex. Maybe something like:
/,[^,]*\n[^,]*,/
Problem with this is it matches the last element of one line and the 1st of the next line.
Here is an example csv that demonstrates the issue:
field a,field b,field c,field d,field e
1,2,3,4,5
test,computer,I like
pie,4,8
123,456,"7
8
9",10,11
a,b,c,d,e
A simple regex replacement won't work, but here's a solution based on preg_replace_callback:
function add_quotes($matches) {
return preg_replace('~(?<=^|,)(?>[^,"\r\n]+\r?\n[^,]*)(?=,|$)~',
'"$0"',
$matches[0]);
}
$row_regex = '~^(?:(?:(?:"[^"*]")+|[^,]*)(?:,|$)){5}$~m';
$result=preg_replace_callback($row_regex, 'add_quotes', $source);
The secret to $row_regex is knowing ahead of time how many columns there are. It starts at the beginning of a line (^ in multiline mode) and consumes the next five things that look like fields. It's not as efficient as I'd like, because it always overshoots on the last column, consuming the "real" line separator and the first field of the next row before backtracking to the end of the line. If your documents are very large, that might be a problem.
If you don't know in advance how many columns there are, you can discover that by matching just the first row and counting the matches. Of course, that assumes the row doesn't contain any of the funky fields that caused the problem. If the first row contains column headers you shouldn't have to worry about that, or about legitimate quoted fields either. Here's how I did it:
preg_match_all('~\G,?[^,\r\n]++~', $source, $cols);
$row_regex = '~^(?:(?:(?:"[^"*]")+|[^,]*)(?:,|$)){' . count($cols[0]) . '}$~m';
Your sample data contains only linefeeds (\n), but I've allowed for DOS-style \r\n as well. (Since the file is generated by a Microsoft product, I won't worry about the older-Mac style CR-only separator.)
See an online demo
If you want a java programmatic solution, open the file using the OpenCSV library. If it is a manual operation, then open the file in a text editor such as Vim and run a replace command. If it is a batch operation, you can use a perl command to cleanup the CRLFs.
Here's a summary of my problem:
Our company's old software had a large database of contacts in it.
We switched to a new program and have no way to easily transfer those contacts to it.
The contacts database appears to have 4 files which can all be opened in Excel, but not MSAccess. The four files contain the following:
File 1: A nicely formatted spreadsheet of names and some other BASIC info for each contact. There is an ID number on each one, but the numbers do not seem to correspond to anything in File 2.
File 2: Info on each contact, but not in rows. Instead it looks something like this :
JHGH_CONTACT_BLOB: 1426367745
EMAIL: SMITH
WEB:
PHONE_COUNT: 1
FAX_COUNT: 0
ADDRESS_COUNT: 0
NOTE_COUNT: 0
555-7364
(I changed some info for privacy reasons)
Each blob of info is on a separate spreadsheet row. Each starts off with the same first line, even the number is the same, so it can't be some sort of ID number.
File 3: A file containing a lot of gobbledygook, interspersed with a few readable bits of text here and there. The readable text looks like it belongs to the database (ie, it is info on contacts like place of work and other notes.)
File 4: Contains one row and one column labeled ID, with the number 12725 in it.
I need to somehow get the info from File 2, into the nicely formatted file 1. In essence, I need to add the phone numbers, emails etc included in a messy fashion in file 2 on their proper rows in file 1.
This probably makes little sense and I thank you for even reading down this far. If you have any suggestions, I'd love to hear them.
Thanks
We have established that you have a DBF file, an FPT file and a CDX file. These are likely to all relate to Visual FoxPro (a now discontinued Microsoft product).
The .dbf file can be opened in Excel via the standard file open dialog by changing "Files of type" to "dBase files (*.dbf)". Going by your original post, Excel seems to be able to open this sensibly in the first place.
The combination of all three files might be accessible by downloading this OLE DB provider for FoxPro which would let you access the database from Excel using the methods outlined here
You can get more info on the specific file structures at the following links: DBF, FPT and CDX. The DBF contains most of the data, the FPT contains binary memo data and the CDX is an index file.
I have a contact manager program and I would like to offer the feature to import csv files. The problem is that different data sources order the fields in different ways. I thought of programming an interface for the user to tell it the field order and how to handle exceptions.
Here is an example line in one of many possible field orders:
"ID#","Name","Rank","Address1","Address2","City","State","Country","Zip","Phone#","Email","Join Date","Sponsor ID","Sponsor Name"
"Z1234","Call, Anson","STU","1234 E. 6578 S.","","Somecity","TX","United States","012345","000-000-0000","someemail#gmail.com","5/24/2010","z12343","Quantum Independence"
Notice that in one data field "Name" there is a comma to separate last name and first name and in another there is not.
My plan is to have a line for each field (ie ID, Name, City etc.) and a statement "import to" and list box with options like: Don't Import, Business>Join Date, First Name, Zip
and the program recognizes those as properties of an object...
I'd also like the user to be able to record preset field orders so they can re-use them for csv files from the same download source.
Then I also need it to check if a record all ready exists (is there a record for Anson Call all ready?) and allow the user to tell it what to do if there is a record (ie mailing address may have changes, so if that field is filled overwrite it, or this mailing address is invalid, leave the current data untouched for this person, overwrite the rest).
While I'm capable of coding this...i'm not very excited about it and I'm wondering if there's a tool or set of tools out there to all ready perform most of this functionality...
I hope this makes sense...
Is there a header row?
usually in CSV files, the first line is the header.
If so you could use the header line to determine the order, just have a list of column names, and only prompt the user if a column name does not match.(this could then be auto added into the predefined list).
EDIT:
even if a header does not exist, its simple enough to add one. The file can be manually edited. Alternatively in your program let the user define it (from your predefined list)
I can't find any tools all ready out there and no one has replied otherwise, so for the sake of leaving a question answered until otherwise notified the answer is: no.