Often I have to write individual SAS programs for various demands where the final output is an EXCEL file. For the sake of reproducibility I often copy and paste the SAS code into a separate EXCEL sheet -- so it is easy to get back to the point when there is a similar request or the need for modifications.
Now I am wondering if there exists a way to make SAS export the current code into an EXCEL sheet in an automated way. Such a task is probably easier if the SAS code has been saved as a physical file on the hard-drive (so it not only exists in the working memory) -- so let us assume that this was indeed the case.
If the program has been saved as a file, then you can read it into a dataset (the same way you read any other file) and export that dataset to Excel. It's up to you how exactly you do both of those things, SAS gives you lots of options.
As far as telling which program is executing, it varies by how you're running things. If you are using DM SAS and the Enhanced Editor, you can use this code:
%let filename= %sysget(SAS_EXECFILEPATH)\%sysget(SAS_EXECFILENAME);
%put &=filename;
and now &filename. holds the current program's path and filename.
If you're running in batch, you have to use the environment variable SYSIN. Other modes have different methods, and it's not always possible (such as if you're in an embedded program in an EG project). See this doc page for more details.
Related
I'm running 12 different python scripts at the same time to screen for a given criteria out of thousands of data points each hour.
I would like to save the output data to a "master csv file" but figured it would be better to put the data into SQLite3 instead as I would be overwhelmed with csv files.
I am trying to transfer the output straight to SQLite3.
This is my script so far:
symbol = []
with open(r'c:\\Users\\Desktop\\Results.csv') as f:
for line in f:
symbol.append(line.strip())
f.close
path_out = (r'c:\\Users\\Desktop\\Results.csv')
i=0
While you can use a sqlite database concurrently, it might slow your processes down considerably.
Whenever a program wants to write to the database file, the whole database has to be locked:
When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY.
So you could end up with more than one of your twelve processes waiting for the database to become available because one of them is writing.
Basically, concurrent read/write access is what client/server databases like PostgreSQL are made for. It is not a primary use case for sqlite.
So having the twelve programs each write a separate CSV file and merging them later is probably not such a bad choice, in my opinion. It is much easier than setting up a PostgreSQL server, at any rate.
I have a huge 1.9 GB data file without extension I need to open and get some data from, the problem is this data file is extension-less and I need to know what extension it should be and what software I can open it with to view the data in a table.
here is the picture :
Its only 2 lines file, I already tried csv on excel but it did not work, any help ?
I have never use it but you could try this:
http://mark0.net/soft-tridnet-e.html
explained here:
http://www.labnol.org/software/unknown-file-extensions/20568/
The third "column" of that line looks 99% chance to be from php's print_r function (with newlines imploded to be able to stored on a single line).
There may not be a "format" or program to open it with if its just some app's custom debug/output log.
A quick google found a few programs to split large files into smaller units. THat may make it easier to load into something (may or may not be n++) for reading.
It shouldnt be too hard to mash out a script to read the lines and reconstitute the session "array" into a more readable format (read: vertical, not inline), but it would to be a one-off custom job, since noone other than the holder of your file would have a use for it.
I come to you completely stumped. I do some side work for a company that uses an old DOS based program to input and retrieve data. This is a legacy piece of software, and they have since moved to either QuickBooks or Outlook for all of their address or billing related needs. However there have been some changes made, and they work with this database fairly regularly. Since the computer that this software is on, is running XP (and none of the other computers in the office can run it) they're looking to phase this software out for when the computer inevitably explodes.
TLDR; I have an old .csv file (roughly two years) that has a good chunk of information on it, but again it's two years old. I have another file called ml.dat (I'm assuming masterlist.dat) that's in the same folder as this legacy software. I open it with notepad and excel and am presented with information like this:
S;Û).;PÃS;*p(â'a,µ,
The above chunk of text is recognized much less within notepad or excel. It's a lot more of the unrecognized squares.
Some of the information is actually readable however. I can for example read the occasional town name, or person's name but I'm unable to get all of the information since there's a lot missing. Perhaps the data isn't in unicode or something? I have no idea. Any suggestions? I'm ultimately trying to take this information and toss it into either quickbooks or outlook.
Please help!
Thanks
Edit: I'm guessing the file might be encrypted since .dat's are usually clear text? Any thoughts?
.DAT files can be anything, they are usually just application data. Since there is readable text, then it is very unlikely that this file is encrypted. Instead you are seeing ASCII representations of the bytes of other content. http://www.asciitable.com/ Assuming single byte values, the number 77 might appear in the file somewhere as M.
Your options:
Search for some utility to load and translate the dat file for that application.
Set up an appropriate dos emulator so you can run this application on another box, or even a virtual machine running freedos or something.
Figure out the file format and then write a program to translate the data.
For #3, you can attach a debugger to the application to trace how the file is read and written. Alternatively you can try to figure out record boundaries (if all the records are the same size, then things are a little bit easier.) Then you can use known values to try to find field boundaries. If you can find (or reverse compile) the source code, then that could also give you insight into the file format.
1 is your best bet, and #2 will buy you some time so that you don't need that original machine anymore. #3 would likely be something to outsource.
If you can find the source or file format, then you just recreate whatever data structure was dumped to the file and read the file into it.
To find which exe opens it, you can do something like:
for %f in (*.exe) do find "ml.dat" %f -c
Assuming the original application was written in C then there would be code something like this to read the first record from the file:
struct SecretData
{
int first;
double money;
char city[10];
};
FILE* input;
struct SecretData secretdata;
input = fopen("ml.dat", "rb");
fread(&data, sizeof(data), 1, input);
fclose(input);
(The file would have been written with fwrite.) Basically you need to figure out the innards of the SecretData structure to be able to read the file.
There likely wasn't a separate utility used to make the file, dumping data and reading it back from a file is relatively easy in most languages.
We have a system based on AcuCobol and vision-files.
We need to dump the entire datafiles in to txt-based files to be imported in to a sql server.
We are using vutil to dump the data right now, but it's creating a fixed-width file and we would want this to be a delimited file of some sort. The command we are using right now is this:
vutil -unload -t sourcefile destinationfile
Now does anyone have any experiance with this, and if so, what would be the best utility for this?
what I'll do to get the job done is this:
Unload your table with a JCL into a file.
Use this file in a batch,
with a copybook matching the description of the table :
(ex: if your table has 3 columns [ID], [NAME] and [ADDRESS], then create a copy CC-1 with:
O1 CC-1
05 ID PIC X (16).
05 NAME PIC X (35).
05 ADDRESS PIC X (35).
For each line move the line into the copy CC-1.
Create a second copy CC-2
O1 CC-2
05 ID PIC X (16).
05 FILLER PIC X VALUE ';'.
05 NAME PIC X (35).
05 FILLER PIC X VALUE ';'.
05 ADDRESS PIC X (35).
05 FILLER PIC X VALUE ';'.
Do a move corresponding of CC-1 to CC-2
Write in the output.
This for each line. You get in the output a CSV file of all your data.
You can reuse the batch and JCL, you just have to change the size of the records and the descriptions of the copy to match the table you are unloading.
Free and easy. :)
Google is your friend... I did a quick search and came up with NextForm which claims to convert Vision to CSV (and other formats).
From the "fine print" it looks like you can download a trial copy but will have to fork out about $400.00 to get a licenced version (that won't drop random records).
I have absolutely no experience with this product so cannot tell you if it really does the job or not.
Edit
Sometimes it is more productive to work with rather than fight against a legacy system.
Have you looked at AcuODBC? This lets Windows programs read/import
Vision files. You could potentially use this to access Vision files from any Windows ODBC enabled application. I believe you may need to compile a data dictionary to make this work, but it could provide a bridge between where you are and where you want to be.
Another approach might be to write your own dump programs in AcuCobol. Read a record, format and
write it. Writting a Cobol program of this complexity should not be too challenging a task provided the files
do not have a complex structure (eg. multiple record types/layouts in the same file, or repeating fields within a
single record).
Your task will be significantly more complex if a single Vision file needs to be post-processed into
multiple SQL Server tables. Unfortunately, complex conversions are fairly
common with Cobol legacy systems
because of their tendancy to use rich/complex/denormalized file structures (unlike SQL which works best with
normalized tabular data).
If the Vision files are more complex that a single SQL Server table, then maybe you could consider using the
AcuXML Interface
to dump the files into XML and then use an XSLT processor to create CSV or whatever other format you
need from there. This would allow you to work with fairly complex Cobol record structures. Again,
this approach means writting some fairly basic AcuCOBOL font end programs.
Based on your comments to my original post, you are not a die-hard Cobol programmer. Maybe it would be
a good idea to team up with someone in your shop that has a working knowledge of the language and
environment before pushing this any further. I have a feeling that this task is going to be a bit more complicated than "dump and load".
If you have Acubench under the Tools Menu there is an option for Vision File Utility, from there you can Unload your Vision Data to a text file which is tab delimited.
From there you can import to Excel as a tab delimited file and then re-save as a csv file.
Alright, so, I haven't programmed anything useful in ages - last time I did was a year ago and as you can imagine my knowledge of programming is seriously rusty. (last thing I 'programmed' was a ren'py game over the weekend. One can imagine the limited uses of this. The most advanced C program I wrote was a tic-tac-toe game a year ago. So yeah.)
Anyways, I've been given a job to write a program that takes two Excel files, both of which have a list of items, each associated with an ID. I need to write a program to search both files for IDs and if the IDs match, the program will need to create a new file with the matched IDs and items. This is insanely beyond my limited C capabilities.
If anyone could help, I would seriously appreciate it.
(also, if this is not possible with C, I'll do my best to work with any other languages)
Export the two files to .csv format and write a script to process the two files. For example, in PHP, you have built in csv read/write capabilities.
You can do this with VBA and create a Macro in one of the files which iterates over the cells in your column in file 1 and compares them to cells in file 2 and writes them to a new .xls file if they match.
Dana points out that the VLOOKUP function will do this quite easily.
Install GnuWin32
Output the excel files as text (csv for example)
sort each file with the -u option to remove duplicates if needed
mix and sort the 2 files
count unique IDs with uniq -c
filter out lines with a value of 1 for the count with grep
remove the count leaving the ID and whatever else you need with cut
If you know Java then you can use Apache POI for your project. You can use the examples given on the Apache POI website to accomplish your task.
Apache POI Excel Documentation: http://poi.apache.org/spreadsheet/quick-guide.html
If you absolutely have to do this on xls/xlsx file from a process, you probably need a copy of Excel controlled by COM automation. You can do this in BV/VBA/C#/C++ whatever, some easier than others. Google for 'Excel automation'.
Rgds,
Martin
Not C, but you may be able to cobble something together very quickly using xlsperl.
It has come in handy for me in the past.