The fastest way to copy a huge dataset to server in SAS - sql-server

Question

Edited on Oct 21th
Background
Raw datasets (changed everday) stored on a MS-SQL based server: Sourcelib.RAW and a LOCAL excel file (remained unchanged).
I need to refresh a dataset WANTlocates in Targerlib. Currently I have SQL codes perform such task in 2-3 minutes. But I want to know if SAS can do the same thing while the processing time won't increase much.
work.IMP is around 6M records and around 50 bytes per record.
The ideal method should be very efficient because as time goes by, the size of raw datasets on the server would be incrediblely huge.
The target file CANNOT be establied at one time and then append new data to it everyday. Because there maybe (even very unlikely) changes in previous data.
As per #Joe, I should allow the target file to just be updated, using proc compare, or update in data step. Here's a related question I posted How to use proc compare to update dataset
Still more than 10GB free space on the sever, which is enough. Avaiable memory in my pc is around 3.5GB (not sure if it matters)
Due to the architecture of server, it's very efficient to do it in MS-SQL. But I REALLY want to know if SAS can deal with this (when the server is not so "compatible")
Process
First I import the data from excel file and then subset&tranpose it to be work.IMP. For some reasons, this file can only be created in this way everday. It CANNOT be store in the server.
Then perform outer join for work.IMP and one raw dataset Sourcelib.RAW1 to get work.HAVE1. Please note that work.IMP is sorted but Sourcelib.RAW1 is unsorted. The outer join is only to (with some criteria) determine each data records.
i.e. case when a.COL1 is '' then b.COL1 else a.COL1 end as COL1
You can consider this process is to adjust Sourcelib.RAW1 by using work.IMP.
PS1: #sparc_spread suggests to do the import procedure directly to the server. But it won't have any benefit than do it in LOCAL. And a hash object here doesn't help either.
Then I subset another raw datasets Sourcelib.RAW2 to work.temp, and then sort it to be work.HAVE2. (The data in Sourcelib.RAW2 is mostly not in order.)
I concatenate work.HAVE1, work.HAVE2 by using proc append (because both two tables are huge) to be work.HAVE
PS2: The sorting in step3 is to avoid sorting at the end of step4. Actually the data Targerlib.WANT doesn't have to be in order. But it's better to be so.
At the very end, I copy work.HAVE to server Targetlib.HAVE.
I did most of the thing in WORK, which only took me few minutes. But step5 could take me half an hour to finish the copy.
As per #Joe, this may mainly due to something related to network transit. I.E minmize the network transit
Question
Any way to improve step5? Or any modification of the whole process will improve the performance?

Answer 1

Couple of thoughts.
First off, assuming this is a SAS dataset and not a SQL database or something else, options compress=binary; is a good idea assuming this is mostly numeric (and options compress=character; if not). Either will reduce the physical size of the dataset significantly in most cases.
Second, 300MB is not very much in the scheme of things. My network would write that in less than a minute. What the conditions of your network are may drive some of the other choices you make; if the single slow point is simply copying data across it, for example, then you need to figure out how to reduce that at the cost of anything else you do.
Assuming you don't change anything else, I would recommend writing have1 directly to the network as have, and then append have2 to it. IE, whatever step creates have1, have that directly write to the network. This includes sort steps, note: so if you create it then sort it, create it locally and sort it with out= the network library. This reduces the total amount of writing done (as you don't write a useless copy of have1 to your local drive). This helps if writing locally is a relevant cost to your total process, but won't help if it's almost entirely network congestion.
Copying files with the OS's copy is almost always superior to any other method of copying, so if network congestion is the only factor you care about, you may want to make it locally (in WORK or in a local but constant directory, like C:\temp\ or similar) and then have the last step of your process be executing copy c:\temp\have.sas7bdat \\networklocation\whereitgoes\. This will usually outperform SAS methods for same, as it can take advantage of efficient techniques.
PROC COPY is another option to get around network congestion; it's probably faster than PROC APPEND (if the local write-out is negligible, as it would be for me for <1 GB data), and has the advantage that it's a bit safer in case something happens during the transport. (Append should be fine also, but with COPY you know for sure nothing was changed from yesterday's file.)
Finally, you may want to figure out some way to allow the target file to just be updated. This isn't all that hard to do in most cases. One example would be to keep a copy of yesterday's file, do a PROC COMPARE to today's file, and then include in the update file every record that is changed (regardless of what the change is). Then delete any matching old records from the target file and append the new records. This is very fast comparatively in terms of total records sent over the network, so saves a lot of time overall if network congestion is the main issue (but takes more CPU time to do the PROC COMPARE).

Answer 2

You can use PROC APPEND to efficiently create a new dataset, not just append to an existing - thus you can use that to basically combines steps 3 and 4 into this:
/* To fulfill requirement 4, delete existing Targetlib.HAVE */
PROC DELETE LIBRARY="Targetlib" DATA="HAVE";
RUN;
/* Targetlib.HAVE does not yet exist, the first APPEND will create it */
PROC APPEND BASE="Targetlib.HAVE" DATA="work.HAVE1";
RUN;
PROC APPEND BASE="Targetlib.HAVE" DATA="work.HAVE2";
RUN;
That should save at least some time, but that still doesn't solve all your problems... I have some additional questions I have put in comments to the question and will alter this answer as much as I can based on them.
Update 1
Here is a way to do the left join and the concatenation in one step and write the results immediately to targetlib. I cannot guarantee this will be faster but it is worth a try. I used key and val as field names, replace as you see fit.
PROC SQL _METHOD;
CREATE TABLE targetlib.HAVE
AS
SELECT
a.key ,
CASE WHEN MISSING (a.val) THEN b.val ELSE a.val END AS val
FROM
Sourcelib.RAW1 AS a
LEFT JOIN
IMP AS b
ON
a.key = b.key
UNION
SELECT
c.*
FROM
Sourcelib.RAW2 AS c
ORDER BY
key
;
QUIT;
RUN;
The _METHOD is a sparsely documented SAS feature that will print the query plan, see this link. This may give you more insight. Also, I am assuming that IMP was already imported from the Excel, and that it is in WORK. Experiment to see whether importing it to targetlib and replacing IMP as b with targetlib.IMP as b is any faster here.
Since you are on Windows, experiment with the data option SGIO=YES after dataset names: e.g. Sourcelib.RAW1 AS a becomes Sourcelib.RAW1 (SGIO=YES) AS a. For more info on Windows SGIO and SAS, see this link and this older but more comprehensive one.
An approach that might be more efficient would be to avoid the join and use a hash object instead: good documentation for hash object can be found here and a good tip sheet is here. It's not clear whether this would be faster - imp has 6m records, but at 50 bytes per record, that's about 300 MB, which does fit into your RAM. The performance of a hash table with that many entries would depend a lot on SAS's hash algorithm. Anyway, here is the code using a hash object. In it, we assume that in the IMP dataset, the val field has been renamed to val2.
DATA targetlib.HAVE (DROP = rc val2);
LENGTH val2 8. ;
IF (_N_ = 1) THEN DO;
DECLARE HASH h (DATASET: "IMP") ;
h.DEFINEKEY ('key');
h.DEFINEDATA ('val2');
h.DEFINEDONE ();
END;
SET
sourcelib.RAW1
sourcelib.RAW2
;
IF MISSING (val) THEN DO;
rc = h.find();
IF (rc = 0) THEN DO;
val = val2;
END;
END;
RUN;
PROC SORT DATA = targetlib.HAVE ; BY KEY ; RUN ;
Try it and see if it is any faster. Once again, experiment with the location of IMP, using SGIO, etc. That PROC SORT at the end could be expensive; if the only reason you were sorting before was because of joining, then skip it.
In general, your approach to SAS should be to do as little I/O as possible, and find ways of combining multiple actions into a single write from a PROC or DATA step.