CTAS operation in netezza consuming /nzscratch - netezza

Question

When I create a table via CTAS(Create Table AS)with the help of 1 or more base tables which contains million of records. Then I can see transient data is getting saved to /nzscratch/tmp dir Or I can say during the CTAS operation, nzscratch/tmp is keep on filling unless it gets completed. Once CTAS is successful, everything gets cleared out from /nzscratch/tmp dir.
So I would like to know if it is expected behavior or not?
2nd thing: Does the same concept applies while running any normal query also which keeps on running for long time or transient data is getting save to memory side in this case?
An example of a CTAS that is generating transient data:
CREATE TABLE T1 AS
SELECT track_seq,
MAX(campaign_label) AS campaign_label,
MAX(creative_label) AS creative_label,
MAX(lob_label) AS lob_label,
MAX(placement_label) AS placement_label,
MAX(site_label) AS site_label
FROM (
SELECT *
FROM master_test
UNION
SELECT *
FROM labels_test_1
)
a
GROUP BY 1;
The relevant DDL:
CREATE TABLE admin.master_test
(
track_seq character varying(40),
campaign_label character varying(200),
creative_label character varying(200),
lob_label character varying(200),
placement_label character varying(200),
site_label character varying(200)
)
DISTRIBUTE ON (track_seq)
;
CREATE TABLE admin.labels_test_1
(
track_seq character varying(40),
campaign_label character varying(200),
creative_label character varying(200),
lob_label character varying(200),
placement_label character varying(200),
site_label character varying(200)
)
DISTRIBUTE ON (track_seq)
;

Answer 1

Generally speaking, you should only see transient data in the /nzscratch/tmp directory if your CTAS query does significant host-based processing (e.g. using a row_number() function with no PARTITION BY clause, which is an all too common mistake for generating surrogate keys on an MPP platform, in my opinion).
You may also see usage there in cases where you are doing a SELECT to a remote system (your desktop, or a BI server) where the receiving system can't keep up with the speed the Netezza system sends the data. In these cases you will see data spooled in /nzscratch as well.
For the specific CTAS example you provided, the culprit is the UNION in the subselect. A UNION must provide duplicate detection, and what is happening in this case is that both tables are being sent up to the host to be processed so that all the rows can be de-duplicated there. This is what is eating up your /nzscratch space.
Here is an alternative using GROUP BY to do the de-deuplication work that should avoid that host-based processing, and keep the work on the MPP backend, by specifying the distribution column in the GROUP BY.
CREATE TABLE T1 AS
SELECT track_seq,
MAX(campaign_label) AS campaign_label,
MAX(creative_label) AS creative_label,
MAX(lob_label) AS lob_label,
MAX(placement_label) AS placement_label,
MAX(site_label) AS site_label
FROM (
select *
from (
SELECT *
FROM master_test
UNION ALL
SELECT *
FROM labels_test_1
)
foo
group by 1,2,3,4,5,6
)
a
GROUP BY 1;