Group polygons with Postgis (beginner) - postgis

Question

Good morning all, I'm trying to group the polygons that touch each other into one polygon.
I use the following formula:
drop table if exists filtre4;
create table filtre4 as
(
select st_unaryunion(unnest(st_clusterintersecting(geom))) as geom
from data
)
It works perfectly when I have less than 6,000,000 items.
example: I have this normal message that appears, with the number of entities created.
https://zupimages.net/viewer.php?id=20/15/ielc.png
But if I exceed 6,000,000 entities, the query ends but no element is created in the table. I have this message which is displayed, but does not return anything to me.
https://zupimages.net/viewer.php?id=20/15/o41z.png
I do not understand.
Thank you.

Answer 1

So, I think you were using PgAdmin to run the queries. Oddly enough, sometimes even if there is a memory error or other runtime errors you will not be notified. (The same happened to me while testing.) In such a case I would recommend you save the query as a sql file and run it with psql to ensure you receive an error message:
psql -U #your_username -d #your_database -f "#your_sqlfiile.sql"
So first what I would do is adjust "work_mem" in the postgresql.conf. The default is 4MB, given your specs you can probably handle more memory per an operation. I would suggest 64MB to start per the following article:
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
Make sure to restart your server after doing so.
So, I used a comparable data set and received memory issues. Adjusting work_mem was the first part. Geohashing applies to your data set because you want smaller groups of clusters in order to fit the processing in memory, and geohashing allows you to order your geometry elements spatially so that it reduces the amount of sorting operations necessary while running ST_CLUSTERINTERSECTING (I don't think you have attributes to group on from what I understand). Here is what the following example does:
Creates the output table or truncates if it exists, creates a sequence or resets it if it exists
"ordered" Pull geometry from input table, and order by its geohash (*geometry must be in degree units like EPSG 4326 in order to geohash)
"grouped" Use the sequence to put the data in to x amount of groups. I divide by 10,000 here, but the idea is your total number of entities divided by x will give you y groups. Try making groups small enough to fit in memory but large enough to be performant. Then, it takes each group, performs ST_CLUSTERINTERSECTING, unnest, and finally a ST_UNARYUNION.
Insert the value with ST_COLLECT and another ST_UNARYUNION of the geometries.
Here's the code:
DO $$
DECLARE
input_table VARCHAR(50) := 'valid_geom';
input_geometry VARCHAR(50) := 'geom_good';
output_table VARCHAR(50) := 'unary_output';
sequence_name VARCHAR(50) := 'bseq';
BEGIN
IF NOT EXISTS (SELECT 0 FROM pg_class where relname = format('%s', output_table))
THEN
EXECUTE '
CREATE TABLE ' || quote_ident(output_table) || '(
geom geometry NOT NULL)';
ELSE
EXECUTE '
TRUNCATE TABLE ' || quote_ident(output_table);
END IF;
IF EXISTS (SELECT 0 FROM pg_class where relname = format('%s', sequence_name))
THEN
EXECUTE '
ALTER SEQUENCE ' || quote_ident(sequence_name) || ' RESTART';
ELSE
EXECUTE '
CREATE SEQUENCE ' || quote_ident(sequence_name);
END IF;
EXECUTE '
WITH ordered AS (
SELECT ' || quote_ident(input_geometry) || ' as geom
FROM ' || quote_ident(input_table) || '
ORDER BY ST_GeoHash(geom_good)
),
grouped AS (
SELECT nextval(' || quote_literal(sequence_name) || ') / 10000 AS id,
ST_UNARYUNION(unnest(ST_CLUSTERINTERSECTING(geom))) AS geom
FROM ordered
GROUP BY id
)
INSERT INTO ' || quote_ident(output_table) || '
SELECT ST_UNARYUNION(ST_COLLECT(geom)) as geom FROM grouped';
END;
$$;
Caveats:
Change the declare variables to your needs.
Since your input geometry is named 'geom' as geom will fail so I would change SELECT ' || quote_ident(input_geometry) || ' as geom to SELECT ' || quote_ident(input_geometry).
Make sure all of your input geometries are valid or ST_UNARYUNION will fail. Checkout ST_ISVALID and ST_MAKEVALID.
As said before geohashing requires the projection to be in degree units. Checkout ST_TRANSFORM, (I transformed my geometry data to 4326).
Let me know if you have any more questions.