Iterate over all Rows and Columns in DBT - Jinja - loops

Question

I have a table that looks something like this:
id
name
table_name
agg_type
agg_column
condition,unique
1
count_of_courses
stg_courses
count
null
null
2
count_of_bundles
stg_bundles
count
null
null
3
count_of_products
stg_products
count
null
null
4
sum_of_gmv
stg_transactions
sum
revenue_usd
null
5
count_of_course_admins
stg_users
count
user_id
is_course_admin
My goal is to use this table as metadata to feed into a large jinja loop that loops over each table_name and performs different aggregations (count, sum) depending on the metadata from that table above. I'd then union all of the table results together.
It might look something like this:
{% set results =
["count_of_courses", "courses", 'count'],
["count_of_bundles", "bundles", 'count'],
["count_of_products", "products", 'count'],
["sum_of_gmv", "stg_transactions", 'sum', 'revenue_usd'],
["count_of_course_admins", "users", 'count', 'user_id', 'is_course_admin is true', 'distinct']
%}
{% for i in results %}
SELECT
user_id,
{{ i[3] }}( --where i[3] refers to the agg_type (COUNT/SUM)
{%- if i[3] == 'sum' -%}
{{- i[4] -}}
{%- elif i[6] == 'distinct' -%}
distinct {{ i[4] }}
{%- else -%}
*
{%- endif -%}
FROM analytics.{{ i[2] }} -- where i[2] refers to the table name
{% if not loop.last %}
union all
{% endif %}
{% endfor %}
Looking for help on how to deal with agate tables, lists, etc. The source data is coming from a dbt seed (which is essentially a table) so I'll need a way to convert the table into a list of some sort.
I have been successful manually passing a list from the set operator in dbt (as shown above) but I'm not able to convert the dbt sedd/model into this list format (also shown below).
["count_of_courses", "courses", 'count'],
["count_of_bundles", "bundles", 'count'],
["count_of_products", "products", 'count'],
["sum_of_gmv", "stg_transactions", 'sum', 'revenue_usd'],
["count_of_course_admins", "users", 'count', 'user_id', 'is_course_admin is true', 'distinct']
I'd love to know how best I can translate the table (from above) into this list format so I can do all of this dynamically and not have to manually add to a list.
Thanks.

Answer 1

For future readers:
I experimented and researched a bit more and eventually landed on the solution of a generic macro which essentially stores SQL query results in a list. It looks like this:
{# This macro collects data from a query and inserts it into a list #}
{% macro query_to_list(query) %}
{% set query_to_process %}
{{ query }}
{% endset %}
{% set results = run_query(query_to_process) %}
{% if execute %}
{% set results_list = results.rows %}
{% else %}
{% set results_list = [] %}
{% endif %}
{{ return(results_list) }}
{% endmacro %}
Beyond this, I can now call the macro and store the results in a variable, iterate over them, etc. Here's what this might look like. I'm giving a simplified version but essentially, you should be able to parse any of the values from the query/variable and use them in magical ways:
Passing a Query into the Macro
{% set query %}
select id, name, table_name
from {{ ref( 'rollups' ) }}
order by id
{% endset %}
Iterating over the Query Results
{% for i in query_to_list(query) %}
{{ log(dbt_utils.pretty_log_format(i), info=True) }}
select
user_id,
'{{ i[2] }}' as table_name,
'{{ i[1] }}' as name
from {{ref( [i[2]]|join('') ) }} {# Extracts the name of the table. We use 'join' to convert it to a string #}
{# If there is a condition, we extract that here #}
{% if i[6] %}
where {{ i[6] }}
{% endif %}
group by 1, 2, 3, 4
{% if not loop.last %}
union all
{% endif %}
{% endfor %}

Answer 2

That's a smart way to build your models !
Maybe the best function for your use case would be run_query (ref here).
I can't test the code below but it should be something similar:
{% set fetch_operations_query %}
SELECT * FROM your.seed.table
{% endset %}
{% set results = run_query(fetch_operations_query) %}
{% if execute %}
{% set results_list = results.rows %}
{% else %}
{% set results_list = [] %}
{% endif %}
{% for row in result_list %}
SELECT
user_id,
{{ row[3] }}( --where row[3] refers to the agg_type (COUNT/SUM)
{%- if row[3] == 'sum' -%}
{{- row[4] -}}
{%- elif row[6] == 'distinct' -%}
distinct {{ row[4] }}
{%- else -%}
*
{%- endif -%}
FROM analytics.{{ row[2] }} -- where row[2] refers to the table name
{% if not loop.last %}
union all
{% endif %}
{% endfor %}