How to self join a CTE? - sql-server

Question

Hi This is the query i have created to fetch records from a history table This query is working fine but taking too much time to execute because it is selecting data twice from the table and the table has 20 millon or above records. So i want to optimize it. I will try to explain what this query does and what i want to achieve.
I want to select two rows for each Id (ActivityId) from the table first is where the data is min and second where the date is maximum, So i can see how much progress happened in the mean time. Now what i am doing is first selecting the data where date is min as CTE and then selecting the data where date is maximum as CTE2. I can select both rows in a single CTE but I am not able to get a single record from both rows. Like i only need the Progress and Planned Field from the maximum date row and select all the other fields (most fields are common on both rows). So how can i achieve this. and please ask for clarification if needed.
CTE:
WITH cte AS
(
SELECT Row_number() OVER (partition BY pmaph.activityid ORDER BY date) r1,
pma.planenddate AS planenddate ,
pma.planstartdate AS planstartdate,
pmaph.activityid,
pmaph.projectmilestoneactivproghist_id AS promileavtiid,
umd.uom_name AS uomname,
pmm.milestonename AS milestonename,
pma.activityname AS activityname ,
pmp.projectname AS projectname,
Replace((Rtrim(Ltrim(CONVERT(VARCHAR(12),Cast(pmaph.date AS DATETIME),106)))),' ','-') AS rdate,
Isnull(pmaph.actual_progress,0) AS actualprogress,
Isnull(pmaph.planned_progress,0) AS plannedprogress
FROM projectmilestoneactivityprogresshistory AS pmaph
LEFT JOIN dbo.pm_project AS pmp
ON pmaph.projectid=pmp.projectid
LEFT JOIN dbo.pm_activity AS pma
ON pmaph.activityid=pma.activityid
LEFT JOIN dbo.pm_milestone AS pmm
ON pmaph.milestoneid=pmm.milestoneid
LEFT JOIN dbo.uomdetail AS umd
ON pma.uom_id=umd.uom_id
WHERE pmaph.client_id=1030), cte2 AS( r2,isnull(pmaph.actual_progress,0) AS actualprogress, isnull(pmaph.planned_progress,0) AS plannedprogress, replace((rtrim(ltrim(CONVERT(varchar(12), cast(pmaph.date AS datetime),106)))),' ','-') AS rdate,pmaph.activityid FROM projectmilestoneactivityprogresshistory AS pmaph WHERE pmaph.client_id=1030)
SELECT cte2.rdate AS todate,
cte2.actualprogress AS end_actualprogress,
cte2.plannedprogress AS end_plannedprogress,
cte.actualprogress AS start_actualprogress,
cte.plannedprogress AS start_plannedprogress,
cte.rdate AS fromdate ,
cte.planenddate,
cte.planstartdate,
cte.activityid,
cte.uomname,
cte.milestonename,
cte.activityname,
cte.projectname
FROM cte
JOIN cte2
ON cte.activityid= cte2.activityid
WHERE cte.r1=1
AND cte2.r2=1