Performance issues with CellTemplate of GridViewColumn - wpf

Question

I am using a ListView with it's View set to GridView to display items in dynamically generated columns. The ListView's virtualization is enabled. For this example I want to display up to 10 rows with 50 columns (but the count of columns is dynamic).
The first thing I do is to create the columns in code behind in a loop.
"Items" is just an ObservableCollection of strings in my view model for one row.
int index = 0;
foreach(Header header in Headers)
{
GridViewColumn column = new GridViewColumn();
column.Header = // the view model of the header;
column.HeaderTemplate = Resources["HeaderTemplate"] as DataTemplate;
column.DisplayMemberBinding = new Binding("Items" + "[" + index + "]");
m_MyGridView.Columns.Add(column);
index++;
}
Now, when I change the ItemsSource of my ListView, the data gets updated quiet fast.
Unfortunately, each string in every cell has to be formatted (font color, foreground etc). So I cannot use DisplayMemberBinding, but CellTemplate to assign a DataTemplate to every cell.
For this purpose I create a DataTemplate in xaml in my control that holds the ListView. This contains just a TextBlock with some bindings to properties of the cell's view model to define the string style of the TextBlock.
Now, Items is not just a collection of string but a collection of CellViewModels that not only contain the value to display but also information of how to display it. This information is then bound inside the DataTemplate.
The CellTemplate is set up in code and assigned to each GridViewColumn's CellTemplate:
foreach(Header header in Headers)
{
GridViewColumn column = new GridViewColumn();
column.Header = // the view model of the header;
column.HeaderTemplate = Resources["HeaderTemplate"] as DataTemplate;
column.CellTemplate = createCellTemplate(index);
m_MyGridView.Columns.Add(column);
index++;
}
DataTemplate getCellTemplate(int index)
{
FrameworkElementFactory elementFactory = new FrameworkElementFactory (typeof (ContentControl));
elementFactory.SetBinding(ContentControl.ContentProperty, new Binding("Items" + "[" + index + "]"));
DataTemplate dt = Resources["ValueTemplate"] as DataTemplate;
elementFactory.SetValue(ContentControl.ContentTemplateProperty, dt);
DataTemplate dataTemplate = new DataTemplate ();
dataTemplate.VisualTree = elementFactory;
return dataTemplate;
}
When I do this, things get really slow when I'm changing the ItemsSource of the ListView. It now takes nearly a second to render the new cells each time I change the ItemsSource to display different data.
So is there a way to speed things up? I thought about having the formatted string already in my CellViewModel by using a Paragraph, but unfortunately DisplayMemberBinding only supports string, so I have to use the CellTemplate.
Edit: so, here is the listview i'm using:
<ListView ItemsSource="{Binding ElementName=Control, Path=ItemViewModels}">
<ListView.ItemsPanel>
<ItemsPanelTemplate>
<VirtualizingStackPanel IsItemsHost="True" VirtualizingStackPanel.IsVirtualizing="True" />
</ItemsPanelTemplate>
</ListView.ItemsPanel>
<ListView.View>
<GridView x:Name="m_MyGridView"/>
</ListView.View>
</ListView>
and the DataTemplate for the cells:
<DataTemplate x:Key="ValueTemplate">
<Border Background="{Binding Path=BackgroundColor">
<TextBlock Text="{Binding Path=Value}"
Foreground="{Binding Path=ForegroundColor}"
FontStyle="{Binding Path=FontStyle}"
FontWeight="{Binding Path=FontWeight}"
</TextBlock>
</Border>
</DataTemplate>
The function getCellTemplate is called only once when the headers are created. When the ItemsSource changes, it is not called, because they have been created already.