Why won't redshift accept my fixedwidth text file - sql-server

Question

I am reading a varchar(500) column from a SQL Server 2008 R2 database to import into Redshift via a fixedwidth text file.
To pull down the record into a fixed width file, I started out by using a StringBuilder to write out a block of text at a time. I was using AppendFormat and the alignment specifier to align the different records. At certain points, once every 400k lines, I would write the contents of StringBuilder into a StreamWriter to write to disk.
I noticed that there was an issue with text when I tried loading the files into Redshift, the upload into Reshift failed due to extra columns, (there were more columns than my fixed width specification accommodated for).
When I tested the StringBuilder against a regular string, the widths match what I intended them to match, 500 characters.
The discrepancy came when I tried writing my records to disk. I kept getting the same issue when I wrote the aforementioned database column to disk using WriteLineformat the StreamWriter object.
The collation on the database is SQL_Latin1_General_CP1_CI_AS. I understand that strings from the database get converted the database collation to UTF-16. I think there is no problem there, as stated from the test I performed above. I think the issue I'm having is from taking the string in UTF-16 form and writing them to disk using StreamWriter.
I can expect any type of character from the database field, except for a newline or carriage return. I'm pretty confident that white space is trimmed before being pushed into the database column using a combination of the TSQL functions Ltrim and Rtrim.
Edit: Following is the code I use in Powershell
$dw = new-object System.Data.SqlClient.SqlConnection("<connection string details>")
$dw.open()
$reader = (new-object System.Data.SqlClient.Sqlcommand("select email from emails",$dw)).ExecuteReader()
$writer = new-object system.IO.StreamWriter("C:\Emails.txt",[System.Text.Encoding]::UTF8)
while($reader.read())
{
$writer.writelineformat("{0,-500}",$reader["email"])
}
$writer.close()
$reader.close()
Obviously I'm not going to give you the details of my connection string or my table naming convention.
Edit: I'm including the AWS Redshift article that explains that data can only be imported into Redshift using UTF-8 encoding.
http://docs.aws.amazon.com/redshift/latest/dg/t_preparing-input-data.html
Edit: I was able to get a sample of the outputted file through
get-content -encoding utf8
The content inside of the file is definitely UTF-8 proper. All of the line endings within. It's seems like my main issue is with Redshift taking multi-byte characters for fixed width files.

Answer 1

I suspect that the issue is caused by the fact that StreamWriter by default uses UTF-8 so in some instances you will get double byte characters as utf-8 is variable width.
Try using unicode, which will match your database encoding, StreamWriter has an overload which supports encoding.

Answer 2

Just so that anyone seeing this understands. My problem is really with redshift. One thing I have noticed is that the service seems to have processing issues with fixedwidth files. This seems to be specific to Amazon, since the underlying system that runs Redshift is ParAccel. I had issues in the past with Fixedwidth files. I have been able to confirm that there is an issue with Redshift accepting multi-byte characters within fixedwidth version of the S3 Copy command.