MalformedStreamException when posting from Tcl script - tomcat6

Question

I've made a servlet which uses the org.apache.commons.fileupload api to upload a CSV file which it should then load into a MySQL table. This works perfectly when posting to the servlet from a form in a browser. However, it fails when trying to post a form via a Tcl script http://www.tcl.tk/man/tcl/TclCmd/http.htm#M20.
The servlet throws the following MalformedStreamException:
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
The servlet is hosted on Tomcat version 6.0.16.
The connection is made successfully by the Tcl script as it receives an HTTP/1.1 200 OK response and the servlet does return some print statements back to the client Tcl script. However it fails when trying to read the input stream.
Tcl script:
package require http
proc upload { url csv } {
set boundary "-----WebKitFormBoundary[clock seconds][pid]"
set fid [open $csv r]
if {[catch {read $fid [file size $csv]} data]} {
return -code error $data
}
close $fid
set content {}
append content "--${boundary}\n"
append content "Content-Disposition: form-data; name=\"db\"\n\n"
append content "test\n"
append content "--${boundary}\n"
append content "Content-Disposition: form-data; name=\"table\"\n\n"
append content "testing\n"
append content "--${boundary}\n"
append content "Content-Disposition: form-data; name=\"file\"; filename=\"$csv\"\n"
append content "Content-Type: text/csv\n\n"
append content "$data\n"
append content "${boundary}--"
set headers "connection keep-alive"
set token [::http::geturl $url -keepalive 1 -headers $headers -query $content -type "multipart/form-data; boundary=$boundary"]
upvar 0 $token state
if {$state(http) == "HTTP/1.1 200 OK"} {
# no error reported in http headers
puts stdout $state(http)
puts stdout $state(body)
return 1
} else {
# error reported in http headers
puts stdout $state(http)
puts stdout $state(body)
return 0
}
}
set csv "data.csv"
set url "http://ecat:8080/MySqlImport/MySqlImportServlet"
set retVal [upload $url $csv]

Answer 1

Generating a proper multipart message can be highly aggravating. The mime package in Tcllib can help. The top example on this page looks highly relevant.

Answer 2

While Donal Fellows most probably got it right, I want to point out one (typical) mistake with the Daniel's approach: when one uses a high-level language which has the concept of a "string of characters"—as opposed to a "string of bytes"—one is oblidged to take special care when preparing the data being processed to be placed on "the wire".
In this case "the wire" is the HTTP protocol and so the data prepared should most probably go through something like [encoding convertto utf-8 ...] or something like this depending on the intent. (By the way, Tcl has rather extensive support for binary strings so that they can be appended to, concatenated etc without losing their property of being binary.)
The next thing which can be recommended is to get accustomed with a network sniffer (or a specific HTTP sniffer). A great cross-platform solution is Wireshark, on Windows platforms there's Network Monitor (a generic sniffer) and Fiddler (HTTP-specific). The idea is to first capture and study closely the "reference" session performed by a browser then to capture and analyze the misbehaving session performed by your program. So you just compare the sessions and see where the generated traffic differ when it comes to HTTP payload.

Answer 3

Thank you for the help on this one but I have found my specific issue and it was a simple one in the end. It seems as though the ServletFileUpload.parseRequest(request) method in apache.commons.fileupload requires the line endings to be in windows format.
As you can see in the question the $content line endings was \n changing this to \r\n corrected the problem.
Using just \r meant that the error
org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
Stream ended unexpectedly
no longer occured, however it failed to recognise the different items or parts in the request. The combination of \r\n line endings and it works correctly.