I have been using SQL for like 10 years but now I realised I never knew how the client actually receive and process data it gets from the server.
My question is, how does the result from Microsoft SQL Server actually look like in raw format? The same as that result from HTTP server contains HTTP headers and a Content-Type header to tell what the body format is (mostly HTML for web pages).
The protocol name is TDS (Tabular Data Stream).
Some documentation is available at MSDN.
There is a very basic example of data being transferred for simple query select 'foo' as 'bar'
Request
Packet header (type, legth, etc)
01 01 00 5C 00 00 01 00
Packet data
16 00 00 00 - headers total length
12 00 00 00 - first header length
02 00 - type
00 00 00 00 00 00 00 01 00 00 00 00 - data
0A 00 73 00 65 00 6C 00 65 00 63 00 74 00 20 00
27 00 66 00 6F 00 6F 00 27 00 20 00 61 00 73 00
20 00 27 00 62 00 61 00 72 00 27 00 0A 00 20 00
20 00 20 00 20 00 20 00 20 00 20 00 20 00 - sql
Response
Packet header (type, legth, etc)
04 01 00 33 00 00 01 00
Packet data
columns metadata
81 - record id
01 - count
first column
00 00 00 00 00 - user type
20 00 - flags
A7 - type
03 00 - length
09 04 D0 00 34 - colation
03 - column name length
62 00 61 00 72 00 - column name bytes
rows
D1 - record id
03 00 - length
66 6F 6F - value
ending data
FD - record id
10 00 - status
C1 00
01 00 00 00 00 00 00 00 - rows total
We can also look at parser implementation thanks to reference sources.
Related
I call a stored procedure using JDBC:
Connection con = DriverManager.getConnection("jdbc:sqlserver://myhost;databaseName=mydb;encrypt=false","user", "pass");
con.setAutoCommit(false);
CallableStatement cs = con.prepareCall("{call abc(?,?,?,?,?,?,?)}");
cs.setDate(1, java.sql.Date.valueOf(LocalDate.of(2023, 1, 10)));
cs.setDate(2, java.sql.Date.valueOf(LocalDate.of(2023, 1, 9)));
cs.setString(3, "PROD");
cs.registerOutParameter(4, Types.NUMERIC);
cs.registerOutParameter(5, Types.NUMERIC);
cs.registerOutParameter(6, Types.NUMERIC);
cs.registerOutParameter(7, Types.NUMERIC);
cs.executeQuery();
I get a SQLServerException:
2023-01-13T11:22:48.975-06:00 DEBUG 21888 --- [restartedMain]
c.m.s.jdbc.internals.SQLServerException : ***
SQLException:SQLServerCallableStatement:5
com.microsoft.sqlserver.jdbc.SQLServerException: The statement did not
return a result set. The statement did not return a result set.
But I get no error calling it with Hibernate:
StoredProcedureQuery query = em.createStoredProcedureQuery("abc")
.registerStoredProcedureParameter(
"date1",
LocalDate.class,
ParameterMode.IN
)
.registerStoredProcedureParameter(
"date2",
LocalDate.class,
ParameterMode.IN
)
.registerStoredProcedureParameter(
"name",
String.class,
ParameterMode.IN
)
.registerStoredProcedureParameter(
"value1",
BigDecimal.class,
ParameterMode.OUT
)
.registerStoredProcedureParameter(
"value2",
BigDecimal.class,
ParameterMode.OUT
)
.registerStoredProcedureParameter(
"value3",
BigDecimal.class,
ParameterMode.OUT
)
.registerStoredProcedureParameter(
"value4",
BigDecimal.class,
ParameterMode.OUT
)
.setParameter("date1", LocalDate.of(2023, 1, 10))
.setParameter("date2", LocalDate.of(2023, 1, 9))
.setParameter("name", "PROD");
query.execute();
log.info("value2 is {}", query.getOutputParameterValue("value2"));
What am I doing wrong with my JDBC call?
I looked at the TRACE log from the SQL Server driver. The only difference in bytes going going to SQL Server is ^ and H in the first line:
JDBC
03 01 02 26 00 5E 01 00 16 00 00 00 12 00 00 00 ...&.^..........
02 00 00 00 00 00 00 00 00 00 01 00 00 00 FF FF ................
0A 00 00 00 00 00 E7 40 1F 09 04 D0 00 34 96 00 .......#.....4..
45 00 58 00 45 00 43 00 20 00 74 00 65 00 6D 00 E.X.E.C. .a.b.c.
Hibernate
03 01 02 26 26 00 48 01 00 16 00 00 12 00 00 00 ...&.H..........
02 00 00 00 00 00 00 00 00 00 01 00 00 00 FF FF ................
0A 00 00 00 00 00 E7 40 1F 09 04 D0 00 34 96 00 .......#.....4..
45 00 58 00 45 00 43 00 20 00 74 00 65 00 6D 00 E.X.E.C. .a.b.c.
The Hibernate code calls execute() which expects no result set. However, the JDBC code is calling executeQuery() and that expects a result set. Since the stored procedure doesn't return a result set, the SQL Server exception is thrown.
I changed the JDBC code to execute(), and it works the same as the Hibernate code.
From byte buffer received server, I want to copy struct.
The format of buffer is fixed size bytes as below.
00000000 83 27 48 12 6c 00 00 00 01 02 00 00 01 01 00 02 |.'H.l...........|
00000010 10 01 d2 02 96 49 00 00 00 00 87 d6 12 00 00 00 |.....I..........|
00000020 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 02 01 02 3c 01 01 00 00 00 01 01 01 01 18 10 |....<...........|
00000040 2c 01 90 01 01 6c 07 03 c8 02 01 02 03 9c 0a 0b |,....l..........|
00000050 0c 00 00 00 01 01 00 00 00 00 00 00 00 01 01 01 |................|
00000060 01 01 01 01 01 01 01 01 01 00 01 01 01 00 00 00 |................|
My struct is below.
type HeaderT struct {
magicValue [8]byte
bodyLength [4]byte
bodyVersion [1]byte
...
}
My implementation is at below.
func onMessageReceived(client MQTT.Client, message MQTT.Message) {
payload := message.Payload()
fmt.Printf("Received message on topic: %s\nMessage: \n%s\n", message.Topic(), hex.Dump(payload))
header := HeaderT {}
err := binary.Read(bytes.NewBuffer(payload[:]), binary.LittleEndian, &header) // <-- error occurred at this line
...
}
My code make panic as below.
panic: reflect: reflect.Value.SetUint using value obtained using
unexported field
goroutine 38 [running]: reflect.flag.mustBeAssignable(0x1a8)
/usr/local/go/src/reflect/value.go:231 +0x1ee reflect.Value.SetUint(0x12540e0, 0xc0001a2000, 0x1a8, 0x83)
/usr/local/go/src/reflect/value.go:1551 +0x2f encoding/binary.(*decoder).value(0xc000148d88, 0x12540e0,
0xc0001a2000, 0x1a8)
/usr/local/go/src/encoding/binary/binary.go:548 +0x7c6 encoding/binary.(*decoder).value(0xc000148d88, 0x125cfc0,
0xc0001a2000, 0x1b1)
/usr/local/go/src/encoding/binary/binary.go:510 +0x104 encoding/binary.(*decoder).value(0xc000148d88, 0x129fa00,
0xc0001a2000, 0x199)
/usr/local/go/src/encoding/binary/binary.go:523 +0x2c5 encoding/binary.Read(0x12fcf80, 0xc00018a150, 0x1300c60, 0x14d76d0,
0x1248040, 0xc0001a2000, 0x0, 0x0)
/usr/local/go/src/encoding/binary/binary.go:248 +0x342 main.onMessageReceived(0x13012a0, 0xc000140000, 0x1300c00,
0xc000192000)
The issue is that none of HeaderT's fields are "public".
Notice that all the fields start with a lowercase letter - that means the fields are unreachable to any code outside of your package.
From the spec:
Exported identifiers
An identifier may be exported to permit access to it from another package. An identifier is exported if both:
the first character of the identifier's name is a Unicode upper case letter (Unicode class "Lu"); and
the identifier is declared in the package block or it is a field name or method name.
All other identifiers are not exported.
Try Exporting them by capitalizing their names:
type HeaderT struct {
MagicValue [8]byte
BodyLength [4]byte
BodyVersion [1]byte
...
}
I've got an image file of a hard drive from a client that wants a database extracted out of it. The client does not know any details except that the database was once installed on the server from which the image was created.
I found out that it is a UNIX system with a Informix DBS installed, but I am unable to find any database files. I'm not sure about the version of Informix, but it seem that it was installed around 15 years ago.
I'm not able to boot from the image. I'm just viewing the files.
Do informix database files have an extension and what could it be? Any other tips how to identify the database files?
Do you know whether the database was Informix Standard Engine (SE) or Informix (Informix Dynamic Server — IDS — or one of its multitude of namesakes over the years)?
Standard Engine
If it was SE, then the database files are in a directory database.dbs and the files holding the indexes and data for the tables have extensions .idx and .dat. That's pretty much fixed and very easy. The files and the directory should belong to group informix; the owner will be whoever created the database or the table within the database.
Informix Dynamic Server
If it was IDS, then there is no guaranteed naming convention, and Informix didn't even recommend one. Depending on the state of the disk, I'd look for large files that are owned by user informix and belong to group informix and have 660 (-rw-rw----) permissions. The files will have structure, but it isn't easy to discern it.
For example, I have 'chunk 0 of the root dbspace' in a file toru_31.rootdbs.c0 (server name toru_31 — a naming standard I imposed on my systems). It starts with:
0x0000: 00 00 00 00 01 00 AB 89 03 00 00 18 30 01 C0 06 ............0...
0x0010: 00 00 00 00 00 00 00 00 49 42 4D 20 49 6E 66 6F ........IBM Info
0x0020: 72 6D 69 78 20 44 79 6E 61 6D 69 63 20 53 65 72 rmix Dynamic Ser
0x0030: 76 65 72 20 43 6F 70 79 72 69 67 68 74 20 32 30 ver Copyright 20
0x0040: 30 31 2C 20 32 30 31 31 20 20 49 42 4D 20 43 6F 01, 2011 IBM Co
0x0050: 72 70 6F 72 61 74 69 6F 6E 2E 00 00 00 00 00 00 rporation.......
0x0060: 00 00 00 00 00 00 00 00 00 00 03 00 00 08 00 00 ................
0x0070: BA 2D CC 50 1A 00 00 00 00 00 00 00 C8 00 00 00 .-.P............
0x0080: 31 31 37 33 05 00 00 00 00 00 00 00 00 00 00 00 1173............
0x0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
However, 'chunk 1 of the root dbspace' (file toru_31.rootdbs.c1) starts:
0x0000: 00 00 00 00 02 00 F1 C6 00 00 00 18 18 00 E4 07 ................
0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
* (125)
0x07F0: 00 00 00 00 00 00 00 00 00 00 00 00 61 C6 92 00 ............a...
0x0800: 69 6E 66 6F 72 6D 69 78 69 6E 66 6F 72 6D 69 78 informixinformix
* (127)
0x1000: 02 00 00 00 02 00 BE B6 06 00 08 08 20 00 D8 07 ............ ...
0x1010: 00 00 00 00 00 00 00 00 10 06 00 00 01 00 00 00 ................
0x1020: EA 06 00 00 03 00 00 00 75 08 00 00 08 00 00 00 ........u.......
0x1030: A1 08 00 00 08 00 00 00 B5 0A 00 00 30 03 00 00 ............0...
There is very little information there to give the game away. Although the appearance of informix (128 lines containing informix twice) looks like a tell-tale sign, it isn't — I created the chunk with a program that writes informix over the disk space. It simply shows where the server has not yet written data to this chunk.
You can look for $INFORMIXDIR. In fact, if there's a directory /INFORMIXTMP, you can look in the text file /INFORMIXTMP/.infxdirs to see where Informix has been installed. From those directories, you can look in $INFORMIXDIR/etc for onconfig files - they're text files that contain an entry ROOTPATH, which is the pathname of chunk 0 of the root dbspace — basically, the starting point for the whole system. There's usually an onconfig.std which is a template; the naming convention is not firmly fixed, though I always use onconfig.servername (so the config file for toru_31 is onconfig.toru_31). You may also find other files around in $INFORMIXDIR/etc. For example, there's a file oncfg_toru_31.31 (prefix oncfg_, followed by server name toru_31 followed by dot and the server number 31) which contains information about the chunks and other disk space etc used by the server. You might also see binary files analogous to .conf.toru_31 and .infos.toru_31 — the .infos file is normally present only while the server is up but the .conf file persists. These files have some limited information in them, most notably the name of the onconfig file.
If you can find these files on the disk, then you can proceed to identify where the data was stored on the disk.
I'm trying to connect to SQL Server with Windows Authentication. Microsoft C# and C sources can be found at NetCpp.
In Delphi I have code like this:
function TTDS7SSPI.MakeSPN: string;
const
szBracketedInstanceFormatString = '%s/[%s]:%s';
szBracketedEmptyInstanceFormatString = '%s/[%s]%s';
szClearInstanceFormatString = '%s/%s:%s';
szClearEmptyInstanceFormatString = '%s/%s%s';
szBracketedFormatString = '%s/[%s]:%d';
szClearFormatString = '%s/%s:%d';
var
NeedBrackets: Boolean;
FmtString: string;
begin
NeedBrackets := Pos(':', FHostName) > 0;
if FInstanceName <> '' then begin
// Make an instance name based SPN, i.e. MSSQLSvc/FQDN:instancename
if NeedBrackets then begin
if FInstanceName = '' then
FmtString := szBracketedEmptyInstanceFormatString
else
FmtString := szBracketedInstanceFormatString;
end
else begin
if FInstanceName = '' then
FmtString := szClearEmptyInstanceFormatString
else
FmtString := szClearInstanceFormatString;
end;
Result := Format(FmtString, [SQL_SERVICECLASS, FHostName, FInstanceName]);
end
else begin
// Make a TCP port based SPN, i.e. MSSQLSvc/FQDN:TcpPort
Assert(FPort > 0);
if NeedBrackets then
FmtString := szBracketedFormatString
else
FmtString := szClearFormatString;
Result := Format(FmtString, [SQL_SERVICECLASS, FHostName, FPort]);
end;
end;
function TTDS7SSPI.GetAuth: TBytes;
var
pkgInfo: PSecPkgInfo;
SecBuf: SecBuffer;
BuffDesc: SecBufferDesc;
status: SECURITY_STATUS;
attrs: Cardinal;
tsExpiry: TTimeStamp;
const
NEG_STR: WideString = 'Negotiate'; // 'NTLM'; // 'Kerberos';
begin
Result := nil;
status := QuerySecurityPackageInfo({$IFDEF FPC}PSecChar{$ELSE}PSecWChar{$ENDIF}(NEG_STR), pkgInfo);
if status <> SEC_E_OK then
raise Exception.CreateFmt('Couldn''t query package info for %s, error %X', [NEG_STR, status]);
FMaxMessageLen := pkgInfo.cbMaxToken; // 4096;
FreeContextBuffer(pkgInfo);
TTimeStamp(tsExpiry).QuadPart := 0;
status := AcquireCredentialsHandle(nil, {$IFDEF FPC}PSecChar{$ELSE}PSecWChar{$ENDIF}(NEG_STR), SECPKG_CRED_BOTH, // SECPKG_CRED_OUTBOUND
nil, nil, nil, nil, #FCred, tsExpiry); // tsExpiry as var parameter
if status <> SEC_E_OK then
raise Exception.CreateFmt('AcquireCredentialsHandle error %X', [status]);
BuffDesc.ulVersion := SECBUFFER_VERSION;
BuffDesc.cBuffers := 1;
BuffDesc.pBuffers := #SecBuf;
SecBuf.BufferType := SECBUFFER_TOKEN;
SetLength(Result, FMaxMessageLen);
SecBuf.pvBuffer := #Result[0];
SecBuf.cbBuffer := FMaxMessageLen;
{status := QueryCredentialsAttributes(#FCred, SECPKG_CRED_ATTR_NAMES, #attrName);
if status = SEC_E_OK then
FSPN := PWideChar(attrName.sUserName)
else}
// For DAC use "localhost" instead of the server name (Microsoft)
FSPN := WideString(MakeSPN);
FContextAttrib := ISC_REQ_DELEGATE or ISC_REQ_MUTUAL_AUTH or ISC_REQ_INTEGRITY or ISC_REQ_EXTENDED_ERROR;
// ISC_REQ_CONFIDENTIALITY or ISC_REQ_REPLAY_DETECT or ISC_REQ_CONNECTION;
// $8C03C;
// ISC_REQ_MUTUAL_AUTH or ISC_REQ_IDENTIFY or ISC_REQ_CONFIDENTIALITY or ISC_REQ_REPLAY_DETECT or ISC_REQ_SEQUENCE_DETECT or ISC_REQ_CONNECTION or ISC_REQ_DELEGATE;
status := InitializeSecurityContext(#FCred, nil, {$IFDEF FPC}PSecChar{$ELSE}PSecWChar{$ENDIF}(FSPN),
FContextAttrib,
0, SECURITY_NATIVE_DREP, nil, 0, #FCredCtx, #BuffDesc, attrs, #tsExpiry);
if status <= 0 then
raise Exception.CreateFmt('InitializeSecurityContext error %X', [status]);
if (status = SEC_I_COMPLETE_NEEDED) or (status = SEC_I_COMPLETE_AND_CONTINUE) {or (status = SEC_I_CONTINUE_NEEDED)} then begin
status := CompleteAuthToken(#FCredCtx, #BuffDesc);
if status <> SEC_E_OK then begin
FreeCredentialsHandle(#FCred);
Result := nil;
raise Exception.CreateFmt('CompleteAuthToken error %X', [status]);
end;
end
else if (status <> SEC_E_OK) and (status <> SEC_I_CONTINUE_NEEDED) then begin
// SEC_I_CONTINUE_NEEDED
// The client must send the output token to the server and wait for a return token.
// The returned token is then passed in another call to InitializeSecurityContext (Negotiate). The output token can be empty
FreeCredentialsHandle(#FCred);
Result := nil;
raise Exception.CreateFmt('InitializeSecurityContext error %X', [status]);
end;
SetLength(Result, SecBuf.cbBuffer);
end;
function TTDS7SSPI.ParseServerResponse(Buf: TBytes): TBytes;
var
InSecBuff, OutSecBuff: SecBuffer;
InBuffDesc, OutBuffDesc: SecBufferDesc;
status: SECURITY_STATUS;
attrs: Cardinal;
tsExpiry: TTimeStamp;
begin
Assert((Length(Buf) >= 32) or (Length(Buf) <= Integer(FMaxMessageLen)));
InBuffDesc.ulVersion := SECBUFFER_VERSION;
InBuffDesc.cBuffers := 1;
InBuffDesc.pBuffers := #InSecBuff;
OutBuffDesc.ulVersion := SECBUFFER_VERSION;
OutBuffDesc.cBuffers := 1;
OutBuffDesc.pBuffers := #OutSecBuff;
Assert(Length(Buf) > 0);
InSecBuff.BufferType := SECBUFFER_TOKEN;
InSecBuff.pvBuffer := #Buf[0];
InSecBuff.cbBuffer := Length(Buf);
OutSecBuff.BufferType := SECBUFFER_TOKEN;
SetLength(Result, FMaxMessageLen);
OutSecBuff.pvBuffer := #Result[0];
OutSecBuff.cbBuffer := Length(Result);
status := InitializeSecurityContext(#FCred, #FCredCtx, {$IFDEF FPC}PSecChar{$ELSE}PSecWChar{$ENDIF}(FSPN),
FContextAttrib,
0, SECURITY_NATIVE_DREP, #InBuffDesc, 0, #FCredCtx, #OutBuffDesc, attrs, #tsExpiry);
if status <> SEC_E_OK then begin
Result := nil;
raise Exception.CreateFmt('InitializeSecurityContext error %X', [status]);
end
else
SetLength(Result, OutSecBuff.cbBuffer);
end;
The SPN I got is like MSSQLSvc/3R-XP:MSSQL2008 (client and server both on 3R-XP, instance MSSQL2008). InitializeSecurityContext has status SEC_I_CONTINUE_NEEDED. Everything works without errors except that the server does not return any of the rows from the query, only TDS_DONE.
The SQL Server log says:
Login succeeded for user '3R-XP\me'. Connection made using Windows authentication. [CLIENT: 192.168.0.100]
Also I tried to compare OLEDB and mine data sent and received. I can't see the first packet sent by OLEDB due to SSL encryption. Mine SSPI login data
4E 54 4C 4D 53 53 | NTLMSS
50 00 01 00 00 00 97 B2 08 E2 09 00 09 00 2D 00 | P.............-.
00 00 05 00 05 00 28 00 00 00 05 01 28 0A 00 00 | ......(.....(...
00 0F 33 52 2D 58 50 57 4F 52 4B 47 52 4F 55 50 | ..3R-XPWORKGROUP
The server response of OLEDB (connect to another PC due to the fact that WinPCAP can only work with real adapters, so the host name is 'hp-6320' and the client name is '3R-Win7' here) is:
000000 04 01 00 A5 00 00 01 00 ED 9A 00 4E 54 4C 4D 53 | ...........NTLMS
000010 53 50 00 02 00 00 00 0E 00 0E 00 38 00 00 00 15 | SP.........8....
000020 82 8A E2 A3 6E FC 4B 59 86 13 D6 00 00 00 00 00 | ....n.KY........
000030 00 00 00 54 00 54 00 46 00 00 00 05 01 28 0A 00 | ...T.T.F.....(..
000040 00 00 0F 48 00 50 00 2D 00 36 00 33 00 32 00 30 | ...H.P.-.6.3.2.0
000050 00 02 00 0E 00 48 00 50 00 2D 00 36 00 33 00 32 | .....H.P.-.6.3.2
000060 00 30 00 01 00 0E 00 48 00 50 00 2D 00 36 00 33 | .0.....H.P.-.6.3
000070 00 32 00 30 00 04 00 0E 00 68 00 70 00 2D 00 36 | .2.0.....h.p.-.6
000080 00 33 00 32 00 30 00 03 00 0E 00 68 00 70 00 2D | .3.2.0.....h.p.-
000090 00 36 00 33 00 32 00 30 00 06 00 04 00 01 00 00 | .6.3.2.0........
0000A0 00 00 00 00 00 | .....
SQL Server response with my code (machine '3R-XP')
04 01 00 89 00 00 01 00 | ........
ED 7E 00 4E 54 4C 4D 53 53 50 00 02 00 00 00 0A | .~.NTLMSSP......
00 0A 00 38 00 00 00 15 C2 8A E2 B0 17 7A 15 A4 | ...8.........z..
21 2A 96 38 E6 3D 01 00 00 00 00 3C 00 3C 00 42 | !*.8.=.....<.<.B
00 00 00 05 01 28 0A 00 00 00 0F 33 00 52 00 2D | .....(.....3.R.-
00 58 00 50 00 02 00 0A 00 33 00 52 00 2D 00 58 | .X.P.....3.R.-.X
00 50 00 01 00 0A 00 33 00 52 00 2D 00 58 00 50 | .P.....3.R.-.X.P
00 04 00 0A 00 33 00 52 00 2D 00 58 00 50 00 03 | .....3.R.-.X.P..
00 0A 00 33 00 52 00 2D 00 58 00 50 00 00 00 00 | ...3.R.-.X.P....
00 | .
It looks the same. But after that second InitializeSecurityContext OLEDB returns the value
000000 11 01 01 A2 00 00 01 00 4E 54 4C 4D 53 53 50 00 | ........NTLMSSP.
000010 03 00 00 00 18 00 18 00 78 00 00 00 FA 00 FA 00 | ........x.......
000020 90 00 00 00 0E 00 0E 00 58 00 00 00 04 00 04 00 | ........X.......
000030 66 00 00 00 0E 00 0E 00 6A 00 00 00 10 00 10 00 | f.......j.......
000040 8A 01 00 00 15 82 88 E2 06 01 B1 1D 00 00 00 0F | ................
000050 18 B1 57 6E 0F 9B BE 6A AF 2A D4 76 8D B2 19 72 | ..Wn...j.*.v...r
000060 33 00 52 00 2D 00 57 00 69 00 6E 00 37 00 6D 00 | 3.R.-.W.i.n.7.m.
000070 65 00 33 00 52 00 2D 00 57 00 49 00 4E 00 37 00 | e.3.R.-.W.I.N.7.
000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
000090 00 00 00 00 00 00 00 00 3B 97 82 77 95 74 1E 7C | ........;..w.t.|
0000A0 A8 D1 C5 2F 5F 82 7A 9C 01 01 00 00 00 00 00 00 | .../_.z.........
0000B0 EE 4C 92 1E 68 10 D1 01 B3 93 23 3B A9 14 0C EF | .L..h.....#;....
0000C0 00 00 00 00 02 00 0E 00 48 00 50 00 2D 00 36 00 | ........H.P.-.6.
0000D0 33 00 32 00 30 00 01 00 0E 00 48 00 50 00 2D 00 | 3.2.0.....H.P.-.
0000E0 36 00 33 00 32 00 30 00 04 00 0E 00 68 00 70 00 | 6.3.2.0.....h.p.
0000F0 2D 00 36 00 33 00 32 00 30 00 03 00 0E 00 68 00 | -.6.3.2.0.....h.
000100 70 00 2D 00 36 00 33 00 32 00 30 00 06 00 04 00 | p.-.6.3.2.0.....
000110 01 00 00 00 08 00 30 00 30 00 00 00 00 00 00 00 | ......0.0.......
000120 01 00 00 00 00 20 00 00 9B 51 53 D8 0E 0F C8 EB | ..... ...QS.....
000130 F9 11 AB 3D B3 FB 86 F6 D0 D2 97 3C 4C F7 E0 48 | ...=.......<L..H
000140 C4 BF 2F 60 DC CA AB 10 0A 00 10 00 14 5E 11 19 | ../`.........^..
000150 42 DC 79 32 B1 DC 04 C0 C9 48 8D 2C 09 00 2A 00 | B.y2.....H.,..*.
000160 4D 00 53 00 53 00 51 00 4C 00 53 00 76 00 63 00 | M.S.S.Q.L.S.v.c.
000170 2F 00 68 00 70 00 2D 00 36 00 33 00 32 00 30 00 | /.h.p.-.6.3.2.0.
000180 3A 00 31 00 34 00 33 00 33 00 00 00 00 00 00 00 | :.1.4.3.3.......
000190 00 00 7D 45 28 4F E6 4B 38 90 BD F6 91 61 A7 E8 | ..}E(O.K8....a..
0001A0 8D 26 | .&
while for my code it returns
11 01 00 50 00 00 00 00 4E 54 4C 4D 53 53 50 00 | ...P....NTLMSSP.
03 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 | ........H.......
48 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 | H.......H.......
48 00 00 00 00 00 00 00 48 00 00 00 00 00 00 00 | H.......H.......
48 00 00 00 15 C2 88 E2 05 01 28 0A 00 00 00 0F | H.........(.....
As you can see all structures are empty (size 0, allocated 0, offset 48). Is there something wrong? How to fix that stuff? I've tried different flags etc already, the results are the same or even worse. OLEDB works so it seems that the server is configured properly.
With WinAPIOverride I found that authentication uses Bindings (see QueryContextAttributes SECPKG_ATTR_UNIQUE_BINDINGS) retrieved from SSL handshake in negotiation InitializeSecurityContext as SECBUFFER_CHANNEL_BINDINGS member.
So far I made SSPI based SSL handshake, got Bindings that looks like
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 17 00 00 00 20 00 00 00 | ............ ...
74 6C 73 2D 75 6E 69 71 75 65 3A 66 55 6F 05 7F | tls-unique:fUo.
DD 90 31 4F 87 02 52 | ..1O..R
found that those empty NTLMSSP message seems proper (with some extra at the end) while client and server on same machine, ODBC driver sends like
A1 77 30 75 A0 03 0A 01 01 A2 5A 04 58 4E 54 4C .w0u......Z.XNTL
4D 53 53 50 00 03 00 00 00 00 00 00 00 58 00 00 MSSP.........X..
00 00 00 00 00 58 00 00 00 00 00 00 00 58 00 00 .....X.......X..
00 00 00 00 00 58 00 00 00 00 00 00 00 58 00 00 .....X.......X..
00 00 00 00 00 58 00 00 00 15 C2 88 E2 0A 00 5A .....X.........Z
29 00 00 00 0F 9E 3F 5C EE FF F1 AF 9A 44 4C 3A ).....?\.....DL:
6F C3 20 0F 8B A3 12 04 10 01 00 00 00 9C B1 60 o. ............`
36 3B 84 96 09 00 00 00 00 6;.......
remote last authentication data looks like (ODBC driver)
4E 54 4C 4D 53 53 50 00 03 00 00 00 18 00 18 00 NTLMSSP.........
78 00 00 00 3C 01 3C 01 90 00 00 00 0E 00 0E 00 x...<.<.........
58 00 00 00 04 00 04 00 66 00 00 00 0E 00 0E 00 X.......f.......
6A 00 00 00 10 00 10 00 CC 01 00 00 15 82 88 E2 j...............
0A 00 5A 29 00 00 00 0F E0 87 5F 85 21 5A 73 17 ..Z)......_.!Zs.
04 6C 1A F5 9C BA F7 42 33 00 52 00 2D 00 57 00 .l.....B3.R.-.W.
69 00 6E 00 37 00 6D 00 65 00 33 00 52 00 2D 00 i.n.7.m.e.3.R.-.
57 00 49 00 4E 00 37 00 00 00 00 00 00 00 00 00 W.I.N.7.........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
37 F2 38 62 5B 9C 7E 07 6F 89 9F 33 B2 92 3C 5C 7.8b[.~.o..3..<\
01 01 00 00 00 00 00 00 47 39 01 AD AC 4F D1 01 ........G9...O..
0D 36 47 06 7E 70 B8 A4 00 00 00 00 02 00 18 00 .6G.~p..........
48 00 50 00 2D 00 45 00 4C 00 49 00 54 00 45 00 H.P.-.E.L.I.T.E.
42 00 4F 00 4F 00 4B 00 01 00 18 00 B.O.O.K.....
while SSPI with Bindings looks slightly bigger (8 first bytes here is TDS packet header)
11 01 01 EE 00 00 00 00 4E 54 4C 4D 53 53 50 00 | ........NTLMSSP.
03 00 00 00 18 00 18 00 78 00 00 00 46 01 46 01 | ........x...F.F.
90 00 00 00 0E 00 0E 00 58 00 00 00 04 00 04 00 | ........X.......
66 00 00 00 0E 00 0E 00 6A 00 00 00 10 00 10 00 | f.......j.......
D6 01 00 00 15 82 88 E2 0A 00 5A 29 00 00 00 0F | ..........Z)....
AD A5 C9 05 8C 25 E1 A9 C5 3E 17 BD 3D 19 E3 EB | .....%...>..=...
33 00 52 00 2D 00 57 00 69 00 6E 00 37 00 6D 00 | 3.R.-.W.i.n.7.m.
65 00 33 00 52 00 2D 00 57 00 49 00 4E 00 37 00 | e.3.R.-.W.I.N.7.
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
00 00 00 00 00 00 00 00 02 0B C5 A2 01 17 DB AC | ................
D8 26 9E 1B AF A1 77 32 01 01 00 00 00 00 00 00 | .&....w2........
B8 28 90 0F AD 4F D1 01 41 F1 DF 7C BE 85 5D B6 | .(...O..A..|..].
00 00 00 00 02 00 18 00 48 00 50 00 2D 00 45 00 | ........H.P.-.E.
4C 00 49 00 54 00 45 00 42 00 4F 00 4F 00 4B 00 | L.I.T.E.B.O.O.K.
01 00 18 00 48 00 50 00 2D 00 45 00 4C 00 49 00 | ....H.P.-.E.L.I.
54 00 45 00 42 00 4F 00 4F 00 4B 00 04 00 18 00 | T.E.B.O.O.K.....
48 00 50 00 2D 00 45 00 6C 00 69 00 74 00 65 00 | H.P.-.E.l.i.t.e.
62 00 6F 00 6F 00 6B 00 03 00 18 00 48 00 50 00 | b.o.o.k.....H.P.
2D 00 45 00 6C 00 69 00 74 00 65 00 62 00 6F 00 | -.E.l.i.t.e.b.o.
6F 00 6B 00 07 00 08 00 B8 28 90 0F AD 4F D1 01 | o.k......(...O..
06 00 04 00 02 00 00 00 08 00 30 00 30 00 00 00 | ..........0.0...
00 00 00 00 01 00 00 00 00 20 00 00 DC 75 9C 98 | ......... ...u..
70 C7 28 D7 BC C7 1E 14 48 70 0E 3B 8B A4 94 7C | p.(.....Hp.;...|
32 05 44 FD 85 5F D3 54 DB 6C 84 22 0A 00 10 00 | 2.D.._.T.l."....
B1 3B 92 CC 6C 5B E2 CD 0F 24 19 5F 6F 73 47 73 | .;..l[...$._osGs
09 00 3E 00 4D 00 53 00 53 00 51 00 4C 00 53 00 | ..>.M.S.S.Q.L.S.
76 00 63 00 2F 00 48 00 50 00 2D 00 45 00 4C 00 | v.c./.H.P.-.E.L.
49 00 54 00 45 00 42 00 4F 00 4F 00 4B 00 3A 00 | I.T.E.B.O.O.K.:.
4D 00 53 00 53 00 51 00 4C 00 32 00 30 00 30 00 | M.S.S.Q.L.2.0.0.
38 00 00 00 00 00 00 00 00 00 00 00 00 00 F0 46 | 8..............F
20 EB 45 EC C8 67 9F E3 45 45 9C 79 76 47 | .E..g..EE.yvG
QueryContextAttributes(#FCtxHandle, SECPKG_ATTR_NEGOTIATION_INFO, #NegInfo) returns state SECPKG_NEGOTIATION_COMPLETE so everything suppose to be fine, server log shows that "Authentication successful" but there is still not enough rights to get results of queries or server errors like "Cannot find the object "all_types" because it does not exist or you do not have permissions" while simple queries like "SET LOCK TIMEOUT 100" runs without errors.
So my thoughts that Windows Authentication in the eyes of own creator doesn't looks secure enough to allow it to some third-party applications. Guest account enabled and have permissions to read/write data and it works through ODBC driver.
and can it be configured not to happen?
I'm usually finding myself saving a result of a query as a .csv and processing it later on my Unix machine. The characters being null separated makes me have to filter those chars and is a bit of a pain.
So, these are the questions:
Why is this so?
EDIT:
Because it outputs in UTF-16 by default. Easiest conversion would then be:
iconv -f utf-16 -t utf-8 origFile.csv > newFile.csv
Can it be disabled somehow? How?
Here's a piece of a hexdump of a file thus generated. Each char is followed by a
null char (00):
00000cf0 36 00 36 00 32 00 0d 00 0a 00 36 00 38 00 34 00 |6.6.2.....6.8.4.|
00000d00 30 00 36 00 32 00 31 00 36 00 0d 00 0a 00 36 00 |0.6.2.1.6.....6.|
00000d10 38 00 34 00 30 00 36 00 33 00 36 00 34 00 0d 00 |8.4.0.6.3.6.4...|
00000d20 0a 00 36 00 38 00 34 00 30 00 36 00 38 00 34 00 |..6.8.4.0.6.8.4.|
00000d30 32 00 0d 00 0a 00 36 00 38 00 34 00 30 00 37 00 |2.....6.8.4.0.7.|
00000d40 30 00 32 00 31 00 0d 00 0a 00 36 00 38 00 34 00 |0.2.1.....6.8.4.|
00000d50 30 00 37 00 37 00 39 00 37 00 0d 00 0a 00 36 00 |0.7.7.9.7.....6.|
00000d60 38 00 34 00 30 00 37 00 39 00 32 00 31 00 0d 00 |8.4.0.7.9.2.1...|
00000d70 0a 00 36 00 38 00 34 00 30 00 38 00 32 00 34 00 |..6.8.4.0.8.2.4.|
00000d80 31 00 0d 00 0a 00 36 00 38 00 34 00 30 00 38 00 |1.....6.8.4.0.8.|
00000d90 36 00 36 00 31 00 0d 00 0a 00 36 00 38 00 34 00 |6.6.1.....6.8.4.|
00000da0 30 00 38 00 37 00 35 00 31 00 0d 00 0a 00 36 00 |0.8.7.5.1.....6.|
00000db0 38 00 34 00 31 00 30 00 32 00 35 00 34 00 0d 00 |8.4.1.0.2.5.4...|
00000dc0 0a 00 36 00 38 00 34 00 31 00 30 00 34 00 34 00 |..6.8.4.1.0.4.4.|
The file is being outputted in Unicode, not ASCII. Unicode uses twice as many bits to represent each character, hence the preceding 00's.
There might be an option to save as ANSI or ASCII, which should use 8 bit characters.
I know this is an old post...but for new visitors...
When you are saving data from Microsoft SQL Management Studio, you will notice that the 'Save' button has a little arrow next to it. If you select the little arrow you can select 'Save With Encoding...' this will allow you to select the encoding you desire.
On Unix, I suggest the use of iconv -futf-16le -tutf-8 to filter your output. :-)