I have large string in my postgresql database. It's a base64 encoded mp3 and I have to select the column with that large string and get all the data with one select. If I write normal select
SELECT * FROM public.song_data WHERE id=1;
it will return just 204 kB from that string and the string has 2.2 MB.
Also the datagrip shows me just 204 kB of data from that string. Is there a way to get all the data with just one select?
It's strange. Are you sure so your data was not trimmed somewhere? You can use function length for check of actual size.
postgres=# select length('aaa');
┌────────┐
│ length │
╞════════╡
│ 3 │
└────────┘
(1 row)
Two MB are nothing for Postgres, but some clients (or protocols) can problem with it. Sometimes is necessary to use functions lo_import and lo_export as workaround for client / protocol limits. For selecting data from table you should to use SELECT statement. There is not any other way. Theoretically you can transform any string to large object and then by function lo_export you can download this large object from database with LO special protocol. For 2MB it should not be necessary I think.
Please, try to check if your data was stored to postgres correctly. Theoretical limit for text, varchar is 1GB. Practical limit is less value - about 100MB. It is significantly higher value than 2MB.
Postgres has special data type for binary data - bytea. It does conversation to hex code by default and base64 encoding is supported too.
You can select number of chars you want to show using left with negative values (which removes characters from the end) and concat:
SELECT left(concat(public.song_data,' '), -1) from mytable;
The client was actually the problem. Datagrip can’t print of 2MB. I tried another client (DBeaver and Heidisql) and it was ok. Then I selected a row of 2MB with a php select and I got all that data.
Related
In a project I'm working on, I need to stream potentially large data sets from a Postgres database to the client, for analytics purposes.
The application is built in Rails (irrelevant for this question) and after a bit of research I'm currently able to stream query results by using COPY in Postgres:
COPY (SELECT row_to_json(t) from (#{query}) t) TO STDOUT;
Sources (for who's interested):
https://shift.infinite.red/fast-csv-report-generation-with-postgres-in-rails-d444d9b915ab
https://github.com/brianhempel/stream_json_demo
This works, but it yields every row as a key-value pair, e.g.:
["{\"id\":403457,\"email\":\"email403457#example.com\",\"first_name\":\"Firstname403457\",\"last_name\":\"Lastname403457\",\"source\":\"adwords\",\"created_at\":\"2015-08-05T22:43:07.295796\",\"updated_at\":\"2017-01-19T04:48:29.464051\"}"]
In the spirit of minimising the size (in bytes) of the response and especially since this is getting served through the web, I want to return just an array of values for every row, i.e.:
["[403457, \"email403457#example.com\", \"Firstname403457\", \"Lastname403457\", \"adwords\", \"2015-08-05T22:43:07.295796\", \"2017-01-19T04:48:29.464051\"]"]
Is there a way to achieve this within Postgres, even by nesting functions, starting from the query above?
You could create a simple SQL function that converts a row into the desired format:
CREATE FUNCTION row2json(anyelement) RETURNS json
LANGUAGE sql STABLE AS
'SELECT json_agg(z.value) FROM json_each(row_to_json($1)) z';
Then you use that to transform the output:
SELECT row2json(mytab) FROM mytab;
If performance is more important than JSON output, just cast the result to a string:
SELECT CAST(mytab AS text) FROM mytab;
I am setting up a SQL Azure database. I need to write data into the database on daily basis. I am using 64-bit R version 3.3.3 on Windows10. Some of the columns contain text (more than 4000 characters). Initially, I have imported some data from a csv into the SQL Azure database using Microsoft SQL Server Management Studios. I set up the text columns as ntext format, because when I tried using nvarchar the max was 4000 and some of the values got truncated even though they were about 1100 characters long.
In order to append to the database I am first saving the records in a temp table when I have predefined the varTypes:
varTypesNewFile <- c("Numeric", rep("NTEXT", ncol(newFileToAppend) - 1))
names(varTypesNewFile) <- names(newFileToAppend)
sqlSave(dbhandle, newFileToAppend, "newFileToAppendTmp", rownames = F, varTypes = varTypesNewFile, safer = F)
and then append them by using:
insert into mainTable select * from newFileToAppendTmp
If the text is not too long, the above does work. However, sometimes I get the following error during the sqlSave command:
Error in odbcUpdate(channel, query, mydata, coldata[m, ], test = test, :
'Calloc' could not allocate memory (1073741824 of 1 bytes)
My questions are:
How can I counter this issue?
Is this the format I should be using?
Additionally, even when the above works, it takes about an hour to upload about 5k of records. Is it not too long? Is this the normal amount of time it should take? If not, what could I do better.
RODBC is very old, and can be a bit flaky with NVARCHAR columns. Try using the RSQLServer package instead, which offers an alternative means to connect to SQL Server (and also provides a dplyr backend).
I'm converting one of our Delphi 7 projects to Delphi X3 because we want to support Unicode. We're using MS SQL Server 2008/R2 as our database server. After changing some database fields from VARCHAR to NVARCHAR (and the fields in the accompanying ClientDatasets to ftWideString), random crashes started to occur. While debugging I noticed some unexpected behaviour by the TClientDataset/DbExpress:
For a NVARCHAR(10) databasecolumn I manually create a TWideStringField in a clientdataset and set the 'Size' property to 10. The 'DataSize' property of the field tells me 22 bytes are needed, which is expected since TWideStringField's encoding is UTF-16, so it needs two bytes per character and some space for storing the length. Now when I call 'CreateDataset' on the ClientDataset and write the dataset to XML (using .SaveToFile), in the XML file the field is defined as
<FIELD WIDTH="20" fieldtype="string.uni" attrname="TEST"/>
which looks ok to me.
Now, instead of calling .CreateDataset I call .Open on the TClientDataset so that it gets its data through the linked components ->TDatasetProvider->TSQLDataset (.CommandText = a simple select * from table)->TSQLConnection. When I inspect the properties of the field in my watch list, Size is still 10, Datasize is still 22. After saving to XML file however, the field is defined as
<FIELD WIDTH="40" fieldtype="string.uni" attrname="TEST"/>
..the width has doubled?
Finally, if I call .Open on the TClientDataset without creating any fielddefinitions in advance at all, the Size of the field will afterwards be 20(incorrect !) and Datasize 42. After saving to XML, the field is still defined as
<FIELD WIDTH="40" fieldtype="string.uni" attrname="TEST"/>
Does anyone have any idea what is going wrong here?
Check the fieldtype and it's size at the SQLCommand component (which is before DatasetProvider).
Size doubling may be a result of two implicit "conversions": first - server provides NVarchar data which is stored into ansi-string field (and every byte becomes a separate character), second - it is stored into clientdataset's field of type Widestring and each character becomes 2 bytes (size doubles).
Note that in prior versions of Delphi string field size mismatch between ClientDataset's field and corresponding Query/Command field did not result in an exception but starting from one of XE*'s it offten results in AV. So you have to check carefully string field sizes during migration.
Sounds like because of the column datatype being changed, it has created unexpected issues for you. My suggestion is to
1. back up the table,multiple ways to doing this,pick your poison figuratively speaking
2. delete the table,
3. recreate the table,
4. import the data from the old table to the newly created table. See if that helps.
Sql tables DO NOT like it when column datatypes get changed, and unexpected issues may arise from doing just that. So try that, and worst case scenario, you have wasted maybe ten minutes of your time trying a possible solution.
What is the max size of a file that I can insert using varbinary(max) in SQL Server 2008 R2? I tried to change the max value in the column to more than 8,000 bytes but it won't let me, so I'm guessing the max is 8,000 bytes, but from this article on MSDN, it says that the max storage size is 2^31-1 bytes:
varbinary [ ( n | max) ]
Variable-length binary data. n can be a value from 1 through 8,000. max indicates that the maximum storage size is 2^31-1 bytes. The storage size is the actual length of the data entered + 2 bytes. The data that is entered can be 0 bytes in length. The ANSI SQL synonym for varbinary is binary varying.
So how can i store larger files in a varbinary field? I'm not considering using a FILESTREAM since the files I want to save are from 200kb to 1mb max, The code I'm using:
UPDATE [table]
SET file = ( SELECT * FROM OPENROWSET ( BULK 'C:\A directory\A file.ext', SINGLE BLOB) alias)
WHERE idRow = 1
I have been able to execute that code successfully to files less or equal than 8000 bytes. If i try with a file 8001 bytes size it will fail. My file field on the table has a field called "file" type varbinary(8000) which as I said, I can't change to a bigger value.
I cannot reproduce this scenario. I tried the following:
USE tempdb;
GO
CREATE TABLE dbo.blob(col VARBINARY(MAX));
INSERT dbo.blob(col) SELECT NULL;
UPDATE dbo.blob
SET col = (SELECT BulkColumn
FROM OPENROWSET( BULK 'C:\Folder\File.docx', SINGLE_BLOB) alias
);
SELECT DATALENGTH(col) FROM dbo.blob;
Results:
--------
39578
If this is getting capped at 8K then I would guess that either one of the following is true:
The column is actually VARBINARY(8000).
You are selecting the data in Management Studio, and analyzing the length of the data that is displayed there. This is limited to a max of 8192 characters in results to text, if this is the case, so using DATALENGTH() directly against the column is a much better approach.
I would dare to say, use file stream for files bigger than 1 MB based on the following from: MS TechNet | FILESTREAM Overview.
In SQL Server, BLOBs can be standard varbinary(max) data that stores
the data in tables, or FILESTREAM varbinary(max) objects that store
the data in the file system. The size and use of the data determines
whether you should use database storage or file system storage. If the
following conditions are true, you should consider using FILESTREAM:
Objects that are being stored are, on average, larger than 1 MB.
Fast read access is important.
You are developing applications that use a middle tier for application logic.
For smaller objects, storing varbinary(max) BLOBs in the database
often provides better streaming performance.
"SET TEXTSIZE" Specifies the size of varchar(max), nvarchar(max), varbinary(max), text, ntext, and image data returned by a SELECT statement.
select ##TEXTSIZE
The SQL Server Native Client ODBC driver and SQL Server Native Client OLE DB Provider for SQL Server automatically set TEXTSIZE to 2147483647 when connecting. The maximum setting for SET TEXTSIZE is 2 gigabytes (GB), specified in bytes. A setting of 0 resets the size to the default (4 KB).
As mentioned, for big files you should prefer file stream.
Just ran into a major headache when concatenating several #varchar(max) variables together to build an email based on several different queries.
For efficiencies sake, I was using several varchars to build the email at once, rather than going through roughly the same query two or three or more times to build it using only one varchar.
This worked, right up until my varchars got to longer than 8000 characters. Then the concatenation of them all into one varchar (which I could shove into the #body parameter of msdb.dbo.sp_send_dbmail) returned "", and even LEN() wouldn't actually give me a length.
Anyhow, I've gotten around this by doing roughly the same queries several times and building the email with only one varchar(max).
TL;DR
I'm not happy with the solution. How could I have appended these varchar(max) variables to each other?
One thing I've hit in the past which may or may not help here: SQL seems to "forget" what datatype its working with when you concatenate varchar(max). Instead of maintaining the MAX, it devolves to conventional varcharnitude, meaning truncation at 8000 characters or so. To get around this, we use the following trick:
Start with
SET #MyMaxVarchar = #aVarcharMaxValue + #SomeString + #SomeOtherString + #etc
and revise like so:
SET #MyMaxVarchar = cast(#aVarcharMaxValue as varchar(max)) + #SomeString + #SomeOtherString + #etc
Again, this may not help with your particular problem, but remembering it might save you major headaches down the road some day.
This may not have happened in your case, but there's a "gotcha" embedded in SQL Management Studio involving VARCHAR(MAX): SQL Studio will only output so many characters in the results grid. You can test this:
SELECT #MyLongVar, LEN(#MyLongVar)
You may find that the length of the actual data returned (most text editors can give you this) is less than the length of the data stored in the variable.
The fix is in Tools | Options | Query Results | SQL Server | Results to Grid; increase Maximum Characters Retrieved | Non XML data to some very large number. Unfortunately the maximum is 65,535, which may not be enough.
If your problem does not involve outputting the variable's value in SQL Studio, please disregard.
I have found that MS SQL silently does NOTHING when attempting to concatentate a string to a NULL value. therefore this solution always works for me:
UPDATE myTable
SET isNull(myCol, '') += 'my text'
WHERE myColumnID = 9999