I'm using Django that is connected to AWS' RDS MSSQL. So I'm to input very long texts, but they are generally longer than 4000 characters. In Django, this specific field is using TextField and the data type in MSSQL is nvarchar(max). So my question now is how can I configure and increase its max length. I've got the assumption that it's supposed to be infinite (of course it's not).
Help is desperately needed. I've read about it briefly here: How to get entire VARCHAR(MAX) column with Python pypyodbc. However, I'm using RDS and I have no idea in doing this as at all.
Thank you all in advanced!
If you are using django-pyodbc-azure, upgrading from 1.11.11.0 to 1.11.13.1 fixed the issue for me.
Related
I am beginner in python and Flutter (4 months that i actually code) self-taught . There is a lot of information and I am facing a problem. I don't know how to go about it.
I am building a dictionary app in Flutter. I extracted all the words and definitions from the French Wiktionary with Scrapy and I processed all the data with pyspark. All data was inserted into an ObjectBox database with python and which has a final size of 460 mb for 355,000 elements. I compressed it with Brotli and its final size is 65 mb. So I end up with a compress mdb file. And now I'm stuck.
I thought of extracting the database to read it live with ObjectBox in Flutter but it does not read the mdb files directly and it gets complicated, I can't find any documentation on the subject. Moreover I tell myself that if the live extraction is not done on the internal memory, it will reserve ram? (If I understood correctly) This will not lead to a crash?
Or is it possible to extract it when installing the App so that objectBox can read it directly?
Or maybe I'm scratching my head over nothing. Directly read a Json but I'm afraid that the queries will be long because the word search is live for the user That this one writes a letter, my program must return me the words beginning with this one.
What would you do to maximize performance in the background ? Thanks for your help .
welcome on StackOverflow! :)
I think your question is a duplicate of this one:
How setup dart objectbox with a local database pre-populated?
as long as you refer to this NoSQL database by ObjectBox in both python and flutter and you're trying to use this package in the former and this in the latter.
Is that the case? Does the answer on the other question help you? If not, can you pls. elaborate what's missing/failing?
In general if you add a few links/references + narrow down the number of things/topics you ask might help (you can ask follow-up questions in comments later on answers or just post a new question if you still need infos). Not that I'm a pro here, could just make it easier for others to answer IMHO.
We are (still) using ZF1, to write/read from a Microsoft SQL database.
As long as a table field is of type NVARCHAR, everything is fine, special characters (like, chinese, polish, etc.) and emojis are saved correctly, 1:1 equal to the entered text in an html form.
But when it comes to column type TEXT, there are Question marks instead of the special characters/emojis in these columns.
Funny enough that everything is fine on servers where we got PHP5.6 running, with FreeTDS and php-mssql installed on them. This problem occurs on a system where PHP7.4 is installed, using only SQLSRV as the db driver.
According to this thread, a simple N should be enough...as long as the fields are set as NVARCHAR fields. We are using the update and insert methods of Zend Framework, so we got no clue how to save all these characters into TEXT columns.
When connecting, our bootstrap file is using UTF-8 as the character set, used as an driver option.
Is there any workaround we could try to accomplish this? We cannot change the column type because there are reasons why some colums are TEXT.
Why are you using text? It (text) has been deprecated for 16 years. text, however, does not support unicode characters, that would be ntext; but that too has been deprecated for 16 years.
To store up to 2GB of characters, that need to be unicode, use nvarchar(MAX). For a column that already exists, ALTER it.
ALTER TABLE dbo.YourTable ALTER COLUMN YourColumn nvarchar(MAX);
As per OP response in the comments the database setup can be put aside given that the TEXT column type behaves as expected when using PHP 5.6 and the operation fails on the same database when using PHP 7.4.
This allows to narrow the issue to runtime PHP charset handling. Not knowing all the specifics between the upgrade of PHP 5.6 to 7.4, the issue may lie in:
Different default charset in php.ini
Different behavior in the mssql driver with charset handling
Different operating system default locale which may affect the script
Different php file encoding (the encoding of the .php file itself may alter the charset used at runtime)
Since the issue at hand cannot be unambiguously identified based on the provided information, I may only suggest to check and eliminate the different possibilities one by one.
try detecting the actual encoding of the input text using any of the mb_* functions (maybe refer to this question)
try adding a comment in your script file containing special characters and be sure to save the file as UTF8 (or desired encoding)
try setting explicitly the driver charset (refer to this question) (or maybe in the connection string, if supported)
try setting the default php locale using setlocale and other functions of the sort
Those steps may guide you through a resolution but without guarantees.
Is there a way to read Excel 2010/2013 files natively ?
We are importing Excel files into SQL Server and have come across a specific issue whereby it looks as though the Excel driver decides the type of a destination data column depends upon testing the contents of only the first 65K odd rows.
This has only just started happening within the past 3 weeks, before then we had managed to convince Excel of the error of its ways by a simple registry hack that forced it to read the entire set of rows.
The problem is that we have some datasets that contain, say 120,000 rows and these may have all numeric values for the first 80,000, then it will have some non-numeric yet vital information that we wish to retain.
Yes, the data is not correctly typed, we know.
Because the source data type has been determined by the Excel driver to be a float it promptly turns all our non-numeric values into NULLs - not very useful.
If there was some other way to read an Excel file not using the standard ODBC/OLEDB drivers that might help.
We have tried saving it into various other formats before importing but of course all these exports use the Excel driver which has the problem.
I think the closest we have got is to save it as XML (which is frankly huge at 800MB) and then shred it using standard xpath queries and some pretty dodgy workarounds to handle no doubt well-formed but still tricky variations on how column data is represented.
Edit: changed title to more closely reflect the issue
As well as the registry key, when connectting to your excel file have you tried setting the following:
;Extended Properties="IMEX=1"
See here
Also see this MSDN article
I am using wicket, spring and mysql. When I try to insert cyrilic characters to the database instead of the characters I see question marks ('?????').
When I use the i18n files I don't have such kind of problem. Also, when inserting characters from other place (i.e. mysql workbench) there is no problem. In other words, the problem is not in the database. The database and my html documents are in utf-8.
Could you please help me solve this problem.
Cheers!
Nikolay
Wicket has two more encoding-related settings, try this (in you Application initialization):
getMarkupSettings().setDefaultMarkupEncoding("utf-8");
getRequestCycleSettings().setResponseRequestEncoding("utf-8")
This question already has answers here:
Storing images in SQL Server?
(8 answers)
Closed 4 years ago.
I want to insert my product's image into a database and then want to get those image on request. please suggest.
I disagree with Aaron. Storing images in the file system always leaves you with the potential problem of orphaned files and/or database entries. It is difficult to maintain "referential integrity". There may be certain scenarios where you don't want to go the blob route, because the images are particularly large, etc, but in most standard application scenarios I would store the image in the database.
The best way to store your images in a SQL Server 2005 database is to store the path of where the image actually exists.
See this for more information.
https://stackoverflow.com/questions/805519/save-image-in-database
This is a very common SQL Server discussion. The general rule of thumb is if the files to be stored are larger than say 1MB then store them in the file system.
For a complete discussion of the issues involved consult the following Microsoft Research document.
http://research.microsoft.com/apps/pubs/default.aspx?id=64525
On a side note SQL Server 2008 uses Filestream technology to cater for this scenario.
SQL Server Filestream
inserting is easily done into a field of BLOB type.
getting the image back will depend on what language you will be using. Do you use PHP?
SQL Server 2005 does have an Image datatype. Use it.
Regarding if it is good or bad, it depends on you requirements.
Best way is upgrade on SQL Server 2008 and use FileStream (content stored in the file system)
Skipping the whole "are images in the db bad" debate, you need to solve a few problems here.
1) Data Storage: you need to have someplace to put the images. In SQL 2005, you would be best off using a VARBINARY(MAX) column.
2) Upload: you will need to upload the image into your database. Check out the FileUpload command. In terms of actually INSERTING stuff, the .NET data type would be byte[]. Your ORM should handle this for you in any case. Also remember you should store the file size, file name and the mimetype provided, you'll need those soon enough.
3) Display: here's the one that trips most folks up. You need to make a way to request the image file for display. The best way is to implement a custom IHttpHandler. Then SELECT the image data, file name, mimetype and file length. Push the mimetype and filename in the header, set the content length to the file length and stream the binary data to the output stream.
Another option that works very easy
Convert your image to Base64 string
store it on a varchar field
To display it, just convert it from base64 to binary....
This works with any DB engine, in .net conversion from binary to base64 takes microseconds...