i have a very big table and requirement is to fetch the records based on char(10) field but actually it stores date .
it is taking plenty of time if i use 'between' in where condition. i have tried converting into date filed in where condition, still no luck.
i dont know why they store date in varchar, but i can't change anything now. Please help
Related
I have a dataframe that I am writing to a table. I don't necessarily need to retain the data day to day, so I would prefer to just overwrite it. I tried using the following function assuming that it would replace the data rather than the table:
df.to_sql('new_test', con=connStr, if_exists='replace', index=False, schema='master')
The issue with this is I assume the SQL language on the backend dropped the table rather than truncated the data within. I suspect this because it completely scrubbed by table of all constraints and changed a datetime column to a timestamp column. Is there a way around this? Thanks in advance.
I have a pivot table in Excel, which I'd like to be able to performing Grouping on using the SaleDate column.
However, when I've created my Pivot Table, right click an element in the field and choose Grouping... I get the error that:
Cannot Group on this selection
Which I've figured is because there is either
1) Blanks in the column, or
2) The column is not of date type Date in Excel
I've copied the whole column to Notepad++ and performed a Find what: (a blankspace) but that gave nothing in return, i.e. there are no blank spaces in the columns.
That leaves option number 2, and since I can't filter the SaleDate column on Year or Month it seems to in fact be interpreted as a text rather than a Date.
I'm using a SQL database as a source, which I have tried to adjust to parry this (my raw data to the SQL is of data type numeric, hence I first need to convert it to varchar and subsequently to date (note that these are Three different approaches I have used to adjust the date in SQL. I have noticed that the table to which I save the data is indeed of data type date in SQL):
left(convert(date,convert(varchar,Rawdate,110),110),7) as SaleDate
convert(date,convert(varchar,Rawdate,112),112) as SaleDate
convert(date,convert(varchar,Rawdate,110),110) as SaleDate
which returns, in order, yyyy-mm, yyyy-mm-dd, yyyy-mm-dd but none of these works to either Group on in the Pivot Table, or filter on Months or year in Excel.
While I never worked specifically on Excels that utilize SQL-Server directly, I know SQL-Server has many date & time types, unlike .NET's C# which has fewer, or Excel which has only one(1). The types of SQL-Server itself are not that cooperative with each other to begin with(2), so I wouldn't be surprised if issues arise from even the tiniest differences when trying to port to other technologies, which I faced a few times.
With that in mind, and your evidence of a likely failure in date conversion, plus the chained conversions you mentioned, my first suggestion is to feed the Excel a different date type, and my first choices would be datetime or datetime2, for being the most popular, the most complete, and the most similar to Excel's lonely type.
(1): It's more like zero, it's just an integer with everybody around it giving it special treatment, which they fail at half the time.
(2): Why would int to/from datetime be fine, but int to/from datetime2 is not...
If you make the field of interest a rowfield, and click on the "Filter" triangle in the column heading, then often right to the bottom of the list in that PivotFilter box you'll see the item(s) causing you the problem. Be aware that it might be text that simply looks like a date, or it might be something more obvious as per the basic example in the screenshot below:
As per my comment, another way to diagnose what's going on is by taking just a few of the items that you are sure are dates, putting them into a range, making a PivotTable out of them, and seeing if you can group them. If you can, then you know that the problem is indeed likely some text in the data. If you can't, then it's likely you have text that still needs to be converted to dates...but you'll need to post some examples here in order for us to give you suggestions on how to turn it into something Excel recognizes as a date.
I'm running the following query on a table from SQLPro for MSSQL:
SELECT * FROM MyTable where Date = '2016-06-28'
The Date column contains fields formatted as datetimes for example: '2016-06-27 19:00:00:000'. When I run the query it returns results whose entries in the Date column are one day earlier than the date I queried for. So in the above example all results returned have date '2016-06-27'. There is data for the date I'm looking for since if I query for '2016-06-29' I get the data for the 28th.
Further, when I query from a cursor using pymssql I get the data for the right dates so it seems like the issue is with SQLPro and not the database itself. Anyone know what's going on/how to fix it?
Developer here. It is indeed a bug with SQLPro. I've sent a fix to #stableMatch which should resolve the issue.
I have few questions on what data types to use and how to define some fields from my site. My current schema is in MySQL but in process of changing to PostregSQL.
First & Last name -> Since I have multi-lang, tables all support UTF-8, but do i need to declare them as nvarchar in-case a user enters a Chinese name? If so, how do i enforce field validation if it is set to accept alphabets only as i assume those are English alphabets and not validating for valid chinese or arabic alphabets? And i don't think PostregSQL supports nvarchar anyways?
To store current time line - > Example I work in company A from Jan 2009 to Present. So i assume there will be 3 field for this: timeline_to, timeline_from, time_line present where to & from are month/year varchars and present is just a flag to set the current date?
User passwords. i am using SHA 256 + salting. so i have 2 fields declared as follows:
password_hash - varchar (64)
password_salt- varchar (64)
Does this work if the user password needs to be between 8 and 32 chars long?
birth time -> I need to record birth time for the application to calculate some astrological values. so that means hour, minute and am/pm. So best to store these are 3 separate single select lists with varchar or use a time data type in the back end and allow users to use single select list in front end?
Lastly for birth month and year only, are these int or varchar if i store them in separate rows? They all have primary keys of int for reporting purposes so int makes more sense? or should i store them in 1 field only as date type?
NCHAR, not NVARCHAR.
Never make anything variable that you can make fixed; it is an added burden to pack/unpack on every access. Which means never, ever use var for indexed columns, you will have a very sluggish index. Disk space is cheap these days.
you need a Language column at the Person level that tells you what language to use in your various parsing and validation requirements.
Let's say you have Person, Employer, and Employment tables. The columns you discuss are in Employment.
you need a StartDate column and EndDate column, they are DATETIME datatype.
You do not need "present" as a separate column. "Present" is always the value of the newest Employment row, unless set to something different. Set a Default of the highest date the db can handle, eg. 9999-12-31; which can be overridden by an explicit entry.
No. You only need one CHAR(256) column. Hank has explained it.
For any component of a date or time, use the DATETIME datatype. That is what it is there for. The database handles it consistently, and indexes it perfectly. You perform DATE arithmetic on it, using db various functions(). And you avoid all the problems of coding it as INTs, etc (no invalid dates or times allowed).
BirthDateTime is one DATETIME column.
I have no idea, never dealt with that field much.
You might consider allowing NULL here and using it as a special meaning for Present. If your application logic sees a non-null start date and a null end date, you can infer this. If they are both NULL, then no information can be inferred.
Since you're hashing, you'll always get a 256-bit hex string as the output no matter what the input is, so yes, 8-32 character passwords will all work.
Use a DATETIME in the backend. You can do things like MONTH() to extract the parts right in your SQL syntax. Of course, you'll have to format the date just right for SQL to accept it, but that's not too hard.
Again, all extractable with the DATETIME functions in SQL.
i need to find data between 2 date's and time's.
i use one field for date , and one field for time.
is it be better to use only one field for date & time ?
i see that it came in dd/mm/yyyy hh:mm:ss format that
can contain date and time.
this question is for acceess and for sql-server
thank's in advance
In nearly all circumstances, date and time are needed together. Both Access and SQL server have a date/time data type. In Access, even if you specify the format as time, you can show a date. This is because all datetime data is stored as a number, and time is the decimal portion:
Say I store data: 10:31:46, I can type lines in the immediate window that illustrate the storage of datetime, like so:
?CDec(DlookUp("TimeFormattedField", "Test"))
0.438726851851852
?Year(DlookUp("TimeFormattedField", "Test"))
1899
?Format(dlookup("F4", "Table2"),"dd/mm/yyyy")
30/12/1899
This is because zero (0) is a valid date.
It is very easy to get the different portions of a datetime field, so store datetime as a single fields, because that is what you are going to get, anyway.
I like to store date and time separately. In general, I almost never need time in my apps. One case where I store them separately is in some of my logging routines. This is mostly because I only ever query on dates and never on date+time.
If you need to query on both date and time, then storing them separately is really problematic, because then you have to concatenate two fields for comparison, and that means your criteria won't use any indexes on the two fields. This may not be an issue for a few thousand records, but for anything above that, it can quickly become quite a performance drag. It's also a major issue if you're using a server back end, since all the rows will have to be pulled across the wire, instead of Access/Jet/ACE being able to hand off the selection to the server.
depends on the requirement. If you are using sql server 2008+ then if you store in separate is not a problem, as well as it is as easy option to write the query