Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
We're developing a new product at work and it will require the use of a lightweight database. My coworkers and I, however, got into a debate over the conventions for database creation. They were of the mindset that we should just build quick outline of the database and go in and indiscriminately add and delete tables and stuff until it looks like what we want. I told them the proper way to do it was to make a script that follows a format similar to this:
Drop database;
Create Tables;
Insert Initial Data;
I said this was better than randomly changing tables. You should only make changes to the script and re-run the script every time you want to update the design of the database. They said it was pointless and that their way was faster (which holds a bit of weight since the database is kind of small, but I still feel it is a bad way of doing things). Their BIGGEST concern with this was that I was dropping the database, they were upset that I was going to delete the random data they put in there for testing purposes. That's when I clarified that you include inserts as part of the script that will act as initial data. They were still unconvinced. They told me in all of their time with databases they had NEVER heard of such a thing. The truth is we all need more experience with databases, but I am CERTAIN that this is the proper way to develop a script and create a database. Does anyone have any online resources that clearly explain this method that can back me up? If I am wrong about this, then please fell free to correct me.
Well, I don't know the details of your project, but I think its pretty safe to assume you're right on this one, for a number of very good reasons.
If you don't have a script that dictates how the database is structured, how will create new instances of it? What happens when you deploy to production or it gets accidentally deleted or the server crashes? Having a script means you don't have to remember all the little details of how it was all set up (which is pretty unlikely even for small databases).
It's way faster in the long run. I don't know about you, but in my projects I'm constantly bringing new databases online for things like unit testing, new branches, and deployments. If I had to recreate the database from by hand every time it would take forever. Yes it takes a little extra time to maintain a database script but it will almost always save you time over the life of the project.
It's not hard to do. I don't know what database you're using but many of them support exporting your schema as a DDL script. You can just start with that and modify it from them on. No need to type it all up. If your database won't do that, it's worth a quick search to see if a 3rd party tool that works with your database will do it for you.
Make sure your check your scripts into your source control system. It's just as important as any other part of your source code.
I think having a data seeding script like you mentioned is a good idea. But keep it as a separate script from the database creation script. This way your can a developer seed script, a unit testing seed script, a production seed script, etc.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I need to do research about log files not stored in the database. I do not know much about database systems so i need someone to give at least some ideas about it. What i was told is that some of the log files was not written in a bank's database.Log files are coming from various sources like atms,website vs. For example, the reason could be high rate of data flow causing some data to be left out.
The question is what are the reasons behind it and what could be the solutions to them?
I would really appreciate if you could share some articles about it.
Sorry if i could not explain it well. Thanks in advance
Edit:what i meant was not there is a system not writing some of log files to database intentionally. What i tried to mean is that some of the log files are not written into database and the reason is not known and my intention is to identify the possible reasons and solutions to them.the database belongs to a bank and as you can imagine, lots of data is flowing to database per second
Well, the questions is not very clear, so let me rephrase it:
What are the reasons why application logs are not stored in a database
It depends of the context, and there are different reasons:
First question, is why you might store logs in database? Usually you do it because they contains relevant data to you that you want to manipulate.
So why not store always these datas:
you are not interested by the log, except when something goes wrong, but then it's more debugging than storing log.
you don't want to mix business data (users, transaction, etc...) with not so important / relevant data
the amount of log is too important for your current system and putting them in a database might crash it completly
you might want to use another system to dig into the log, with a different typoe of storage (haddop, big data, nosql )
when you do database backup, you usually backup all the database. Logs are not 'as important' as other critical data, are bigger, and then would take too much place
there is no need to always put logs in database. Using plain text and some other tools (web server log for instance) is usually more than enough.
So that's for these reason that logs are in general not stored in the same database than the application.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I work on a project that has very well defined lines of responsibility. There are about six to ten of us and we currently do all of our work in Excel, building a single spreadsheet with maintenance requirements for ships. A couple of times during the project process we stop all work and compile all of the individual spreadsheets into one spreadsheet. Since each person had a well defined area, we don't have to worry about one person overwriting another person's work. It only takes an hour, so it isn't that huge of a deal. Less than optimal, sure, but it gets the job done.
But each person fills out their data differently. I think moving to a database would serve us well by making the data more regimented with validation rules. But the problem is, we do not have any type of share drive or database server where we can host the database, and that won't change. I was wondering if there was a simple solution similar to the way we were handling the Excel spreadsheet. I envisioned a process where I would wipe the old data and then import the new data. But I suspect that will bring up other problems.
I am pretty comfortable building small databases and using VBA and whatnot. This project would probably have about six tables, and probably three that would have the majority of the data for any given project (the others would be reference tables and slow-to-change data). Bottom line is, I am wondering if it is worth it, or should I stick with Excel?
Access 2007 onwards has an option for "Collecting email replies" which can organise flat data, but it can only be a single query that's populated so might be a bit limiting.
The only solution I can think of that's easier than you currently use is to create the DB with some VBA modules that export all new/updated data to an XML/csv file and attached this to an email. You'd then have to create a VBA module that would import the data from these files into the current table.
It's a fair amount of work to get set up but once working might be fairly quick and robust.
Edit, just to add, I have solved a similar problem but I solved it with VB.net and XML files rather than Access.
You can link Access databases to other databases (or import from them). So you can distribute a template database for users to add records to and then email back. When receiving back, you would either import or link them to a master database and do whatever you needed to do with the combined data.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
What is the best practice for a team working on a same Database?
Should developers use their Local database? Or a shared development database instance?
In my experience a team should have (at least) one shared database for integraton.
During development each team member should have an independed database otherwise changes of database schema could hinder other team members. These instances could also be on a centralized server.
I can only talk about the way the team I'm in currently works, which fits our needs well enough:
We have one central data model script that updates any database automatically to the latest schema version. Developers check in changes to this script together with changes to the source code (single commit on the same repository). The nightly builds update a central database copy, followed by a batch of automated tests on that database, and the human QA team also uses this same database the next day for all their testing.
We don't allow schema changes on the central database instance any other way than via the integration builds. For developing the schema change script, changes are developed on a separate database instance, either on the central server, or locally (depending on personal preference).
I don't think it depends at all. Each environment should have it's own database instance. Personally, I would never reccommend that everyone on the team works on the same copy of the source, and I view the database code and instance the same way.
If you are having problems with missing database changes, this is a symptom of a different development process issue. It would be analagous to forgetting to add a code file to source control.
Jeff Atwood has a pretty good article on source controlling database code.
Different developers supposedly work on different issues - how do you avoid stepping on other people's toes while unit testing?
I would absolutely advocate an integration/test environment, which is updated via a Continuous Integration process. This environment often serves as a litmus test for your deployment procedure as well.
At Redgate we'd recommend that each developer is given their own instance, as sandboxing ensures that developers don't tread on each other's toes. However, there are pros and cons with both models.
In our experience talking to database developers, roughly half of database development is performed on a shared environment, and half on a dedicated per-developer environment.
From experience, a shared database is better. Our code used to break all the time because someone added a column on their local database, then uploaded their new source code to SVN. Everyone else's implementation would then break until they figured out what had changed in the database.
I would have a shared database for development. We had one or two dev databases too for miscellaneous testing.
Having seperate database instances for a developer help them work in isolation however if we are a large team and working on multiple things at the sametime then it is good to have a shared enviornment too so that all the changes that needs to be delivered at the same time are not breaking each other and their dependecies could also be figured out correctly. This also helps in identifying where any application may break due to a change. So it is better to have a shared enviornment too along with some isolated enviornment replica copy of the development enviornment just in case we might want to test something big and complicated that is very critical and need to work anyhow.
But the only problem is to keep multiple enviornment in sync with each other as any missing change can be fatal.
I know people here are speaking in terms of their past experience but this is not such an easy question to answer universally. There are pros and cons to each approach and should be addressed based on the type of application and team you're working with.
If you don't know which route to go .. I would suggest going with shared development database first to see if you encounter any problems. If this solution works, it should be the preferred method since it eliminates the "integration" step. However, depending on the type of team and environment needs, you may need to adapt.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
As a tester, how much should one expect a tester to know about databases?
Is it only writing queries in sql or do we need to know about stored procedures, triggers etc.
Might want to tag this with a testing tag.
Depending what your product is, a tester may or may not need to know specific things. From my perspective, the professional tester is the last line of defense before a product gets into the wild. Thus, testers should at the root of it be using the product like users (crazy, manic users who are only interested in pain and misery, of course).
That mindset in place, you can use black-box testing (testers use no more information than users), white-box testing (testers use all information, source code, etc.), or something in between.(1) In my company we are more black-box, but even for products it is helpful to have a detailed understanding of implementations. This is true not necessarily from a development perspective, but to give ideas about where the complexities - and therefore, often, bugs - lie. Depending on the quality of your developers, the testers will need to be more or less capable of determining this on their own, as it is a rare programmer that can thoroughly test her own code.
Once you have settled what you are testing and what your customers are doing, then you will know if testers need to know SQL, procedures, triggers, etc. If you are delivering, for example, a hosted database solution, then your testers will have to know these things. If you use a traditional, non-custom database server on the back end of your delivered software package and you are a black-box shop, then your testers don't necessarily need to know anything about SQL at all - the software should be handling it. (That's not to say it's not helpful in debugging, test selection, etc., but I'd always rather have a good tester with no knowledge of my field, than an average tester with coincidental domain knowledge.)
Looking at the question again, if you are just asking from a personal skills perspective - then yes, it is always a good thing to learn something else, and it will undoubtedly come in handy eventually. :)
1) I have a strong preference toward black-box testing, but there are plenty of arguments for white-box as well, so it would be worthwhile to review the differences if you're in a position to determine overall testing strategy on a project.
Shouldn't know much, but if you know how to write queries (you understand the language) you know stored procedures and triggers :)
You don't really need to know anything about db administration tho.
I've never expected a tester to have any real knowledge of sql... Unless they were testing a database implementation ;)
My feeling is that testers should be on par with end users plus certain other knowledge. The "other knowledge" is a bit ambiguous based on the project needs but usually is one or more of the following:
Has detailed knowledge of the business plan and exactly what the program is supposed to do. In this case they need to work hand in hand with your business people.
Knows how to do edge case checking. For example, entering 1/1/1800 or 12/31/2040 in a birthdate field. Is this allowed? Other examples include entering negative amounts or even alpha characters in numeric entry fields.
Has experience doing what the users typically do. This might be simply from on the job training such as sitting next to a user for a week or two every so often.
Has experience or knowledge of tools of the trade. There are a ton of testing tools out there which allow recording and playback of testing scenarios. I would expect a tester to be proficient in those tools.
Now, sometimes running simple sql queries is part of the job. For example, you might have to verify a workflow process that kicks off automated systems. In this case, being able to execute sql and reading the results is important.
However, those queries are usually provided to the testing staff by development or a dba. Again, most people can be trained on how to copy / paste, maybe change a parameter, and execute a query while interpreting results in less than an hour.
Obviously, it depends on your unique situation, as any software shop will do business differently than elsewhere.
I've worked with testers who knew nothing of SQL. We trained them to simply execute the sproc tests we wrote and notice if they changed. They would write up the logic for the test cases, and we'd do the code implementation.
So I'd say the testers could know nothing of SQL if your situation would allow them to merely be trained to do whatever tasks they're really needed for.
However, in an ideal world, I'd vote to say the testers should have a good-enough knowledge of code or SQL to write their own tests themselves, though this may introduce layers of arguments in interfacing when you have 2 testing teams (1 strict dev, 1 strict testing).
You will be a better tester if you know SQL. You don't need to know it in depth, but it's useful to know for a number of reasons:
You'll be able to do simple queries
to check the data that is displayed
on the screen is correct with
respect to the database, without having to go to a developer to check. This saves time.
You can start to analyse a bug
without having to go to a developer,
for instance, if the data in the
database is incorrect, if it is
displayed incorrectly. This will
save time, and help you to target
your tests more effectively. This will help you find bugs.
You'll understand (a little) of the
job of the developer, and you can
start to understand whats possible
and what is not. This will help you with your relationship with the developers.
You can improve the quality of your
testing: in the past, I've written
simple Excel spreadsheets which
calculated values based upon
database queries (you are
independently checking the
developers work). This helps the overall quality of the product.
The principle is that your core competence is testing, but it is always a good idea to know a little of all of the surrounding competences: A tester should know a little development, a analysis, a project management, but not to the same depth. In the same way, a developer should know a little about how a tester works (AND analysis and project management).
You don't NEED to know SQL, but it's a good idea. The same applies to triggers and stored procedures.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I have been developing web/desktop applications for about 6 years now. During the course of my career, I have come across application that were heavily written in the database using stored procedures whereas a lot of application just had only a few basic stored procedures (to read, insert, edit and delete entity records) for each entity.
I have seen people argue saying that if you have paid for an enterprise database use its features extensively. Whereas a lot of "object oriented architects" told me its absolute crime to put anything more than necessary in the database and you should be able to drive the application using the methods on those classes?
Where do you think is the balance?
Thanks,
Krunal
I think it's a business logic vs. data logic thing. If there is logic that ensures the consistency of your data, put it in a stored procedure. Same for convenience functions for data retrieval/update.
Everything else should go into the code.
A friend of mine is developing a host of stored procedures for data analysis algorithms in bioinformatics. I think his approach is quite interesting, but not the right way in the long run. My main objections are maintainability and lacking adaptability.
I'm in the object oriented architects camp. It's not necessarily a crime to put code in the database, as long as you understand the caveats that go along with that. Here are some:
It's not debuggable
It's not subject to source control
Permissions on your two sets of code will be different
It will make it more difficult to track where an error in the data came from if you're accessing info in the database from both places
Anything that relates to Referential Integrity or Consistency should be in the database as a bare minimum. If it's in your application and someone wants to write an application against the database they are going to have to duplicate your code in their code to ensure that the data remains consistent.
PLSQL for Oracle is a pretty good language for accessing the database and it can also give performance improvements. Your application can also be much 'neater' as it can treat the database stored procedures as a 'black box'.
The sprocs themselves can also be tuned and modified without you having to go near your compiled application, this is also useful if the supplier of your application has gone out of business or is unavailable.
I'm not advocating 'everything' should be in database, far from it. Treat each case seperately and logically and you will see which makes more sense, put it in the app or put it in the database.
I'm coming from almost the same background and have heard the same arguments. I do understand that there are very valid reasons to put logic into the database. However, it depends on the type of application and the way it handles data which approach you should choose.
In my experience, a typical data entry app like some customer (or xyz) management will massively benefit from using an ORM layer as there are not so many different views at the data and you can reduce the boilerplate CRUD code to a minimum.
On the other hand, assume you have an application with a lot of concurrency and calculations that span a lot of tables and that has a fine-grained column-level security concept with locking and so on, you're probably better off doing stuff like that directly in the database.
As mentioned before, it also depends on the variety of views you anticipate for your data. If there are many different combinations of columns and tables that need to be presented to the user, you may also be better off just handing back different result sets rather than map your objects one-by-one to another representation.
After all, the database is good at dealing with sets, whereas OO code is good at dealing with single entities.
Reading these answers, I'm quite confused by the lack of understanding of database programming. I am an Oracle Pl/sql developer, we source control for every bit of code that goes into the database. Many of the IDEs provide addins for most of the major source control products. From ClearCase to SourceSafe. The Oracle tools we use allow us to debug the code, so debugging isn't an issue. The issue is more of logic and accessibility.
As a manager of support for about 5000 users, the less places i have to look for the logic, the better. If I want to make sure the logic is applied for ALL applications that use the data , even business logic, i put it in the DB. If the logic is different depending on the application, they can be responsible for it.
#DannySmurf:
It's not debuggable
Depending on your server, yes, they are debuggable. This provides an example for SQL Server 2000. I'm guessing the newer ones also have this. However, the free MySQL server does not have this (as far as I know).
It's not subject to source control
Yes, it is. Kind of. Database backups should include stored procedures. Those backup files might or might not be in your version control repository. But either way, you have backups of your stored procedures.
My personal preference is to try and keep as much logic and configuration out of the database as possible. I am heavily dependent on Spring and Hibernate these days so that makes it a lot easier. I tend to use Hibernate named queries instead of stored procedures and the static configuration information in Spring application context XML files. Anything that needs to go into the database has to be loaded using a script and I keep those scripts in version control.
#Thomas Owens: (re source control) Yes, but that's not source control in the same sense that I can check in a .cs file (or .cpp file or whatever) and go and pick out any revision I want. To do that with database code requires a potentially-significant amount of effort to either retrieve the procedure from the database and transfer it to somewhere in the source tree, or to do a database backup every time a minor change is made. In either case (and regardless of the amount of effort), it's not intuitive; and for many shops, it's not a good enough solution either. There is also the potential here for developers who may not be as studious at that as others to forget to retrieve and check in a revision. It's technically possible to put ANYTHING in source control; the disconnect here is what I would take issue with.
(re debuggable) Fair enough, though that doesn't provide much integration with the rest of the application (where the majority of the code could live). That may or may not be important.
Well, if you care about the consistency of your data, there are reasons to implement code within the database. As others have said, placing code (and/or RI/constraints) inside the database acts to enforce business logic, close to the data itself. And, it provides a common, encapsulated interface, so that your new developer doesn't accidentally create orphan records or inconsistent data.
Well, this one is difficult. As a programmer, you'll want to avoid TSQL and such "Database languages" as much as possible, because they are horrendous, difficult to debug, not extensible and there's nothing you can do with them that you won't be able to do using code on your application.
The only reasons I see for writing stored procedures are:
Your database isn't great (think how SQL Server doesn't implement LIMIT and you have to work around that using a procedure.
You want to be able to change a behaviour by changing code in just one place without re-deploying your client applications.
The client machines have big calculation-power constraints (think small embedded devices).
For most applications though, you should try to keep your code in the application where you can debug it, keep it under version control and fix it using all the tools provided to you by your language.