Related
My lab is doing a lot of sequencing, but the way the sequences are documented makes it difficult to retrieve them or keep track of the data. I would like to create a database that has following features:
-A Graphical user interface to allow one to upload/retrieve/view data, and can incorporate links to quickly BLAST or analyse the sequences with other online tools.
-allows one to access it in the command line
-that has another section on the GUI that has records of what's in the lab, what needs to be ordered etc.
I wanted to know if there are general database templates I can adopt and modify to suit my lab needs? I have no experience in database design but have read about mySQL.
What are the first steps I should take in embarking on this project?
Thank you!
This is an interesting question and problem domain (one I now have expierence with btw). Your first step is to decide on a general architecture and then select technologies for this.
For the web/graphical side, there are lots of off the shelf components (I assume you are aware of tools like AntiSMASH, JBrowse, etc). But you will need to evaluate these. That is way outside the scope of the db side however.
On the database side, PostgreSQL performs admirably here. I have worked on a heavily loaded 10+TB db which was specifically storing sequencing data, BLAST reports, and so forth. If you add stuff like PostBIS on top of that, you get something quite functional.
A lot of the heavier portions of the industry however are using Hadoop because of the fact that the quantity of data available is increasing very rapidly but the amount of expertise required to make that work is also appropriately higher.
I need to implement yet another database website. Let's say roughly 5 tables, 25 columns, and (eventually) thousands to tens of thousands of rows. Easy data entry and maintenance are more important than presentation of the data to non-privileged users. It's a niche site, so performance is not a concern. We'll have no trouble finding somewhere to host it.
So: what's a good platform for this? Intituitively I feel that there ought to be some platform that allows this to be done with no code written - some web version of MS Access. Obviously I'm happy to code business rules, and special logic that distinguishes this from every other database app.
I've looked at Drupal (with Views) and it looks possible, but with quite a bit of effort. Will look at Al Fresco next. A CMS-y platform helps because then you can nicely integrate static content, you get nice styling, plugins, etc etc.
Really good data entry (tracking changes, logging, ability to roll back, mass imports...) would be great. If authorised users could do arbitrary SQL queries (yes, I know...) that would be a big bonus. Image management support a small bonus.
Django is what you are looking for. In fact, you could probably set up what you ask without much coding at all, just configuration.
Once complete, authorised users can add 'rows' with a nice but simple GUI, or, of course, you can batch import via database commands.
I'm a Python newbie, and I've already created 2 Django-based sites. I have created more than a dozen Drupal-based sites, and Django is easier and produces significantly faster sites.
Your need somewhat sits between two chairs : bespoke application and CMS-based. I'd advocate for the CMS approach, if and only if you feel the need for content structure customization will grow in the future, slowly removing the need for direct SQL queries.
I am biased since working with eZ Publish for many years now, but it satisfies the requirements you expressed natively :
Really good data entry (tracking changes, logging, ability to roll back, mass imports...)
[...] Image management support a small bonus.
An idea of the content edition feel can be watched here:
http://ez.no/Demos-Videos/eZ-Publish-Administration-Interface-Video-Tutorial
and you can download and test-drive eZ Publish Community Edition there : http://share.ez.no/latest
It is a PHP-based solution, strong professional community (http://share.ez.no), over 1100 add-ons available on http://projects.ez.no. The underlying libs are mostly relying on Apache Zeta Components, high-quality, robust set of PHP5 libraries.
Last note : the content model is abstracted, meaning you'd not have to create a new table everytime a new type of content should be stored : a simple content class definition from the administration interface, and the rest is taken care of, including the edition interface for the new content type. Might remove the need for hardcore SQL queries ?
Hope it helped,
Drupal can do most of what you need (I don't know of a module that will let you enter arbitrary SQL queries), but you will end up with some overhead of tables and modules you don't really need. It's up to you to decide if that's a problem or not. I don't think the overhead would hurt performance in your case.
The advantages of using Drupal would be the large community, the stability of the platform and the flexibility to add more functionality when needed. Also, the large user base ensures that most code has been tested rather well.
I highly recommend Drupal. It is very simple (also internally codebase is small and clean) it has dosens of possibilities and tremendous support. Once you start with Drupal you will never go to anything else.
Note that I'm not connected with Drupal staff, I've just created dosens of Drupas sites and many of them in just a minutes. My last one took me 2 hrs, see it here http://iPadDevZone.com
UPDATE #1:
It really depends on your DB schema complexity. The best case is that you just use CCK module (part of core now) and create your node type. Node is Drupal name for content. All you do is just web admin your node type fields (text, image, numbers, dates, custom, etc). Then, if user creates content with this node type he/she can enter all the fields which are stored in separate db table fields. This is however hidden for you - if you wish not to know about it - it is just a web gui. Then you choose how the node is presented, which properties as shown and where.
Watch videos in CCK resources section in the bottom of this page: http://drupal.org/project/cck
If you need to do some programming then it is also very easy to use so called PHP code sniplets which are entered as part of your content (node) and executed when the page is displayed.
Drupal has node revisions built in the core. You can see all the versions and roll back if you wish.
You can set the permissions in quite granular level so you can control what your users may or may not.
I would take a look at Symphony. I havn't been using it myself, but it seems like it's really easy to use and to customize!
http://symphony-cms.com/
Seems to me an online database system would be better than a CMS system.
So in addition to what's been posted above:
www.quickbase.com (by Intuit) - think around $150/mo
www.rollbase.com - check on price, full featured
www.rhythmdata.com - easy to set up, but don't think it's got the advanced features you're looking for.
Good luck!
B
I appreciate these answers, but most of them are really platforms that are much better at something else (eg, Drupal really is a CMS, and has some support for custom fields - but it's not at all easy). Since this is a brand new site from scratch, it doesn't really make sense to start with something that does custom database fields as an afterthought, I think.
The closest I've found is Zoho Creator. It really is like "MS Access for Web 2.0" - and even supports importing from Access. The pricing could get expensive though. It feels like it might eventually be quite constraining. I'm still evaluating.
Are there any other products like Zoho Creator?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I have evaluated several development tools for converting my informix SQL-based app. They are: Genero/4Js, FileMaker, Oracle APEX, VFP, Clarion and Access 2007. I have a CRUD pawnshop app (see video-demo www.frankcomputer.com) This app centers on customers who pawn, sell or buy merchandise. I need to have one CRUD multi-table form which displays one customer master on the top half with all of the customers associated items which they have pawned, sold or purchased on the bottom half. Can CRUD be accomplished from within one form in Access? The main reason I'm leaning towards Access is because of its integration with Excel, Word and other modules, plus many people have it and experienced using it, plus it's Microsoft. Can anyone who has developed apps with Access tell me if I can mimic my INFORMIX-SQL based app and what limitations does Access have? Also, can a touch-screen POS front-end like Microsoft Dynamics be used or are other POS application generators/rapid development systems available to re-write my current app?
I suspect if you ask Foxpro developer, they would tell you that's the best tool of choice.
And I'm sure if you ask a filemaker developer, they would tell you to choose their tool.
So much of the question is, for the most part if you ask an access developer, that developer would also answer yes.
I would be hard pressed to imagine that all of the above tools you mentioned above, all of them have the capability of displaying information from more than one table in a screen. That's pretty much a requirement for any development system today. So in a nutshell, you're really asking the wrong question here.
I don't think the question is do they have the capability of displaying information from more than one table. They all can do this. Perhaps a fair question would be how much work and how well does each product slice and dice together these multiple tables?
In access you place text boxes and controls on a form, and to display related data, you can you place a control called a sub-form control. This approach allows you to model this classic can typical master to child record table relationship, and do so without having to write one line of code.
And of course you're not limited to one to many, but you can actually insert two sub forms side by side, and have a one to many, and in turn have the 2nd sub form control display many more records from that second table.
Here's a screen shot of what I mean:
In above you have one main record at the top with information about donation date and event. On the left you have a list of people and their donation amount (one to many).
Then on the right side, for each person, you have the donation amount split out into multiple accounts. (and the green box shows red when the amounts don't balance).
So, the above creates that classic accounting problem that just about every accounting package from Quickbooks all the way up to hi end accounting packages have done from day one when splitting out funds to multiple accounts.
The above form has very little code in it, and most of the relationships and setups and filtering and displaying of the child records is all handled automatically by access.
So at the end of the day, I'm pretty much of the view that all of products you mention above are capable of modeling and developing these types of screens. And, they all will result in an screen and user experience that would be relatively similar to what you have now.
Now course I'm a biased towards access, and I believe that I can build screens like the above quicker, faster, and with less hassle less code and effort than most of the other products you mentioned .
However, at the end of the day what platform and tools you use and find as appropriate is certainly not going to be centered around the ONE QUESTION and ONE CONCEPT that you have need to display multiple pieces of information on a form for multiple tables. As mentioned, this is gonna be given for any modern development system, including web based development systems.
Other considerations and factors is what type of reports, and outputs to you need? Do you need his column are reports, or do you need to send an invoice style forms type report to a printer that's preprinted invoice forms. I think these are bigger questions then your current question.
The real question here's not can any modern development system display multiple pieces of data on a form, they all can. The REAL factors and issues here are what platforms, hardware requirements, and systems do you need the software to run on?
So the issue is will some of the locations have multiple users? Will some of the locations need secure backup or some type of encryption? How do you plan to issue bug fixes, and updates to the next great version of the software?
Other issues are how many developers will you have working on this. What kind of distribution method will you use for the software. What kind of support infrastructure will you have to give customers support and installing the software. So, this list goes on and on and all these issues dwarf the question about the ability to display multiple pieces of information on one form.
In addition to all of the above issues, you need to consider your own training and skill set in development of software. To really master any software development system, you need to invest a considerable amount of your own time to learn. While I think the access is a very good RAD (rapid application development) tool, I will actually say that access has a considerably larger learning curve then say that of VB6 for example.
Choosing a platform is very much like a marriage, you have to invest enormous amounts of your time (months, and even years) to really learn and become proficient at developing software with such a system .
If you're jumping into a new set of tools, then the following list of skill sets needs to be taken into consideration **:
Stage 1 Innocent (never heard of the product)
Stage 2 Aware (Has read an article about X)
Stage 3 Apprentice (has attended a three-day seminar)
Stage 4 Practitioner (ready to use X on a real project)
Stage 5 Journeyman (uses X naturally and automatically in his job)
Stage 6 Master (has internalized X, knows when to break the rules)
Stage 7 Expert (writes books, gives lectures, looks for ways to extend x)
One should NEVER attempt a project with a team consisting with Stage 3 or lower people. (**** Page-Jones, Meilir. "The Seven Stages of Expertise in Software Engineering", American Programmer, July-Aug 1990).
So you just can't jump into a new tool and expect to be proficient at developing complex applications. I have an article here about converting a legacy application into ms access .
There are some great lessons in this article:
Notes on Conversion of a Pick (Multi-Value database) Application to a Relational database system.
http://www.members.shaw.ca/AlbertKallal/Articles/fog0000000003.html
Good luck in whichever platform you choose.
I would like to know what should be a good option to implement a system like that:
The system will be use by teachers in a school. They can login and see a list of the pupils that they have in the current class/subject. In this list they could mark the pupils that are not attending the class, and some other options...
The system should also give the option to parents to justify the absences of their childs.
The secretary can access to print some reports related.
This is more or less the system I would like to create.
As you see, doesn't look so complex, but I would like to have an idea about the tools I should use.
As I said before, this is not HOMEWORK, so I hope the next answers take this in consideration.
I'm developing this real system, but I'm confuse about what to use. I was thinking in mysql and php. So I would like to know, even if looks so easy for you, the steps to follow or the different alternatives that I should consider.
UPDATE:
I'm talking with my tutor, and we will go to the company that provides services with the servers, so I can ask them what we need. So normally I thinks the best is to ask for a phpmyadmin (cos we already have sql server). Should I ask sth else that I will need or will make it easier?
Greets
I would use a web front end, with a database containing tables for Classes (which contains the teacher), a table for students, a table hanging off that with details of classes the student takes, and a table for absence details.
You can then set it up so that when a teacher accesses the system they can see the classes they teach, then they select a class it will show the students who take that class. Clicking on a student's name will take you to page where you can enter the absence.
It's a fairly simple set up, but I've developed something similar and it works very well.
By and large there are no tools for developing a certain kind of application. Tools are graded by complexity of application, choice of OS / platform and what programming languages you know.
So if you are a Windows kind of guy you might choose to use SQL Server Express, VB.Net and maybe Visio. Whereas if you're a FOSS fan then perhaps MySQL, PHP and Dia would suit you better. (Of course many FOSS products - including the three I mention - are available in versions which run in Windows, so it is not necessarily clear cut).
This is part of your internship? If so, you ought to use something which is compatible with what your employers use.
edit
Okay, so your internship is with a school, and they don't do much in the way of IT. In that case, if your skillz are mad enough I would suggest you go for the LAMP stack, as it is cheaper.
I would be careful about looking to roll this out to parents. That will elevate your application to a whole new world of concerns: unauthorised access to citizens' private data is a big deal right now, so this will land you with a lot of complexities you do not want to have to handle. Build an app for the teachers only. Get that working. Then if you have time on your hands look at deploying it to a wider audience.
A simple GUI and MySQL Backend should suffice.
Create a List of pupils, put 4 buttons next to each pupil (there, missing, excused, late) and just store it in a database for each class.
Report generation isn't that hard either, just execute the necessary selects (something like: select pupil_id, count(*) from class where state='missing') whenever someone clicks the reports->[...] menu items.
Of course it greatly depends on which other specs exist, but since this sounds like homework i think this should suffice.
OK, so you'll need access from multiple clients, so web based would be ok. You'll need persistent data that is rather primitive. So you might use almost any persistence solution. A RDBMS would be the most common choice, but CouchDB or other BigTable'ish implementations would also suffice.
Other than that you're free in your choice. Java, PHP, Ruby, C++ use whatever you like. Mysql, postgres, couchdb, xml flatfile are all valid choices for persistence.
Without more informations it's not possible to pinpoint a set of technologies ;-)
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have used Relational DB's a lot and decided to venture out on other types available.
This particular product looks good and promising: http://neo4j.org/
Has anyone used graph-based databases? What are the pros and cons from a usability prespective?
Have you used these in a production environment? What was the requirement that prompted you to use them?
I used a graph database in a previous job. We weren't using neo4j, it was an in-house thing built on top of Berkeley DB, but it was similar. It was used in production (it still is).
The reason we used a graph database was that the data being stored by the system and the operations the system was doing with the data were exactly the weak spot of relational databases and were exactly the strong spot of graph databases. The system needed to store collections of objects that lack a fixed schema and are linked together by relationships. To reason about the data, the system needed to do a lot of operations that would be a couple of traversals in a graph database, but that would be quite complex queries in SQL.
The main advantages of the graph model were rapid development time and flexibility. We could quickly add new functionality without impacting existing deployments. If a potential customer wanted to import some of their own data and graft it on top of our model, it could usually be done on site by the sales rep. Flexibility also helped when we were designing a new feature, saving us from trying to squeeze new data into a rigid data model.
Having a weird database let us build a lot of our other weird technologies, giving us lots of secret-sauce to distinguish our product from those of our competitors.
The main disadvantage was that we weren't using the standard relational database technology, which can be a problem when your customers are enterprisey. Our customers would ask why we couldn't just host our data on their giant Oracle clusters (our customers usually had large datacenters). One of the team actually rewrote the database layer to use Oracle (or PostgreSQL, or MySQL), but it was slightly slower than the original. At least one large enterprise even had an Oracle-only policy, but luckily Oracle bought Berkeley DB. We also had to write a lot of extra tools - we couldn't just use Crystal Reports for example.
The other disadvantage of our graph database was that we built it ourselves, which meant when we hit a problem (usually with scalability) we had to solve it ourselves. If we'd used a relational database, the vendor would have already solved the problem ten years ago.
If you're building a product for enterprisey customers and your data fits into the relational model, use a relational database if you can. If your application doesn't fit the relational model but it does fit the graph model, use a graph database. If it only fits something else, use that.
If your application doesn't need to fit into the current blub architecture, use a graph database, or CouchDB, or BigTable, or whatever fits your app and you think is cool. It might give you an advantage, and its fun to try new things.
Whatever you chose, try not to build the database engine yourself unless you really like building database engines.
We've been working with the Neo team for over a year now and have been very happy. We model scholarly artifacts and their relationships, which is spot on for a graph db, and run recommendation algorithms over the network.
If you are already working in Java, I think that modeling using Neo4j is very straight forward and it has the flattest / fastest performance for R/W of any other solutions we tried.
To be honest, I have a hard time not thinking in terms of a Graph/Network because it's so much easier than designing convoluted table structures to hold object properties and relationships.
That being said, we do store some information in MySQL simply because it's easier for the Business side to run quick SQL queries against. To perform the same functions with Neo we would need to write code that we simply don't have the bandwidth for right now. As soon as we do though, I'm moving all that data to Neo!
Good luck.
Two points:
First, on the data I've been working with the past 5 years in SQL Server, I've recently hit the scalability wall with SQL for the type of queries we need to run (nested relationhsips...you know...graphs). I've been playing around with neo4j, and my lookup times are several orders of magnitude faster when I need this kind of lookup.
Second, to the point that graph databases are outdated. Um...no. Early on, as people were trying to figure out how to store and lookup data efficiently, they created and played with graph and network style database models. These were designed so the physical model reflected the logical model, so their efficiency wasnt that great. This type of data structure was good for semi-structured data, but not as good for structured dense data. So, this IBM dude named Codd was researching efficient ways to arrange and store structured data and came up with the idea for the relational database model. And it was good, and people were happy.
What do we have here? Two tools for two different purposes. Graph database models are very good for representing semi-structured data and the relationships between entities (that may or may not exist). Relational databases are good for structured data that has a very static schema, and where join depths do not go very deep. One is good for one kind of data, the other is good for other kinds of data.
To coin the phrase, there is no Silver Bullet. Its very short sighted to say that graph database models are out of date and to use one gives up 40 years of progress. That's like saying using C is giving up all the technological progress we've gone through to get things like Java and C#. That's not true though. C is a tool that is needed for certain tasks. And Java is a tool for other tasks.
I've been using MySQL for years to manage engineering data, and it worked well, but one of the problems we had (but didn't realise we had) was that we always had to plan the schema up-front. Another problem we knew we had was mapping the data up to domain objects and back.
Now we've just started trying out neo4j and it looks like it is solving both problems for us. The ability to add different properties to each node (and relation) has allowed us to re-think our entire approach to data. It is like dynamic versus static languages (Ruby versus Java), but for databases. Building the data model in the database can be done in a much more agile and dynamic way, and that is dramatically simplifying our code.
And since the object model in code is generally a graph structure, mapping from the database is also simpler, with less code and consequently fewer bugs.
And as a additional bonus, our initial prototype code for loading our data into neo4j is actually performing faster than the previous MySQL version. I have no solid numbers on this (yet), but that was a nice additional feature.
But at the end of the day, the choice probably should be based mostly on the nature of your domain model. Does it map better to tables or graphs? Decide by doing some prototypes, load the data and play with it. Use neoclipse to look at different views of the data. Once you've done that, hopefully you know if you're on to a good thing or not.
Here is a good article that talks about the needs that non relational databases fill: http://www.readwriteweb.com/enterprise/2009/02/is-the-relational-database-doomed.php
It does a good job at pointing out (aside from the name) that relational databases arent flawed or wrong, its just that these days people are starting to process more and more data in mainstream software and web sites, and that relational databases just wont scale for these needs.
I am building an intranet at my company.
I am interested in understanding how to load data that was stored in tables (Oracle, MySQL, SQL Server, Excel, Access, various random lists) and loading it into Neo4J, or some other graph database. Specifcally, what happens when common data overlaps existing data already in the system.
Yes, I know some data is best modeled in RDBMS, but I have this idea itching me, that when you need to superimpose several distinct tables, the graph model is better than the table structure.
For instance, I work in a manufacturing environment. There is a major project we are working on and because of the complexity, each department has created a seperate Excel spreadsheet that has a BOM (Bill Of Materials) hierarchy in a column on the left and then several columns of notes and checks made by individuals who made these sheets.
So one of the problems is merging all these notes together into one "view" so that someone can see all the issues that need to be addressed in any particular part.
The second problem is that an Excel spreadsheet sucks at representing a hierarchial BOM when a common component is used in more than one subassembly. Meaning that, if someone writes a note about the P34 relay in the ignition subassembly, the same comment should be associated with the P34 relays used in the motor driver subassembly. This won't occur in the excel spreadsheet.
For the company intranet, I want to be able to search for anything easily. Such as data related to a part number, a BOM structure, a phone number, an email address, a company policy, or procedure. I want to even extend this to manage computer hardware assets, and installed software.
I envision that once the information network starts to get populated you can start doing cool traversals such as "I want to write an email to everyone working on the XYZ project". People will have been associated with the project because they will be tagged as creating and modifying the data within the XYZ project. So by using the XYZ project as a search key, a huge set with everything related to the XYZ project will be created. Including links to people who built the XYZ project. The people links will connect to their email addresses. So by their involvement in the XYZ project, they will be included in my email. This is in stark contrast to some secretary trying to maintain a list of people work on the project. We generate a lot of lists. We spend a lot of time maintaining lists and making sure they are up to date. And most of it doesn't add any value to our products.
Another cool traversal could report all the computers that have a certain piece of software installed, by version. That report could be used to generate tasks to remove extra copies of old software and to update people who need to have the latest copy. It would also be useful for license tracking.
might be a bit late, but there is a growing number of projects using Neo4j, the better known ones listed at Neo4j . Also NeoTechnology, the company behind Neo4j, has some references at their customers page
Note: I am part of the Neo4j team