Suppose you have an database with the largest tables containing about 200.000 rows, and frequently modified. The client wants Excel to connect via ODBC to the database, and work as a frontend to manage the data. The data should be modifiable by up to 25 users concurrently.
My first instinct would be to recommend something else, for example a web frontend. But suppose the client insists on the Excel solution, would you regard it as doable, and what pitfalls would you see in it?
My doubts would be about:
data integrity (how to manage users modifying same data at the same time)
large amounts of data moved unnecessarily (when opening the Excel workbook I imagine that the whole database has to be transferred)
security (showing only parts of data to appropriate users in a secure way would be challenging - see previous point)
using a tool (Excel) for something, in which it doesn't excel (pardon the pun)
I do this all the time. No you don't have to bring in the whole database or even the whole table. I use ADO and VBA and send SQL statements via the Command object. For example, I have a royalty database with an Excel front end.
The user types in an invoice number and a SELECT statement retrieves that one record and populates some custom classes. The user enters/modifies some data and clicks 'Save'. Then the class has a method that writes the record back to the database with and UPDATE or INSERT depending on the situation.
At the end of the month, the user enters a date range and retrieves some records into a report, again just a SELECT statement filling some classes and outputting to a sheet.
Use Transactions so you can roll back if you hit any record locking problems, but with 25 users you probably won't.
At first glance I would suggest treating Excel a bit like a web page, that is, pull only the required data and use a specific form for editing that updates one record at a time via ADO. You need only lock a single record and that for the fraction of time it takes to update. You can check whether or not the record has changed since it was opened for editing and users can be told that they cannot open a record for editing and then leave it sitting around in the edit form or they may lose the changes.
It is usually quite unlikely for such a small group to need to change the same record at the same time.
I do not think you will have much trouble with 25 concurrent users.
Related
I am now working on a project which requires to show the transaction history of one customer and if the product customer buys is under warranty or not. I need to use the data from the current system, the system can provide Web API, which is a .csv file. So how can I make use of the current system data?
A solution I think of is to download all the .csv files and write scripts to insert every record into the database I built which contains the necessary tables and relations to hold the data I retrieve. Then I can have a new database which I want. because I never done this before so I want know if it is feasible?
And one more question would be, if I should store the data locally or use a cloud database like Firebase?
High-end databases like SQL Server and Oracle come with utilities that allow you to read directly from a csv file. Check the docs. Having done this many times, the best procedure I found was to read the file into one holding table. This gives you the chance to examine the data and find any unexpected quirks or missing fields. This allows you to correct the data, where possible.
Then write the scripts to move the data from the holding table into the proper tables you have designed. This must be done in a logical manner. For example, move the customer data before the buy transactions. Thus any error messages you get will not be because you tried to store a transaction before you stored the customer. (You will have referential integrity set up, yes?) This gives you more chances to correct or adjust the data or just identify problems more or less at your leisure.
Whether or not to store the data in the cloud is strictly according to the preferences of your employer.
I am building an excel database, the size of the database will be approximately 1000000 rows of data per year.
I have ran into a problem. Data will be entered to the same database by approximately 12 different people, which causes a problem that the database file will be Read-only most of the day. What options do i have to allow all those people to enter data simultaneously?
Also all those people will be entering numbers based on different categories, which would require a different user interface for data input for each one of them.
Excel 2013 will allow multiple users to edit the same file simultaneously if (1) you have the business/enterprise version and (2) the file is kept on the OneDrive cloud (which is included in the business version).
All the users will be able to see the other users' edits in real time. Each user's cursor shows up as a different color.
Alternatively, consider using QUICKBASE.COM. It is an easy-to-use cloud-based database designed for multiple simultaneous users. If you can figure out how to use Excel, you'll be able to figure out how to use QUICKBASE. It's pretty straightforward.
If company security policy excludes a cloud solution, 1 million records is unwieldy for either Excel or Access. Check the system limits for both and you will see what you are up against.
If you are still forced to use Excel, each user can be set up with their own data entry template with VBA to copy their additions to a master Excel file. This would overcome the read only issue. If you are looking at updating records once added, you are opening a new can of worms from a coding perspective.
I am currently working on a project which has basically 2 users who will be the end users. Till now, we were using access as the front end but since the code was designed by an amateur, it frequently runs into some kind of memory problems and stops functioning on a regular basis. Now we have two options: redesign the access front end from scratch or probably try and build an excel interface.
All the data currently is stored in the sql server and we were only using access to input some text into it which was stored in access locally.
Please do let me know if you have any suggestions about creating excel as the front end. The amount of data stored in the sql server is about 6 table and 2 of them are really huge arnd 150,000 rows and 350 columns. these tables can be linked by unique keys. but ultimately for every unique key we currently have 1 row for it and end up writing comments on it and store and update the comments for further use. Ultimately the end user selects only about 300 rows and ends up editing it and closes it and moves to another 300 rows.
Also, since I am the only person who will be handling this, and the time is also very limited I am looking for a solution which will not be taking too much of my time to maintain the database later.
How is the Access database accessed? i.e. is it one file on a network share that is then used by both users? If so, that is your issue-- especially if both users try to use it simultaneously. That is almost always the cause of corruption on a mdb file. I absolutely agree with Chris that using Excel as a front end is a very bad way to go, however I don't know as you need to go all out with VB.Net. Access is very capable for simple applications when it's implemented correctly.
If multiple people are accessing an Access database, it needs to be "split" i.e. have one mdb file with all the tables and data (the backend) and another file with the user interface (the front end). The backend is placed on a network server or common location (but not dropbox) and is never directly opened. The front end is placed on each user's computer and is linked to the backend. If the database is not currently split, there is a wizard available that can help you do that.
You should be able to also link the same front end to anything in the SQL Server.
My wife works for a medium sized retail chain. Managers from each of the 80 outlets have to fill in one row of performance info for each of their staff (900 in all), but aren't allowed to see the data of other stores' staff.
My wife currently manages this with lots of spreadsheets, because each month the executive change what they want to collect, and their IT team don't have the resources to update their SAS system. She has to manually compile all the data into 1 spreadsheet for analysis which is time consuming and error prone. She's recently gone from having to do this for 20 outlets to 80 outlets and thinks she must be an easier way.
Is there a simple form based system, that can leverage what is already installed (microsoft office and lotus but not MSAccess), or can be run from a network drive. Cloud apps are banned. Excel's security is all wrong. Can word form templates append to a shared data source? Any ideas?
TIA
You could have a single table with all the data, then create 'shadowtables' on this table for each individual store.
in MySQL this would probably be either a 'partition table' (I've never used this so not sure how it works) or the use of temp tables.
You would then need to implement a method whereby when a user logs in at a given location (IP address) a trigger would create the temp table, then populate it with the relevent data for the store at that IP address.
An alternative (probably easier too) would be to have a specied table for each store, then grant users specific priveleges on each table you create. Again you'll need trigers to either populate a single 'master table' with info as it is updated, or you will just send a
select * from outlet1, outlet2... outlet80
again you may decide to create a temp table from the above select, and implement a custom script to create it only when required.
In fact that is probably how I would do it.
Then in you web interface have a button to create the temp table, and display it to the current user (provided they have the required priveleges to view all the tables of course).
I don't know for certain if Lotus is able to implement this, I don't know about its 'database' solution. I know that to do something similar in Access isn't that hard, the only downside would be needing to handle user identification (which Access doesn't do natively), again I don't know about Lotus.
In my experience the 'flat file database systems' don't generally handle user permisions in a native fashion, it is put onto the interface development to hand this.
I'm not sure how helpful the answer is, but it may take you a little way to a solution (even if you end up going for a server/client dbms system)
You can use Lotus for this. A simple start for you:
Create a database with one form and one view
On the form add whatever fields you want but also add a computed-when-composed multi-value field of type "Readers" with formula:
"[Admin]" : #Name( [CANONICALIZE];#userName)
With the exception of those with an Admin role (e.g., your wife), the view will display to each user only the records that the user created. The users will have to create one record per row.
Alternatively you could create an agent in the database that reads the data from an Excel file and builds the documents (records) with the READERS field's value computed as the documents are created.
If that's the route you want to take post a reply here and I'll post some code to (i) prompt a user to select an excel file, (ii) read the excel file data into lotus notes, (iii) implement a READERS field to see that documents are kept confidential between the creator and the Admin role people.
Hope that helps.
I am developing a web application in Java EE technologies(Spring, Spring MVC, Hibernate). In this application I am parsing an Excel file and need to add these data to a sql server database.
Before adding these data to the database I need to get the user input for each row in the Excel file whether they really want to add these data to the database.
I can do this something like this:
First save the data to a table(table_tmp)
Then display the data to the user and from their input add it to the actual table and remove from the temporary table.
But I think there will be a better solution (some kind of temporary storage that I can delete after getting user input).
Can I use some NoSQL solution for this?
Why do you need to store it in the first place? You want to allow them to upload that excel, and come back way after their session expired and select rows they want? Do you really have to make it persistent?
If yes, then do you have any problems with your current setup? If not then you'll be introducing another external component that needs to be administered and that you have to interface with (what happens if mongodb is down, not enough disk space, connection timed out, ...) just to keep a temporary file.
But if you still want to do it, then you might first consider something really simple (and fast), like memcached - a key-value in-memory storage.