Sorry for the overly lengthy explanation below but I thought it'll be helpful in getting your ideas:
A few years ago I helped a friend keep track of the income and expenses of his family run guest house (around 10 rooms). The way it works is that his guest house is managed by staff who provide my fiend with regular updates on income and expenses (eg. rooms occupied, income, types and amounts of expenses, referral sources etc).
I built a quick and simple system in Excel with a user form, which the guest house manager uses to log the information. This info gets saved in the same workbook on another sheet. Since my friend lives in another city, the manager then emails the workbook to him on a monthly basis. My friend has a master copy of the same workbook and every time he receives his monthly update, he copies and pastes (appends) the data into his master copy.
The master workbook also has some dashboards built into it which work off the back of some pivots, formulas or the data directly; and provide my friend with key information like monthly profit figures, expense types, most commonly occupied rooms, avg length of stay etc.
All worked well but now, since the dataset has increased hugely, the workbook has started crashing. He has asked me to fix it, but I think it's time to change the underlying technology which would better suit the purpose. He doesn't want to approach a professional developer as the profits don't justify the costs. I can develop something but I'm not sure which is the best way to do it:
I'm good with SQL and Excel. My VBA knowledge is intermediate and I've just started learning Python. My initial thoughts are:
I could host a webpage on my friend's machine which his manager can remotely access to fill in information which then gets stored in the SQL server - Again, I've only got very limited knowledge around web page development and hosting. There could be an Excel file (or Tableau once I've learnt it) linked to it
and serve as a dashboarding platform. Essentially, instead of storing data in Excel it'll be stored in a SQL database.
If any of you could suggest a better/simpler way of implementing this that'll be greatly appreciated.
Many Thanks
Related
I'm working on creating a better way to view the history of items held at the public library I work at. I have been using a combination of the built in functions of SirsiDynix (our library software), Excel, and Autohotkey to extract, manipulate, and display the data. I am currently stuck on designing a way to view the change in status of an item over time since the system as it is only shows based on the last transaction. For example, if I have the following item:
0000519227318 005.54 WAL 101 EXCEL 2013 TIPS, TRICKS & TIMESAVERS Walkenbach, John, author WE-WH 2013 7 7/13/2013 6/29/2015 35
I can tell you it was created on 7/13/2013, last checked out on 6/29/2015, and has been checked out a total of 7 times. But I am unable to tell you anything about the length of those checkouts, or when they occurred, or if the book had been missing for a year in the middle of that time period.
With Autohotkey and the SirsiDynix Director's Station I have been able to create "daily snapshot" csv files that indicate where an item is every day. However I am having trouble figuring out how to consolidate that information. Originally I was planning to simply add an additional column to the end of the record every day so that after the general item information you would have a series of numbers listing the changing location. The coding I have for AHK to do this is somewhat slow and I'm still working on how I would best display it in Excel regardless. However it occurred to me that there may be a much better way to handle this that could fully automate the process.
So I'm asking whether there are suggestions for either a simple database system to use or an improvement to my current method that could assist me. The queries I plan to do are simply to be able primarily to type in an item number and have a chart display the status of an item, hopefully with something that could also show whenever the total checkouts has increased. I have been looking at Stock Market charts as examples but as many people with those seem to want open,close,hi,low values the responses they get seem beyond what I may need. Additional queries of items with the longest period of time on the shelf relative to total time would be useful although not initially required.
Any help as to what direction I may want to go would be appreciated. I have basic understanding of AutoHotKey and Excel, and I briefly used MySQL several years ago so I have a general feel of how a database can be used.
Not too familiar with your specific software or Autohotkey but for an efficient, secure, and scalable solution, consider any type of relational database management system (RDMS) including server level enterprise systems (some open-source or proprietary): Oracle, SQL Server, DB2, PostgreSQL, MySQL; or file level databases including SQLite and MS Access. One main thing is to try to move out of the concept of flatfile spreadsheets and applications. Excel is simply not a database and should be only used an end-use document for reporting or graphics/analytics using retrieved database content.
With a relational database you can maintain data across normalized related tables linked together by primary and foreign keys. Essentially, you want to build a Library Management System which can comprise of the following tables:
Items -unique list of Items ISBN, Catalog, Title, Author, Publisher, Category (Fiction, Nonfiction, Reference, Media); Primary Key: ItemID (autonumber/auto-increment)
Stock -copies of Items, Condition, Missing/Damaged Status, Cost, and Inventory Quantity; one-to-many relationship with Items table; Primary Key: StockID, Foreign Key: ItemID
Checkouts -infinite history of checkout record including Stock Item, CheckoutDate, CheckinDate, Notes; many-to-many relationship with Stock; Primary Key: CheckoutID, Foreign Key: StockID
Now with this schema, you can better manage each Item throughout its life cycle (new or arrived item, checked out periods, and retired/discard/basement stored) with easy queries for real-time reporting. Additionally, you can use any type of general purpose language (Java, C#, PHP, Python, Perl, VB) which can connect to any aforementioned RDMS to build the interface or tool for this backend system. A host of free consoles are also available including:
PhpMyAdmin (PHP/MySQL)
PgAdmin (PostgreSQL)
SQLite-manager (Firefox addin/SQLite)
Management Studio (SQL Server)
MS Access (Jet/ACE Engine and MS Office) the often misnomer to conflate MS Access as a database when it is actually a GUI that connects by default to a Windows technology, the JET/ACE SQL engine, where this default can be switched out to any aforementioned RDMS
Here, Excel can connect via ODBC/OLEDB VBA for reporting on checked out items status, history, and current shelf stock. Depending on the RDMS, you can even build triggers where as soon as an item is checked out, a record is added to CheckOut table or code it in you tool or scripts. Finally, outputs into txt, csv, xml, pdf reports, email attachments to co-workers, board, etc can be integrated.
I work for a research organization in India and have recently taken up with a program extending immunizations among poor rural communities. They're a fairly large organization but don't really have any IT infrastructure. Data reports on vaccine coverage, logistical questions, meeting attendance etc. come from hundreds of villages, go from pen-and-paper through several iterations of data entry and compilation, finally arriving each month at the central office as HUNDREDS of messy Excel sheets. The organization generally needs nothing more than simple totals and proportions from a large series of indicators, but doctors and high-level professionals are left spending days summing the sheets by hand, introducing lots of error and generally wasting a ton of time. I threw in some formulas and at least automated the process within single sheets, but the compilation and cross-referencing is still an issue.
There's not much to be done at the point of data collection...obviously it would be great to implement some system at the point of entry, but that would involve training hundreds of officials and local health workers; not practical at the moment.
My question: what can be done with the stack of excel sheets every month so we can analyze individually and also holistically? Is there any type of management app. or simple database we can build to upload and compile the data for easy analysis in R or even (gasp) excel? What kind of tools could I implement and then pass on to some relative technophobes? Can we house it all online?
I'm by no means a programmer but I'm an epidemiologist/stats analyst proficient in R and Google products and the general tools of a not-so-tech-averse millenial. I'd be into using this as an opportunity for learning some mySQL or similar, but need some guidance. Any ideas are appreciated...there has to be a better way!
step by step approach would be, first store the raw data from excel sheets and papers in structured database. Once data is maintained in DB you will have many tools to manipulate that data later.
Use any database like MySQL to store excel sheet ; Excel sheets or CSV files can be exported to database directly.
Later with simple database operations you can you can manipulate the data; you can use reports/web application/etc. to display and manage data.
and keep the good Work!
I am building an excel database, the size of the database will be approximately 1000000 rows of data per year.
I have ran into a problem. Data will be entered to the same database by approximately 12 different people, which causes a problem that the database file will be Read-only most of the day. What options do i have to allow all those people to enter data simultaneously?
Also all those people will be entering numbers based on different categories, which would require a different user interface for data input for each one of them.
Excel 2013 will allow multiple users to edit the same file simultaneously if (1) you have the business/enterprise version and (2) the file is kept on the OneDrive cloud (which is included in the business version).
All the users will be able to see the other users' edits in real time. Each user's cursor shows up as a different color.
Alternatively, consider using QUICKBASE.COM. It is an easy-to-use cloud-based database designed for multiple simultaneous users. If you can figure out how to use Excel, you'll be able to figure out how to use QUICKBASE. It's pretty straightforward.
If company security policy excludes a cloud solution, 1 million records is unwieldy for either Excel or Access. Check the system limits for both and you will see what you are up against.
If you are still forced to use Excel, each user can be set up with their own data entry template with VBA to copy their additions to a master Excel file. This would overcome the read only issue. If you are looking at updating records once added, you are opening a new can of worms from a coding perspective.
I've never designed a database before, but I've had experience programming in a few languages and assembler throughout college, as well as some web design, so I'm able to at least pick up what I need to know if I can be pointed in the right direction. One of the tasks of my job is to sort through some data that we've been collecting in the field, using a "sonde" which measures temperature, pH, conductivity, and other parameters. The device sits in a stream 24/7 (except for when we take it out and switch it with our other sonde every couple weeks, so that we can put in a newly calibrated one in the stream and retrieve the data from the one that was in the field). It collects data every 15 minutes or so, and has done so since 2007. Currently, all of our data is spread across multiple excel spreadsheets, and we have additional data from a weather station and another instrument that all gets compiled into quarterly documents. My goal is to design as simple of a database as possible with most of the functionality of a database like this: http://hudson.dl.stevens-tech.edu/hrecos/d/index.shtml. Ours would be significantly simpler as it is not live data (but would instead retrieve data from files that we upload once we'd finished handling the formatting and compilation of all our data). I would very much like the graphing ability on the site that the above database has, but I at least need to be able to select a range of data and select as many variables as I want within that time range and then be able to download a spreadsheet with the generated data (or at least a CSV file).
I realize this is a tough task, and as I have not designed a database before, I suspect it is very much an uphill task. However if I would be able to learn the things necessary to do this, and make it web-accessible, that would be a huge accomplishment and very much impress my boss. Any advice or tips to go off in the right direction would be very much appreciated.
Thanks for your help!
There are actually 2 parts to the solution you're looking for:
The database, which will store your data in a single organized place, and
The application, which is the interface used by people to interact with the database.
Basically, a database by itself is just a container. You need some kind of application which accept criteria from a user, pull the appropriate data meeting the criteria from the database, and display it to the user in a meaningful fashion - in this case, a graph or a spreadsheet.
Normally for web-based apps the database and application are two separate components. However, for a small app with a fairly small number of users, and especially for someone just starting out, you may want to consider an all-in-one solution like InfoDome, sort of like MSAccess for the web.
Either way, you're still going to need to learn about database design. There's many good tutorials out there, just do some searching. DatabaseAnswers.org has been useful for me. They have a set of tutorials as well as a large collection of sample database schemas.
I am working on a product (ASP.NET Web site) developed for educational institutions. There are around 20 educational inst. in my site. For each of them academic year start and end date varies. There are huge number of records in the database for attendance and results.
Now I need to show all previous years data (like attendance, results etc) whenever a student, teacher logs in. There are some reports which compares student performance in various academic years.
Now my problem is how to maintain that huge data ?
I wanted to go with 2 databases. 1 for current academic year, another for all previous yrs.
But my current year DB schema may change for enhancement. So whenever I move the current year data to archive database then it creates problem for me. Please suggest a good way to implement this.
Thanks,
seshu.
Have you thought about table partitioning? It allows you to rapidly move data through sliding windows - so that at the start of a new year, you slide last year's details into an archive partition. (You need to check the SQL Server edition you have to see whether it is enabled)
MSDN details:
http://msdn.microsoft.com/en-us/library/ms345146(SQL.90).aspx
If you want to keep two databases in sync, schema-wise, there are plenty of tools available for that. Here is mine, here is Red Gate's and here is Apex's. There are many more available, including one which comes with Visual Studio Team System Database edition (if you have that already - if you don't then one of the ones I have previously mentioned will be a lot cheaper).