So the solid consensus I got from the answers to this question: Editing a single line in a large text file
was that instead of using a text file I should create a database and store my data there. While I think this is a great idea, I don't know the first thing about databases, the programming languages used for databases, or how to use a database once I have set it up. Could you guys give me a shove in the right direction and point me an absolute noob tutorial that might help me with this?
UPDATE: Hey guys, so I was looking at mySQL and there are a whole bunch of versions! The Cluster CGE looks like the best one, and it says it is good for "real-time open source transactional database designed for fast, always-on access to data under high throughput conditions" which just about hits the nail on the head of what I need. It says commercial next to it though, so I don't know if I would have to pay some god awful fee for it. I tried it anyway, and it said I should have gotten a license already, and until I did I could only use it for 30 days. Im confused...
Can I get this version for free? If so, where do I get the license?
Is this version way overpowered for what I need? I need:
1. A storage medium through which I can store large amounts of data
2. Read and write from in real time with simultaneous access
3. Have two different "keys" (I think I'm using that right, I need to be able to search for entrees based on one of two criteria).
MySQL is a great choice, given your Python flair.
http://dev.mysql.com/tech-resources/articles/mysql_intro.html
Good luck!
Related
This is a bit text Heavy, I apologise. There is a TLDR below.
Intro to myself: I know you all probably heard the same story. Guy wants to make a game etc. Going into it semi blind, and dose not know much of what he's doing. I confess to be somewhat like this.
Next, I'm terrible with terminology, and have a brain like a fish when I comes to long nouns, this is largely due to my (mild) dyslexia, so please forgive me.
I did pre-university computer science at collage some 4 years back. Did some MySQL, some VB.net (windows forms are fun) and JavaScript. I've been following along loosely since then, but haven't done much since then so am quite rusty.
My level of skills can be called enthusiast at best, and muddle-headedly incompetent at worst. Been dabbling in C# since me and a friend use Unity a lot. To by honest college was a little trauma inducing for me and soured my ability to code (minor fear/stage fright, easily distracted), but to get over that I've decided to head out on my own and make something for myself.
Whether I succeed with the project or not, no matter how big or small the project is, isn't important, just so long as I'm creating something I'm satisfied with and I can progress in my skills and learn, that is all the matters. The rest is just a bonus.
With that in mind, I've formulated a project I hope and believe will retain my interest long enough to get some education in.
So without further a do, lets get to why I'm here.
I want to make a space game, a massive space game. Mostly, if not all procedural, but with realistic star counts spanning an entire galaxy. This can range from 1 thousand, to 100 trillion stars (I'd like to go as far as 150t for stress levels then cap at 100), much like it is in real life (Segue 2 - 1k, IC1101 - 100 trillion). Sounds nuts but games like Space Engine have already done this, and that was developed by one person.
Is this practical? I don't really care all that much if this is sort of game is practical or achievable, so long as its realistic-ish on this aspect and a few others. Plus, the thought of "this could work, you just have to be creative and put the effort in" attracts me.
So what's the issue? Why cant I use a normal save file? This issue comes when the game I'm trying to make is somewhat similar to a 4x. Its not exactly, but its close enough to know that there is a lot of information flying around, information that needs to be stored, synced and in some cases processed.
On a small scale, a single home/gaming computer can do this, so having a small single player is fine. However that's not what I want to achieve; The game I would like to create allows multiple people to play over long spans of time, across a galaxy, no matter what size that galaxy is.
This is where I started my research. I need a way to store this information and so I looked up various database engines like PostgreSQL, and others.
I think to myself, for what I'm doing as far as the world map is concerned, for the most part, a standard SQL databased should be fine... right? My confidence started shaking when looking up what game devs are using these day... NoSQL? What on earth is this? Why is google persistent on this? So I looked into it more. I then find out about NewSQL and realise I'm starting to run around in circles. The rabbit hole I did descend.
The flexibility of NoSQL is attractive, but when you see the lack of ACID in terms of fast and reliable queries (if I've read these articles right, I could be wrong), is a massive downside if your trying to look up 1 of a trillion stars or 1 of a quadrillion planets on the map in a fast manner. This is also not taking into account the fact a game is running along side it, sending and creating new data all the time.
Now I could be making a big issue about all this, as my scale of work might me too small to effect even the slow of systems, but how can I really know?
There's also the issue that, I don't know for sure, that somewhere down the line I'll need to create a new datatype for some new feature I create, and go mess things up in my existing test data because the database is old SQL, and you kinda have to deal with a ridged schema. What I need that flexibility to mess around and not worry a table somewhere is going to break, or that I have to bloat the DB with empty or null to non-applicable entries.
Why am I messing around with DB's before the actual game? Well if I cant even create the chess board before I can play chess, there's no point trying to invent chess really... I can make things look pretty later, I need the data first to manipulate.
~~TL:DR:~~
I leu of everything mentioned, I've compiled a small shopping list of a what I'm looking for to store my data:
- Must allow me to integrate into a single application, so my end users can deploy their own privet servers, in a relatively simple manner. What I mean by this, is more a licensing thing, and the easy of integration/use. Simply dragging up some opensource software and stuff it in pre-packaged with the server could lend me in some issues I believe?
- It should not be a cloud databases service, since for this project, I would like to allow "modding" of private servers.
- Relatively scalable, nothing amazing, just something simple yet effective. I like the idea to allow my users the choice to deploy the server software on more than one system with the option to either run a new map or chain to an existing as a slave/cluster thingy, for map chunk loading/dynamic load bearing for busy areas of the map.
- It needs to deal with large-ish amounts of information fast. I'm not expecting a lot of users, but I am expecting large volumes of data per user. Even if that's not the case its better to prepared rather than walk out with your pants down. I don't want to fix problems I can avoid.
- Forgiving enough to deal with a game development process filled with possible changes in data etc.
- Reliable data querying. The map, other than stellar drift and celestial orbits, should be relatively static. And it will become an issue if many users on the same map start looking up planets or stars all at the same time.
############
That's all I can think of for now, and will keep updated if I think of anything else.
So far, for all I could know (which I don't) a standard old style SQL could be the ticket and I'm just making work for myself, but with all the new technology out, like NoSQL and NewSQL, its hard to know what will be the better option for my circumstances. My experience with Databases can be pinned down to the classes done on MySQL and the research I've done recently, so I'm not familiar with the performance and abilities of other software.
So right now, my head is spinning, mystified from over research, and would like to know my options in specific databases engines on the market suitable for my situation.
So I take it to you, great people of stack overflow, please put me out my confusion!
So I'm familiar with programming, but am not exactly a professional one. I don't mind a bit of practice though, as I want to code my own web app for tracking my earnings and spendings. What are the optimal tools for that? By 'optimal' I mean not too complex ones, but at the same time not some simple visual constructors.
Here's what it needs:
A simple page for input;
2 databases, one for money earned and one for money spent;
The DBs are very basic XY ones;
The main feature is building graphs out of the database tables I select in real time.
And here's the question:
What is the best way to make the DBs? A friend of my told me that SQLs are way to complex for this task and suggested to put the data in JSON format. Will I be able to put this data in, let's say, an Excel file if I needed to?
What is the best language to choose so I can work with this data? I don't mind picking some library to ease the task. At first I thought of Python, but a friend suggested to use jQuery, which is more familiar to me.
Please don't suggest getting someone else's app, I'm interested in making this on my own. Hope I made the question clear enough, pleas let me know if it's not. Thanks for your time!
Disclaimer: I started working with spreadsheets in depth this week, prior to that it was basic usage. I've read the rules and this does relate to programming, it's just my ignorance of programming keeps me from asking a specific question. I'm new to this, I want to learn, I have to start somewhere.
I want to create two separate spreadsheet documents, one as a database for another. I want one to be able to query the other in a way similar to the VLOOKUP() function or something along those lines.
These are very large files hence the need for separate documents.
I am learning about scripting and think there might be a way there. If that's the case please appreciate that I literally started reading about scripts this morning and know nothing (yet) about them.
All I need to know is, if it's possible and what functions to use, I'll figure out how to use them. I just don't have a working knowledge of all the script functions, and a limited knowledge of spreadsheet functions.
The IMPORTRANGE() function is limited to 50 per spreadsheet, given how I want to use it, that is not enough. Unless you know a work around. That and I only want one cell of information at a time and it doesn't need to be displayed, just usable.
Also, efficiency is king since I'm working with such large amounts of data. I used to have almost 1500 VLOOKUP functions as I was building what I already have and that sucker was starting to bog down. Then I realized I didn't need a dynamic database for that aspect of the sheet. I killed about two thirds of them and it runs much better. I'd like to keep it that way, or at least try.
Finally I may have bitten off more than I can chew, but this has been a fun challenge for me, and I've met with success so far. Please don't dismiss me out of hand because I don't know the right questions to ask, or I'm trying to fit a square peg in a round hole, everyone has to start somewhere.
Thanks!
This is totally possible, though you will quickly find that spreadsheet functions are too cumbersome for this sort of operation.
With Google Apps Scripts you can query and write to and from multiple workbooks with ease. You would be working in Javascript, using javascript objects and arrays.
Start by reading the Google documentation and checking out their examples.
I'm building a website that will rely on heavy computations to make guess and suggestion on objects of objects (considering the user preferences and those of users with similar profiles). Right now I'm using MongoDB for my projects, but I suppose that I'll have to go back to SQL for this one.
Unfortunately my knowledge on the subject is high school level. I know that there are a lot of relational databases, and was wondering about what could have been some of the most appropriate for this kind of heavily dynamic cluster analysis. Also I would really appreciate some suggestion regarding possible readings (would be really nice if free and online, but I won't mind reading a book. Just maybe not a 1k pages one if possible).
Thanks for your help, extremely appreciated.
Recommondations are typically a graph like problem, so you should also consider looking into graph databases, e.g. Neo4j
We already have a database structure, but it is the structure without normalization and very confused and in need of change, but already has a large volume of stored data, for example, all financial data company, which finance department officials are afraid of losing.
We are undecided about remodeling the entire structure of the database and retrieve the most basic and all that is possible, or continue with the same model along with their problems.
I wonder if someone has made a change like this, if you can actually transfer the data to a new structure.
thanks
Before you do any thing I would BACKUP!!! Next I would create a new database with the ideas that you had in mind. Remember this is were all the real work should be once this is created it is hard to go back. Put a lot of thought in and make the design a bullet proof tiger to the design of your company. Next create some procedures to transform the data you have in the new database as you see fit. It would help if you mentioned the platform(s) you are using and mabey provide some generic examples
I have found SSIS packages work well for projects like this if you are using SQLSERVER. While you will need to still write your transforms out the packages make the work easier for others to see what is happening
Anything can be done by you the developer. However it might make business sense to check out various 3rd party tools. There are many out there and depending on exactly what you are doing you may benefit from doing some research
Yes, it's called "database conversion". It is a very common practice, but it must be done carefully and methodically, ideally by someone who has done many of them and knows the pitfalls. It is not to be done casually by any means. Moreover, it is not unusual in the financial sector to run the "old system" in parallel with the new system for a couple of months, to reconcile month-end reports, before saying goodbye to the old system. Running parallel is a PITA, and can only be done if all of the conversion programs are in place, but it's better to be safe than sorry when the numbers must be correct to the penny.
I had the same problem, the way I solved this is by re-design a new database, then I made a script that copies the data from the old schema to the new one. It's not an easy task because you need to take care of what you are copying from the old model to the new one but it's doable!
absolutely you can migrate the data to an new structure. The real question is 'how difficult (expensive/time consuming/reliable) will the migration be?' To answer that question one would have to know
The accuracy of the existing data - does it have gaps, duplication that disagrees with each other and no way to resolve, errors, etc.
What structure do you imagine going to and is this going to introduce complexity to the migration
the skill level of the person/team doing the migration
How long the migration will take and will the platforms be changing (either the live system being modified or the new system design changing)