I'm working with spark-sql (with Scala) and I found out that there are so many limits when I want to manipulate array data inside a Dataframe for doing something more, without passing to RDD API and reconstruct the schema to come back to spark-sql Dataframe abstraction.
An example of my use case is: "given a nested structure, not just explode the records, but also merge and solve conflicts between records inside the array".
For example, consider a customer with the array of his orders, considering that orders could be ovelapped and for a customer could have more than one order with the same id: I want that in the exploding phase orders conflict between overlapped data will be solved.
Actually I can do that with rdd, but my code seems so tricky and I would like to do that without go at rdd level. Is there a way or a library that extends spark-sql for write clean code and do operation with arrays in a linear mode?
Another use case could be "I want to explode just the array that is a leaf in my nested Dataframe".
Thanks in advance for your help.
Just a question regarding NoSQL DB. As far as I know, operations are done by the app/website outside the DB. For instance, if I need to add an value to a list, I need to
download the intial list
add the new value in the list on my device
upload the whole updated list.
At the end, a lot of data is travelling (twice the initial list) with no added value.
Is there any way to request directly the DB for simple operations like this?
db.collection("collection_key").document("document_key").add("mylist", value)
Or simply increment a field?
Same for knowing the number of documents in a collection: is it needed to download the whole set of document to get the number ?
Couple different answers:
In Firestore, many intrinsic operations can be done "FieldValues", such as increment/decrement (by supplied value, so really Add/subtract). Also array unions, field deletes, etc. Just search the documentation for FieldValue. Whether this is true for NoSQL in general, I can't say.
Knowing the number of documents, on the other hand. is not trivially done in Firestore - but frankly, I can't think of any situations other than artificially contrived examples where you would need to know. Easy enough to setup ways to "count" documents as you create/delete them, and keep that separately, if for some reason you find yourself needing it.
Or were you just trying to generically put down NoSQL as a concept?
I'm a student and starting to relearn again the basics of programming.
The problem I stated above starts when I have read some Facebook posts that most of the programmers use arrays in their application and arrays are useful. And I started to realize that I never use arrays in my program.
I read some books but they only show the syntax of array and didn't discuss on when to apply them in creating real world applications. I tried to research this on the Internet but I cannot find any. Do you guys have circumstance when you use arrays. Can you please share it to me so I can have an idea.
Also, to clear my doubts can you please explain to me why arrays are good to store information because database can also store information. When is the right time for me to use database and arrays?
I hope to get a clear answer because I have one remaining semester before the internship and I want to clear my head on this. I do not include any specific programming language because I know most of the programming language have arrays.
I hope to get an answer that can I can easily understand.
When is the right time for me to use database and arrays?
I can see how databases and arrays may seem like competing solutions to the same problem, but you're comparing apples and oranges. Arrays are a way to represent structured data in memory. Databases are a tool to store data on disk until you need to retrieve it.
The question you pose is kind of like asking: "When is the right time to use an integer to hold a value, vs a piece of paper?" One of them is a structural representation in memory; the other is a storage tool.
Do you guys have circumstance when you use arrays
In most applications, databases and arrays work together. Applications often retrieve data from a database, and hold it in an array for easy processing. Here is a simple example:
Google allows you to receive an alert when something of interest is mentioned on the news. Let's call it the event. Many people can be interested in the event, so Google needs to keep a list of people to alert. How? Probably in a database.
When the event occurs, what does Google do? Well it needs to:
Retrieve the list of interested users from the DB and place it in an array
Loop through the array and send a notification to each user.
In this example, arrays work really well because users form a collection of similarly shaped data structures that needs to be put through a similar process. That's exactly what arrays are for!
Some other common uses of arrays
A bank wants to send invoice and payment due reminders at the end of the day. So it retrieves the users with past due payments from the DB, and loops through the users' array sending notifications.
An IT admin panel wants to check whether all critical websites in a list are still online. So it loops through the array of domains, pings each one and records the results in a log
An educational program wants to perform statistical functions on student test results. So it puts the results in an array to easily perform operations such as average, sum, standardDev...
Arrays are also awesome at keeping things in a predictable order. You can be certain that as you loop forward through an array, you encounter values in the order you put them in. If you're trying to simulate a checkout line at the store, the customers in a queue are a perfect candidate to represent in an array because:
They are similarly shaped data: each customer has a name, cart contents, wait time, and position in line
They will be put through a similar process: each customer needs methods for enter queue, request checkout, approve payment, reject payment, exit queue
Their order should be consistent: When your program executes next(), you should expect that the next customer in line will be the one at the register, not some customer from the back of the line.
Trying to store the checkout queue in a database doesn't make sense because we want to actively work with the queue while we run our simulation, so we need data in memory. The database can hold a historical record of all customers and their checkout outcomes, perhaps for another program to retrieve and use in another way (maybe build customized statistical reports)
There are two different points. Let's me try to explain the simple way:
Array: container objects to keep a fixed number of values. The array is stored in your memory. So it depends on your requirements but when you need a fixed and fast one, just use array.
Database: when you have a relational data or you would like to store it in somewhere and not really worry about the size of the objects. You can store 10, 100, 1000 records to you DB. It's also flexible and you can select/query/update the data flexible. Simple way to use is: have a relational data, large amount and would like to flexible it, use database.
Hope this help.
There are a number of ways to store data when you have multiple instances of the same type of data. (For example, say you want to keep information on all the people in your city. There would be some sort of object to hold the information on each person, and you might want to have a data structure that holds the information on every person.)
Java has two main ways to store multiple instances of data in memory: arrays and Collections.
Databases are something different. The difference between a database and an array or collection, as I see it, are:
databases are persistent, i.e. the data will stay around after your program has finished running;
databases can be shared between programs, often programs running in all different parts of the world;
databases can be extremely large, much, much larger than could fit in your computer's memory.
Arrays and collections, however, are intended only for use by one program as it runs. Your program may want to keep track of some information in order to do its calculations. But the data will be in your computer's memory, and therefore other programs on other computers won't be able to access it. And when your program is done running, the data is gone. However, since the data is in memory, it's much faster to use it than data in a database, which is stored on some sort of external device. (This is really an overgeneralization, and doesn't consider things like virtual memory and caching. But it's good enough for someone learning the basics.)
The Java run time gives you three basic kinds of collections: sets, lists, and maps. A set is an unordered collection of unique elements; you use that when the data doesn't belong in any particular order, and the main operations you want are to see if something is in the set, or return all the data in the set without caring about the order. A list is ordered, though; the data has a particular order, and provides operations like "get the Nth element" for some number N, and adding to the ends of the list or inserting in a particular place in the list. A map is unordered like a set, but it also attaches keys to the data, so that you can look for data by giving the key. (Again, this is an overgeneralization. Some sets do have order, like SortedSet. And I haven't even gotten into queues, trees, multisets, etc., some of which are in third-party libraries.)
Java provides a List type for ordered lists, but there are several ways to implement it. One is ArrayList. Like all lists, it provides the capability to get the Nth item in the list. But an ArrayList provides this capability faster; under the hood, it's able to go directly to the Nth item. Some other list implementations don't do that--they have to go through the first, second, etc., items, until they get to the Nth.
An array is similar to an ArrayList, but it has a different syntax. For an array x, you can get the Nth element by referring to x[n], while for an ArrayList you'd say x.get(n). As far as functionality goes, the biggest difference is that for an array, you have to know how big it is before you create it, while an ArrayList can grow. So you'd want to use an ArrayList if you don't know beforehand how big your list will be. If you do know, then an array is a little more efficient, I think. Although you can probably get by mostly with just ArrayList, arrays are still fundamental structures in every computer language. The implementation of ArrayList depends on arrays under the hood, for instance.
Think of an array as a book, and database as library. You can't share the book with others at the same time, but you can share a library. You can't put the entire library in one book, but you can checkout 1 book at a time.
I run a report daily which lists various files based on a particular identifier which indicates what is within the file. In the hopes of expediting my workflow, I'd like to set these files in a simple report by identifier, along with how frequently each identifier occurred, and organize this list from most to least frequent. Below is what I'm working with:
This is a VBA based terminal emulator with limited object library references, so Excel isn't an option
There's usually anywhere from 0-200 separate identifiers on any report, and they don't always appear on each. Since there are thousands of possible identifiers that could appear, I'd rather just have the macro list what it finds than look for them specifically.
I use a bit of code utilizing ADOdb streaming to write similar reports to a .txt file, which I would like to use here. I'm sure I can make that happen after the data is sorted, but felt it worth mentioning in case the nature of how the filtered/organized list would be utilized would affect how it is addressed.
I'm a novice, self-taught programmer with a particular fear of arrays (and I know this is going to come down to arrays...). So I don't necessarily know what commands and options are available to me in coding. (ie, a lot of what I've read involved possibly using dictionaries or collections, but I'm unsure I even have these available, let alone how to use them.)
I'd rather avoid creating multiple arrays, and creating multiple For loops if possible. It seems like a two dimensional dynamic array SortList(category, frequency) would be the route to go, but I can't find a way to filter unique values into it, while counting any of the values it has already found, and then sorting it afterwards.
I found a REALLY nifty bit of code on one site that can filter unique values without using a loop... (credit to where it's due: http://www.jpsoftwaretech.com/finding-values-in-an-array-without-looping/), but it seems it can only work for one dimensional arrays:
Function IsInArray(arr As Variant, valueToFind As Variant) As Boolean
IsInArray = (UBound(Filter(arr, valueToFind)) > -1)
End Function
Does anyone have any suggestions on how I should try and tackle this dilemma?
8/14 - Suppose this is also important to note: these are not individual files one can access in a folder somewhere; the data I'm working with is presented simply as a list. It might be better to state what I'm doing is having these identifiers read as strings. It's the strings I want to count, sort, and organize... it seems like something that can be done with beginner's programming techniques, I just can't seem to find a way to do it without extraneous arrays or unnecessary nested loops and if/then conditions. Thoughts?
If you are familiar with ADO/DAO you could try querying the files ensuring the field in question is in GROUP BY and outputting the result of a COUNT() of the identifier.
Alternatively you could use a combination of collections and arrays.
GSeg illustrates quite well how you could utilise a collection of arrays here.
Collection of Arrays
By using a collection you can add an identifier as a Key and its count as the second array element.
As you cant duplicate keys in collections, if you use the identifier as a key it will give you a unique list. When you try and add a duplicate key you therefre just need to trap the error and increase the count for that key. The count would be the second element in the array for that collections key.
Hope this makes sense. Good luck.
I'm evaluating CouchBase for an application, and trying to figure out something about range queries on views. I know I can do a view get for a single key, multiple keys, or a range. Can I do a get for multiple ranges? i.e. I want to retrieve items with view key 0-10, 50-100, 5238-81902. I might simultaneously need 100 different ranges, so having to make 100 requests to the database seems like a lot of overhead.
As far as I know in couchbase there is no way to implement getting values from multiple ranges with one view. May be there are (or will be implemented in future) some features in Couchbase N1QL, but I didn't work with it.
Answering your question 100 requests will not be a big overhead. Couchbase is quiet fast and it's designed to handle a lot of operations per second. Also, if your view is correctly designed, it will not be "recalculated" on each query.
Also there is another way:
1. Determine minimum and maximum value of your range (it will be 0..81902 according to your example)
2. Query view that will return only document ids and a value that range was based on, without including all docs in result.
3. On client side filter array of results from previous step according to your ranges (0-10, 50-100, 5238-81902)
and then use getMulti with document ids that left in array.
I don't know your data structure, so you can try both ways, test them and choose the best one that will fit your demands.