This is my structure of the firestore database:
Expected result: to get all the jobs, where in the experience array, the lang value is "Swift".
So as per this I should get first 2 documents. 3rd document does not have experience "Swift".
Query jobs = db.collection("Jobs").whereArrayContains("experience.lang","Swift");
jobs.get().addOnSuccessListener(new OnSuccessListener<QuerySnapshot>() {
#Override
public void onSuccess(QuerySnapshot queryDocumentSnapshots) {
//Always the queryDocumentSnapshots size is 0
}
});
Tried most of the answers but none worked out. Is there any way to query data in this structure? The docs only available for normal array. Not available for array of custom object.
Actually it is possible to perform such a query when having a database structure like yours. I have replicated your schema and here are document1, document2, and document3.
Note that you cannot query using partial (incomplete) data. You are using only the lang property to query, which is not correct. You should use an object that contains both properties, lang and years.
Seeing your screenshot, at first glance, the experience array is a list of HashMap objects. But here comes the nicest part, that list can be simply mapped into a list of custom objects. Let's try to map each object from the array to an object of type Experience. The model contains only two properties:
public class Experience {
public String lang, years;
public Experience() {}
public Experience(String lang, String years) {
this.lang = lang;
this.years = years;
}
}
I don't know how you named the class that represents a document, but I named it simply Job. To keep it simple, I have only used two properties:
public class Job {
public String name;
public List<Experience> experience;
//Other prooerties
public Job() {}
}
Now, to perform a search for all documents that contain in the array an object with the lang set to Swift, please follow the next steps. First, create a new object of the Experience class:
Experience firstExperience = new Experience("Swift", "1");
Now you can query like so:
CollectionReference jobsRef = rootRef.collection("Jobs");
jobsRef.whereArrayContains("experience", firstExperience).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (QueryDocumentSnapshot document : task.getResult()) {
Job job = document.toObject(Job.class);
Log.d(TAG, job.name);
}
} else {
Log.d(TAG, task.getException().getMessage());
}
}
});
The result in the logcat will be the name of document1 and document2:
firstJob
secondJob
And this is because only those two documents contain in the array an object where the lang is set to Swift.
You can also achieve the same result when using a Map:
Map<String, Object> firstExperience = new HashMap<>();
firstExperience.put("lang", "Swift");
firstExperience.put("years", "1");
So there is no need to duplicate data in this use-case. I have also written an article on the same topic
How to map an array of objects from Cloud Firestore to a List of objects?
Edit:
In your approach it provides the result only if expreience is "1" and lang is "Swift" right?
That's correct, it only searches for one element. However, if you need to query for more than that:
Experience firstExperience = new Experience("Swift", "1");
Experience secondExperience = new Experience("Swift", "4");
//Up to ten
We use another approach, which is actually very simple. I'm talking about Query's whereArrayContainsAny() method:
Creates and returns a new Query with the additional filter that documents must contain the specified field, the value must be an array, and that the array must contain at least one value from the provided list.
And in code should look like this:
jobsRef.whereArrayContainsAny("experience", Arrays.asList(firstExperience, secondExperience)).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (QueryDocumentSnapshot document : task.getResult()) {
Job job = document.toObject(Job.class);
Log.d(TAG, job.name);
}
} else {
Log.d(TAG, task.getException().getMessage());
}
}
});
The result in the logcat will be:
firstJob
secondJob
thirdJob
And this is because all three documents contain one or the other object.
Why am I talking about duplicating data in a document it's because the documents have limits. So there are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. So storing duplicated data will only increase the change to reach the limit.
If i send null data of "exprience" and "swift" as "lang" will it be queried?
No, will not work.
Edit2:
whereArrayContainsAny() method works with max 10 objects. If you have 30, then you should save each query.get() of 10 objects into a Task object and then pass them one by one to the to the Tasks's whenAllSuccess(Task... tasks).
You can also pass them directly as a list to Tasks's whenAllSuccess(Collection> tasks) method.
With your current document structure, it's not possible to perform the query you want. Firestore does not allow queries for individual fields of objects in list fields.
What you would have to do is create an additional field in your document that is queryable. For example, you could create a list field with only the list of string languages that are part of the document. With this, you could use an array-contains query to find the documents where a language is mentioned at least once.
For the document shown in your screenshot, you would have a list field called "languages" with values ["Swift", "Kotlin"].
Related
I used json serialization to store list on ids in a field
Model:
public class Video
{
public int Id { get; set; }
public string Name { get; set; }
public virtual IList<int> AllRelatedIds { get; set; }
}
Context:
modelBuilder.Entity<Video>(entity =>
{
entity.Property(p => p.AllRelatedIds).HasConversion(
v => JsonConvert.SerializeObject(v, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
v => JsonConvert.DeserializeObject<IList<int>>(v, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore })
);
});
It works fine, Adding, Editing, Deleting items is easy and in SQL Database it stores as json like
[11000,12000,13000]
Everything is fine BUT!! as soon as want to query on this list I get weird responses.
Where:
_context.Set<Video>().Where(t=>t.AllRelatedIds.contains(11000)) returns null however if I ask to return all AllRelatedIds items some records have 11000 value exp.
Count:
_context.Set<Video>().Count(t=>t.AllRelatedIds.contains(11000)) returns could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().
What's the matter with EF Core? I even tested t=>t.AllRelatedIds.ToList().contains(11000) but made no difference
What I should do? I don't want to have more tables, I used this methods hundreds of times but seems never queried on them.
The Json Serialization/Deserialization happens at application level. EF Core serializes the IList<int> object to value [11000,12000,13000] before sending it to database for storing, and deserializes the value [11000,12000,13000] to IList<int> object after retrieving it from the database. Nothing happens inside the database. Your database cannot operate on [11000,12000,13000] as a collection of number. To the database, its a single piece of data.
If you try the following queries -
var videos = _context.Set<Video>().ToList();
var video = _context.Set<Video>().FirstOrDefault(p=> p.Id == 2);
you'll get the expected result, EF Core is doing it's job perfectly.
The problem is, when you query something like -
_context.Set<Video>().Where(t=> t.AllRelatedIds.Contains(11000))
EF Core will fail to translate the t.AllRelatedIds.Contains(11000) part to SQL. EF Core can only serialize/deserialize it because you told it to (and how). But as I said above, your database cannot operate on [11000,12000,13000] as a collection of integer. So EF Core cannot translate the t.AllRelatedIds.Contains(11000) to anything meaningful to the database.
A solution will be to fetch the list of all videos, so that EF Core can deserialize the AllRelatedIds to IList<int>, then you can apply LINQ on it -
var allVideos = _context.Set<Video>().ToList();
var selectedVideos = allVideos.Where(t=> t.AllRelatedIds.Contains(11000)).ToList();
But isn't fetching ALL videos each time unnecessary/overkill or inefficient from performance perspective? Yes, of course. But as the comments implied, your database design/usage approach has some flaws.
I am beginning to use Dapper and love it so far. However as i venture further into complexity, i have ran into a big issue with it. The fact that you can pass an entire custom object as a parameter is great. However, when i add another custom object a a property, it no longer works as it tries to map the object as a SQL parameter. Is there any way to have it ignore custom objects that are properties of the main object being passed thru? Example below
public class CarMaker
{
public string Name { get; set; }
public Car Mycar { get; set; }
}
propery Name maps fine but property MyCar fails because it is a custom object. I will have to restructure my entire project if Dapper can't handle this which...well blows haha
Dapper extensions has a way to create custom maps, which allows you to ignore properties:
public class MyModelMapper : ClassMapper<MyModel>
{
public MyModelMapper()
{
//use a custom schema
Schema("not_dbo_schema");
//have a custom primary key
Map(x => x.ThePrimaryKey).Key(KeyType.Assigned);
//Use a different name property from database column
Map(x=> x.Foo).Column("Bar");
//Ignore this property entirely
Map(x=> x.SecretDataMan).Ignore();
//optional, map all other columns
AutoMap();
}
}
Here is a link
There is a much simpler solution to this problem.
If the property MyCar is not in the database, and it is probably not, then simple remove the {get;set;} and the "property" becomes a field and is automatically ignored by DapperExtensions. If you are actually storing this information in a database and it is a multi-valued property that is not serialized into a JSON or similar format, I think you are probably asking for complexity that you don't want. There is no sql equivalent of the object "Car", and the properties in your model must map to something that sql recognizes.
UPDATE:
If "Car" is part of a table in your database, then you can read it into the CarMaker object using Dapper's QueryMultiple.
I use it in this fashion:
dynamic reader = dbConnection.QueryMultiple("Request_s", param: new { id = id }, commandType: CommandType.StoredProcedure);
if (reader != null)
{
result = reader.Read<Models.Request>()[0] as Models.Request;
result.reviews = reader.Read<Models.Review>() as IEnumerable<Models.Review>;
}
The Request Class has a field as such:
public IEnumerable<Models.Review> reviews;
The stored procedure looks like this:
ALTER PROCEDURE [dbo].[Request_s]
(
#id int = null
)
AS
BEGIN
SELECT *
FROM [biospecimen].requests as bn
where bn.id=coalesce(#id, bn.id)
order by bn.id desc;
if #id is not null
begin
SELECT
*
FROM [biospecimen].reviews as bn
where bn.request_id = #id;
end
END
In the first read, Dapper ignores the field reviews, and in the second read, Dapper loads the information into the field. If a null set is returned, Dapper will load the field with a null set just like it will load the parent class with null contents.
The second select statement then reads the collection needed to complete the object, and Dapper stores the output as shown.
I have been implementing this in my Repository classes in situations where a target parent class has several child classes that are being displayed at the same time.
This prevents multiple trips to the database.
You can also use this approach when the target class is a child class and you need information about the parent class it is related to.
I have recently started evaluating Dapper as a potential replacement for EF, since I was not too pleased with the SQL that was being generated and wanted more control over it. I have a question regarding mapping a complex object in my domain model. Let's say I have an object called Provider, Provider can contain several properties of type IEnumerable that should only be accessed by going through the parent provider object (i.e. aggregate root). I have seen similar posts that have explained using the QueryMultiple and a Map extension method but was wondering how if I wanted to write a method that would bring back the entire object graph eager loaded, if Dapper would be able to do this in one fell swoop or if it needed to be done piece-meal. As an example lets say that my object looked something like the following:
public AggregateRoot
{
public int Id {get;set;}
...//simple properties
public IEnumerable<Foo> Foos
public IEnumerable<Bar> Bars
public IEnumerable<FooBar> FooBars
public SomeOtherEntity Entity
...
}
Is there a straightforward way of populating the entire object graph using Dapper?
I have a similar situation. I made my sql return flat, so that all the sub objects come back. Then I use the Query<> to map the full set. I'm not sure how big your sets are.
So something like this:
var cnn = sqlconnection();
var results = cnn.Query<AggregateRoot,Foo,Bars,FooBar,someOtherEntity,AggregateRoot>("sqlsomething"
(ar,f,b,fb,soe)=>{
ar.Foo = f;
ar.Bars = b;
ar.FooBar = fb;
ar.someotherentity = soe;
return ar;
},.....,spliton:"").FirstOrDefault();
So the last object in the Query tag is the return object. For the SplitOn, you have to think of the return as a flat array that the mapping will run though. You would pick the first return value for each new object so that the new mapping would start there.
example:
select ID,fooid, foo1,foo2,BarName,barsomething,foobarid foobaritem1,foobaritem2 from blah
The spliton would be "ID,fooid,BarName,foobarid". As it ran over the return set, it will map the properties that it can find in each object.
I hope that this helps, and that your return set is not too big to return flat.
I am using Solrs TermsComponent to implement an autocomplete feature. My documents contain tags which I have indexed in a "tags" field. Now I can use TermsComponent to find out which tags are used in all the stored documents. This works pretty well so far.
However there is some additional requirement: Every document has an owner field which contains the ID of the user who owns it. The autocomplete list should only contain tags from documents, that the user who is requesting the autocomplete is actually owning.
I have tried to set the query parameter, however this seems to be ignored by the TermsComponent:
public static List<String> findUniqueTags(String beginningWith, User owner) throws IOException {
SolrParams q = new SolrQuery().setQueryType("/terms")
.setQuery("owner:" + owner.id.toString())
.set(TermsParams.TERMS, true).set(TermsParams.TERMS_FIELD, "tags")
.set(TermsParams.TERMS_LOWER, beginningWith)
.set(TermsParams.TERMS_LOWER_INCLUSIVE, false)
.set(TermsParams.TERMS_PREFIX_STR, beginningWith);
QueryResponse queryResponse;
try {
queryResponse = getSolrServer().query(q);
} catch (SolrServerException e) {
Logger.error(e, "Error when querying server.");
throw new IOException(e);
}
NamedList tags = (NamedList) ((NamedList)queryResponse.getResponse().get("terms")).get("tags");
List<String> result = new ArrayList<String>();
for (Iterator iterator = tags.iterator(); iterator.hasNext();) {
Map.Entry tag = (Map.Entry) iterator.next();
result.add(tag.getKey().toString());
}
return result;
}
So is there a way of limiting the tags returned by TermsComponent, or do I manually have to query all the tags of the user and filter them myself?
According to this and that post on the Solr mailing list, filtering on the terms component is not possible because it operates on raw index data.
Apparently, the Solr developers are working on a real autosuggest component that supports your filtering.
Depending on your requirements you might be able to use the faceting component for autocomplete instead of the terms component. It fully supports filter queries for reducing the set of eligible tags to a subset of the documents in the index.
Following grails domain class:
class MyClass {
Map myMap
}
Now for myMap, grails automatically creates a new table for the elements in the map. However if I add elements which are too long (e.g. 1024 characters), I get a DB error.
Can I somehow tell grails to make the respective column in myMap's table big enough to allow for larger Strings, or do I have to do this manually in the DB?
I already tried
static constraints = {
myMap(maxSize:1024)
}
which doesn't work (as expected because maxSize should refer to the Map's values and not to the Map itself).
If not via constraints, maybe there's a way to do it via
static mapping { ... }
?
An alternative approach I used successfully was to push the map out into a collection of a collaborator domain class.
class DynaProperty {
String name
String value
static belongsTo = MyClass
static constraints = {
value(maxSize:4000) //Or whatever number is appropriate
}
}
And then in MyClass:
class MyClass {
static hasMany = [dynaProperties:DynaProperty]
}
This is almost a map, and it gives you the ability to use dynamic finders to pull up an individual entry.
what are you trying to accomplish? Is there always the same number of things in the map? If there is you should define those properties on your class.
You can see the problem with your current approach -- there is no way to figure out what might be in the map until runtime, so how can grails possibly create a columns for it? Im surprised it even worked to begin with...