I'm trying to figure out the best way to add multiple product images for a given product. The amount of images will vary depending on the product.
I was thinking about using a pivot table. Do you have any other ideas? I'm also open to critiques of my overall design.
My product db draft
Related
I am more familiar with Excel but want all my information in Notion since it’s so powerful.
I’m trying to track weeks of results for multiple individuals on a team. I would like to see a team view but also see the individual performance on a weekly basis.
I want to have the information from each team member rollup to the database that has the sum of all teams for that particular week.
Team page
Teammate 1
Teammate 2
Coming from a RDBMS background and trying to wrap my head around ElasticSearch data storage patterns...
Currently in SQL Server, we have a star schema data mart, RecordData. Rows are organized by user ID, geographic location that pertains to the rest of the searchable record, title and description (which are free text search fields).
I would like to move this over to ElasticSearch, and have read about creating a separate index per user. If I understand this correctly, with this suggestion, I would be creating a RecordData type in each user index, correct? What is a recommended naming convention for user indices that will be simple for Kibana analysis?
One issue I have with this recommendation is, how would you organize multiple web applications on the ES server? You wouldn't want to have all those user indices all over the place?
Is it so bad to have one index per application, and type per SQL Server table?
Since in SQL Server, we have other tables for user configuration, based on user ID's, I take it that I could then create new ES types in user indices for configuration. Is this a recommended pattern? I would rather not have two data base systems for this web application.
Suggestions welcome, thank you.
I went through the same thing, and there are a few things to take into account.
Data Modeling
You say you use a star schema today. Elasticsearch is typically appropriate for denormalized data where the totality of the information resides in each document unlike with a star schema. If you can live with denormalized, that is fine but I assume that since you already have star schema, denormalized data is not an option because you don't want to go and update millions of documents each time the location name change for example(if i understand the use case). At least in my use case that wasn't an option.
What are Elasticsearch options for normalized data?
This leads us to think of how to put star schema like data in a system like Elasticsearch. There are a few options in the documentation, the main ones i focused were
Nested Objects - more details at https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html . In nested objects the entire information is kept in a single document, meaning one location and its related users would be in a single document. That may make it not optimal becasue the document will be huge and again, a change in the location name will require to update the entire document. So this is better but still not optimal.
Parent - Child Relationship - more details at https://www.elastic.co/guide/en/elasticsearch/guide/current/parent-child.html . In this case the location and the User records would be kepts in separate indices similarly to a relational database. This seems to be the right modeling for what we need. The only major issue with this option is the fact that Kibana 4 does not provide ways to manipulate/aggregate documents based on parent/child relationship as of this writing. So if you main driver for using Elasticsearch is Kibana(this was mine), that kind of eliminates the option. If you want to benefit from the elasticsearch speed as an engine this seems to be the desired option for your use case.
In my opinion once you got right the data modeling all of your questions will be easier to answer.
Regarding the organization of the servers themselves, the way we organize that is by having a separate cluster of 3 elasticsearch nodes behind a Load Balancer(all of that is hosted on a cloud) and then have all your Web Applications connect to that cluster using the Elasticsearch API.
Hope that helps.
I want cluster an image dataset into several groups using K-means, N-cut or other algorithm, but I don't know how to process those images in the dataset first. These groups should have their own special features. Anyone has any suggestions?
My suggestion is that you go ahead and try a number of features.
Which feature works best for your is very much dependent on your use case.
If you are hoping to group photos by mood, group faces by users or group CAD drawings by the type of gear on it require completely different feature extraction approaches. So you will have to kiss a few frogs to find your prince.
You use the mosaic dataset.
A mosaic dataset allows you to store, manage, view, and query small to vast collections of raster and image data. It is a data model within the geodatabase used to manage a collection of raster datasets (images) stored as a catalog and viewed as a mosaicked image. Mosaic datasets have advanced raster querying capabilities and processing functions and can also be used as a source for serving image services. find more here
And also I refere you to this paper-article:
title: CREATING AN IMAGE DATASET TO MEET YOUR CLASSIFICATION NEEDS:A PROOF-OF-CONCEPT STUDY
By:
James D. Hurd, Research Associate
Daniel L. Civco, Director and Professor
Imagine a company that lets people use their web software to build web sites. Each customer has one or more sites that they've built and each of these sites will contain perhaps a hundred "cells" that hold the HTML for various pieces of the site. We know for certain that each cell will only be used in one site. So we have:
Now we need to describe the cells in each site.
One way would be to have a huge (and I mean huge) cell table that contains all the cells created by a million customers. Then each cell record would point back to the siteId where it was used.
But it seems that there should be a way to take advantage of the knowledge that each cell is only one of a hundred or so in its site, so instead of searching a table of 100 million cells we could search a table of 100 cells, or just ask for that table when we want to build that site.
Thanks for your input.
One way would be to have a huge (and I mean huge) cell table that
contains all the cells created by a million customers. Then in each
row of a cell record we'd point back to the siteId where it was used.
This is probably the route I would go. Most modern databases can handle tables with hundreds of millions of rows when you use the proper performance tuning techniques.
I'm not positive but it sounds like you are considering placing cells for each site in a site specific table in an effort to keep the number of rows in each table down. If this is true it seems like you are trying to over optimize before you have any performance problems.
I would focus on creating a well designed normalized database at this point and if you run into performance problems, I would come back here with the specific performance problem.
I am new user to SOLR. I am working on an E-commerce web application which have SQL database. I want to implement SOLR for my "category page" in application where we will show products of that category with specific information like available stock , price and few more details. Also we want to restrict product display on basis of stock availability, if there is no stock then we wont display those products.
I am trying to implement SOLR with Delta import queries to make my category pages faster. And my concern is about performance of page while getting data from SOLR and accuracy of real time data like Stock and Price.
actually my database queries to get product data are little bit complicated and have several joins, so i have to make several related entities in SOLR database. So due to this data upload to SOLR is slow. that makes difficult to upload data (even with delta import query) frequently, so my application lacking real time data like stock of products.
basically i want to know best practice approach to implement SOLR. I am confused with
1. Should i export my all data to SOLR and then get all details from SOLR?
(I am worried about performance and real time data)
2. should i get only index data (say id of products) from SOLR and then get othere details from my SQL database? (not sure about this approach perfomance).
So please help me and suggest me that how can i implement SOLR for my app in best way.
all help is appretiated!!
One good case practice I encountered while developing e-commerce solution and SOLR as the search provider is to retrieve from SOLR only the IDs and get the data from SQL server.
I created a single schema that was updating every time some new products were added in the database.(In my case products were added by a user in an admin console, in your case you can use the #Mauricio Scheffer comment to have latest updates)
One field in the schema was the ID - representing the ID of the product in the database.
After querying SOLR I was receiving N documents suiting the query and with the ID field I was getting all the information from my database and display it to the user.
So the good part is that the user will always get the data from your database ( real time data), and will have his search results displayed very fast because of SOLR.
You can add to the schema different fields that you can use to filter your results such as
category
date
price
size
etc...
And also differnet fields that you need to query upon:
headline
description
name
etc...
And also add the product ID.
So after making the query to SOLR you have a list of product IDs, now the only thing you need to do is to implement a function that will get the products from database based on the ID and display it on the search results page.
This approach performance is pretty good because selecting from database elements based on primary key is the fastest way to retrieve data from sql.
I worked on a website with 1.000.000 products and searching time was always under 0.2 seconds
and page loading time in the browser was under 0.6 seconds on single user queries. ( the server where SOLR and SQL was running was 8Gb ram and quad core 1.8 gb as I remember)
Hope this type of implementation is useful for you.
You can set up a database trigger or data access layer event to send data to Solr whenever there's a change, and configure autoCommit to control freshness.
See also:
http://wiki.apache.org/solr/NearRealtimeSearch
http://java.dzone.com/articles/tuning-solr-near-real-time