We have a huge number of models which we plan to train/test/deploy/use. We have several levels of products - starting from Country -> Region -> Warehouse -> Department -> Class A -> Class B -> Product ID etc., Some of the models will be for overall country level, some Country+Region level, and of course, similarly it will go down to the Class B and Product level. Which means if there are total 10,000 products - we will have 10,000 models in the product level.
In this kind of scenario, where we can have 10s of thousands of models, we were wondering how to manage these models. Naming convention as well as grouping them together for easy look up etc. are part of the model management also.
Our initial thought was to use Sagemaker Model Package Group to group these models - so we can easily organize and find them as needed. However, I just learned that one SageMaker Model Package Group is a flat structure - which means one Model Package Group cannot contain another Model Package Group - so this hierarchical organization of models could not be replicated using Model Package Group.
Question is: How do we achieve this kind model organization in production? This is not an unusual situation or anything, I believe there are many retailers who are dealing with model management scenarios like this. How do they do it? If model package is not used, what else can be used to organize them and how?
Any suggestions/idea will be highly appreciated.
Thanks
Using SageMaker Model registry, you can catalog models by creating model package groups that contain *different versions of a model. You can create a model group that tracks all of the models that you train to solve a particular problem.
So the basic idea is that you have a business problem and you create 1 group for it and store all the models along with their versions.
Going by that definition, having a model package group under another model package group is an invalid scenario. Thats not how model group is defined and not how model group is intended to be used.
Also, this problem is not only restricted to models and you can see it everywhere is s/w engineering. Whether is it storing jar files or python packages etc.
And i would suggest to treat this problem as a software engineering problem and rely on namespace convention. Rather than thinking in a hierarchical way: Country -> Region -> Warehouse -> Department -> Class A -> Class B -> Product ID
Can we think of it in country.region.warehouse.department.classA.classB.product_id ?
Similar to how you also might be managing the docker images you can create such namespaces and go as deep as you want. You are still respecting the hierarchy but also not maintaining the hierarchy per say.
Let me know what you think of this approach.
Related
I'm puzzling over how to set up this relationship in Laravel, (I'm converting a legacy app):
I have Repair Shops, which provide different types of repairs, on different brands of vehicles.
For example, Shop A might repair Brakes but not Exhaust Systems for Ford vehicles. Shops are required to say what services they provide, (Exhaust repair), but adding a brand is optional. I have Shop, Service, and Brand tables in the DB. Shop and Service have a belongsToMany relationship using the provides_service pivot table. In the legacy system I have a 3-way pivot table to specify what Services can be done to each Brand in each Shop.
Laravel doesn't seem to do 3-way relationships well, (or does it? If so, point me there!). So, I feel like it would make sense to create a belongsToMany between the provides_service relation and the Brand. So, is there a way to set up a relationship between a Model and another Relationship in Laravel, or do I have to create a ProvidesService model? Creating a ProvidesService model seems wasteful, but I'm not sure what else to do here.
Instead of having separate relation what if the problem of having one big relation that will store all the data related to a system?
A single database may not fulfill all the needs of a complete system. For example : Lets take a ecommerce website. Here you have following important things :
Customer Information
Order information
Inventory Information etc.
Now if you see these three are not related to each other other than that they are all a part of ecommerce system. You don't need to store the user information along with inventory information in a single table as they are not related.
After this part is answered, Now, you can have doubts so as to why we need normalization. The objectives of normalization as mentioned here are:
To free the collection of relations from undesirable insertion, update and deletion dependencies;
To reduce the need for restructuring the collection of relations, as new types of data are introduced, and thus increase the life span of application programs;
To make the relational model more informative to users;
To make the collection of relations neutral to the query statistics, where these statistics are liable to change as time goes by.
I'm planning to develop a web application in CakePHP that shows information in graphics and cards. I chose CakePHP because the information that we need to show is very structured, so the model approach makes easier to manage data; also I have some experience with MVC from ASP.NET and I like how simple is to use the routing.
So, my problem is that the multiple organizations that could use the app would have their own database with a different schema that the one we need. I can't just set their string connection in the app.php file because their database won't match my model.
And the organization datasource couldn't fit my model for a lot of reasons: the tables don't have the same name, the schema is different, the fields of my entity are in separated tables, maybe they have the info in different databases or also in different DBMS!
I want to know if there's a way to make an interface that achieves this
In such a way that cakephp Model/Entity can use data regardless of the source. Do you have any suggestions of how to do that? Does CakePHP have an option to make this possible? Should I use PHP with some kind of markup language like JSON or XML? Maybe MySQL has an utility to transform data from different sources into a view and I can make CakePHP use the view instead of the table?
In case you have an answer be as detailed as you can.
This other options are possible if it's impossible to make the interface:
- Usw another framework that can handle this easier and has the features I mentioned above.
- Make the organization change their database so it matches my model (I don't like this one, and probably they won't do it).
- Transfer the data in the application own database.
Additional information:
The data shown in graphics are from students in university. Any university has its own database with their own structure and applications using the db, that's why isn't that easy to change structure. I just want to make it as easy as possible to any school to configure their own db.
EDIT:
The version is CakePHP 3.2.
An important appointment is that it doesn't need all CRUD operations, only "reading". Hope that makes the solution easier.
I don't think your "question" can be answered properly, it doesn't contain enough information, not enough details. I guess there is something that will stay the same for all organizations but their data and business logic will be different. But I'll try it.
And the organization datasource couldn't fit my model for a lot of reasons: the tables don't have the same name, the schema is different, the fields of my entity are in separated tables, maybe they have the info in different databases or also in different DBMS!
Model is a whole layer, so if you have completely different table schemas your business logic, which is part of that layer, will be different as well. Simply changing the database connection alone won't help you then. The data needs to be shown in the views as well and the views must be different as well then.
So what you could try to do and what your 2nd image shows is, that you implement a layer that contains interfaces and base classes. Then create a Cake plugin for each of the organizations that uses these interfaces and base classes and write some code that will conditionally use the plugin depending on whatever criteria (guess domain or sub-domain) is checked. You will have to define the intermediate interfaces in a way that you can access any organization the same way on the API level.
And one technical thing: You can define the connection of a table object in the model layer. Any entity knows about it's origin but you should not implement business logic inside an entity nor change the connection through an entity.
EDIT: The version is CakePHP 3.2. An important appointment is that it doesn't need all CRUD operations, only "reading". Hope that makes the solution easier.
If that's true either use the CRUD plugin (yes, you can use only the R part of it) or write some code, like a class that describes the organization and will be used to create your table objects and views on the fly.
Overall it's a pretty interesting problem but IMHO to broad for a simple answer or solution that can be given here. I think this would require some discussion and analysis to find the best solution. If you're interested in consulting you can contact me, check my profile.
I found a way without coding any interface. In fact, it's using some features already included in the DBMS and CakePHP.
In the case that the schema doesn't fit the model, you can create views to match de table names and column names from the model. By definition, views work as a table so CakePHP searches for the same table name and columns and the DBMS makes the work.
I made a test with views in MySQL and it worked fine. You can also combine the data from different tables.
MySQL views
SQL Server views.
If the user uses another DBMS you just change the datasource in app.php, and make the views if it's necessary
If the data is distributed in different DBMS, CakePHP let's you set a datasource for each table, you just add it to app.php and call it in the table if it's required.
Finally, in case you just need the "reading" option, create a user with limited access to the views and only with SELECT privileges.
USING:
CakePHP 3.2
SQL SERVER 2016
MySQL5.7
Not sure how to structure this. Model inheritance seems sensible, but it looks like Django will add a one-to-one link between the related models, which I don't need. Here's my situation: I have two models, for a Game and a Turn within a game. What I'd like to do is provide a "demo" version of these on my website for potential users to play around with. I want them to function just like the real models, but to populate different tables (eg say "demo_game" and "demo_turn") so I can clean them periodically and not "pollute" the real game/turn tables.
What's the best way to structure this? I could just copy the models to new versions, but would rather have a more elegant way to keep them in sync in case I modified one, but there is no need for any db relationship between a model and its demo version.
Create abstract base classes for each type, then derive concrete children.
Sorry if this seems a little crazy but ive been messing around with NHibernate for a while and have come accross a scenario that may not be possible to solve with NHibernate...
I have 1 database that contains a load of static data, imagine it like a huge product lookup database, just to point out this is an example scenario, my actual one is a bit more complex but similar principle to this... It is hosted on a completely different box so i cant do "database2.table1.somecolumn" which i noticed as a possible way round the issue if the 2 DBs were in the same box and server.
Anyway i also have another DB which contains data relating to users, so imagine a user has bought a load of stuff from Generic Website A, you have a list of IDs pertaining to what they have bought and an amount of how many they bought as well as some other information, but the actual data relating to the product is stored in the other database...
So if you imagine you want to combine this data into a PreviousPurchasedProduct model which contained all the information from the 1st database and the additional data from the 2nd db you would have to do a query similar to this: (if they were all on one box)
SELECT db1.products., db2.purchases.
FROM db2.purchases
INNER JOIN db1.products ON db2.purchases.product_id = db1.products.id
WHERE db2.purchases.user_id = XXX;
Now first of all is it possible to map this sort of thing up even though they are in seperate DB hosts, im guessing not and if thats the case can you achieve this flexibility via a child class. So having a product class that purely works off db1 and a derived class that takes the purchases info that only works from db2.
Also is it possible to restrict the db1 portion of data from INSERT/UPDATE/DELETE statements, im pretty sure you can in the default mappings but as this would be outside of the per class scope im not sure what flexibility i have...
Thanks for reading my waffle :D
I would recommend you first understand how to solve this problem without NHibernate. Once you have one or more clean solutions that work without NHibernate, come back and update your question to say something like "how do I represent this SQL in NHibernate?".
Querying across databases can quickly bring database vendor specific quirks into play and you never mention which database vendor(s) you are dealing with. If both databases are the same vendor, you may be able to link the databases somehow using database vendor specific techniques (but linking isn't necessarily a good solution, so you'll want to try to uncover alternatives as well).