I posted something much like this on openhab and home assistant forum too, I will decide what to do based on what I hear...
I am trying to produce an open source Energy Recovery Ventilator, and software is not my forte.
I frankly find the sheer variety and quantity of buzzwords and subsystems in the home automation sphere difficult to navigate. I am unclear on why exactly things have to be so complicated... anyway.
I am using a raspberry pi pico running micropython. Do you think it would be practical to make it appear as basically a fan with several different modes to a matter hub? Maybe report back some info so the user can see some status updates etc. ?
What I want is basically to allow it to be controlled by a hub, which may be running on a phone or someone’s PC, so the hub’s user interface etc. Can be used to make the device turn off an on, up and down on a schedule, it can be connected to other devices like a co2 detector, smart switch, etc.
I need, sooner or later, possibly with the help of module(s) running on the pico to cache data (like schedule data) get the time or whatever, a dictionary which I will use for the rest of the system to interface to. The main loop consults the dictionary to determine behaviour at any given moment. The hub checks what time of day it is, etc. And sends that info along.
Is this sort of thing doable?
I tried to look into making the thing Alexa compatible and ye gads it would take me months to get that stuff working. They make everything so complicated
Found some stuff for esp32 devices like esphome, but it is not practical to use as a module in a larger system. MQTT looks like it could play an important role, but doesn't quite get me there and for some reason Alexa, Google home etc still cant really talk to mqtt devices very well, esp. including device setup etc. Basically, envision a little hardware device that just serves up some fields and takes back some fields, then appears as a device to Google home's app etc. I need that, but a software module that runs on a pico. Is it practical to roll this or is it going to be an ungainly undertaking?
Related
I'll start with the scenario I am most interested in:
We have multiple devices (2 - 10) which all need to know about
a growing set of data (thousands to hundreds of thousands of small chunks,
say 100 - 1000 bytes each).
Data can be generated on any device and we
want every device to be able to get all the data (edit: ..eventually. devices are not connected and/or online all the time, but they synchronize now and then) No data needs
to be deleted or modified.
There are of course a few naive approaches to handle this, but I think
they all have some major drawbacks. Naively sending everything I
have to everyone else will lead to poor performance with lots of old data
being sent again and again. Sending an inventory first and then letting
other devices request what they are missing won't do much good for small
data. So maybe having each device remember when and who they talked to
could be a worthwhile tradeoff? As long as the number of partners
is relatively small saving the date of our last sync does not use that much
space, but it should be easy to just send what has been added since then.
But that's all just conjecture.
This could be a very broad
topic and I am also interested in the problem as a whole: (Decentralized) version control probably does something similar
to what I want, as does a piece of
software syncing photos from a users smart phone, tablet and camera to an online
storage, and so on.
Somehow they're all different though, and there are many factors like data size, bandwith, consistency requirements, processing power or how many devices have aggregated new data between syncs, to keep in mind, so what is the theory about this?
Where do I have to look to find
papers and such about what works and what doesn't, or is each case just so much
different from all the others that there are no good all round solutions?
Clarification: I'm not looking for ready made software solutions/products. It's more like the question what search algorithm to use to find paths in a graph. Computer science books will probably tell you it depends on the features of the graph (directed? weighted? hypergraph? euclidian?) or whether you will eventually need every possible path or just a few. There are different algorithms for whatever you need. I also considered posting this question on https://cs.stackexchange.com/.
In your situation, I would investigate a messaging service that implements the AMQP standard such as RabbitMQ or OpenAMQ, each time a new chunk is emitted, it should be sent to the AMQP broker which will broadcast it to all devices queues. Then the message may be pushed to the consumers or pulled from the queue.
You can also consider Kafka for data streaming from several producers to several consumers. Other possibility is ZeroMQ. It depends on your specific needs
Have you considered using Amazon Simple notification service to solve this problem?
You can create a topic for each group of device you want to keep in sync. Whenever there is an update in dataset, the device can publish to the topic which in turn will be pushed to all devices using SNS.
My problem is this...
I have two sites, one acting as an "Admin" site, the other as general "User" site. I need to broadcast live audio from the "Admin" site to all clients of the "User" site. I need to do this with <1 sec of latency.
Some restrictions include:
No install on "User" machines (the idea being the whole thing sits on the web)
If there needs to be a 3rd party plugin then Silverlight is preferred*
Any help much appreciated here
*I have tried IceCast with a flash client, IIS Smooth Streaming, Internet radio, all of which give us a latency of >5 secs.
Have you tried Flash with a server like Red5? You're generally going to get subsecond latency (though not much less than that), as it's designed for realtime communications. There's a learning curve with Flex and ActionScript, but if you're at all familiar with XAML, you can pick it up from the sample apps that come with Red5 pretty quickly.
Failing that, if there aren't too many clients, you can use one of the two real-time peer-to-peer solutions out there, namely Flash over RTMFP or WebRTC over JSEP/ICE/RTP. If you can ensure that all the clients are using Chrome, then WebRTC is probably your best bet. If you can ensure that they're not using Chrome, then Flash is a good choice. The current Flash Pepper client on Chrome is buggy up the wazoo when it comes to audio processing, and no sign of a fix in sight. (It doesn't support echo cancellation, and the volume of the audio goes up and down horribly.) So if you're using Flash, steer clear of recording and broadcasting your audio on Chrome. And I wouldn't recommend either approach if you have more than half a dozen clients - the number of audio streams is gonna overwhelm your "Admin" browser pretty quickly, I think. Better to push that out to something like a Red5 server.
Silverlight is a bad choice for more reasons than I can count. I'm saying this as a guy who spent several years trying to implement a realtime communication solution on Silverlight. Don't do it.
Have you ever encountered something so easy to develop but stopped a while to think of server requirements for your project ? It is my case.
I want to compete with a gaming site, they have multiplayer Flash games like poker, rummy, backgammon, and other card games, 8 games in total. For each game they have rooms and tables.
I'll use Silverlight with Sockets. I already managed to develop the policy server, the Socket Server app using WinForms, the Client Socket app in Silverlight. I own a VPS for tests, so there is no problem in developing what I want, the problem is How to calculate server requirements, RAM, bandwidth, internet speed based on the following requirements:
Server should support 24.000 users / day or 1000 users / hour
Each game room should have it's own tables where users can play
Users should not lose scores and game speed should be fast in general
I just wonder how to handle the following situation: if 1000 users are connected through Socket connection to a room full of tables and one user leave a table, all 1000 users must be updated and UI should reflect the changes. Let's say that I'll update the clients by sending a small Message of 100 bytes to each user, this will eat 100 bytes * 1000 users = 100 kb, and this just for 1 UI change, for 1 Game and for 1 Room, not counting all my other games and rooms. Also 1000 iterations that sends bytes to clients should be very time consuming.I am a developer, but not experienced in those situations. Please advice. Numbers will be great.
Until you've built -- and optimized -- your actual applications, you cannot predict much about the hardware required for some level of performance.
You have to finish the apps first. Then you can measure their performance under load. Then you can decide how much to spend on what levels of performance.
The best answer I can offer you is to run stress tests and see how much load a single server can support. While running those tests, monitor memory, IO, CPU and disk activity (if relevant) to understand which resource is running out first.
We deploy our applications on Amazon's EC2 cloud infrastructure. That lets us easily (within minutes) add or remove capacity as needed. Perhaps it's worth considering for your situation.
Always follow these two rules
“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson
First of all you should think more about how and when to send what information to which clients. Not every client needs to be informed about every table change.
That there are only so much informations that a client needs, and you need to decide when/how it will be transmitted. Also you should pack the informations into meaningfull packets. Whats happening at a table is only interesting for that table.
Also you need to profile your application to make sure you know what ressources it consumes. Cardgames should not eat up so much ressources. But the important point is to FIRST build it, and when you HAVE a bottleneck, then try to fix it.
It's very difficult to guess at these things at this point.
From a pragmatic standpoint, you may eventually want to look into a) a cloud-hosting type service for better bandwidth price-scaling for you, or b) a very experienced full-service hosting company that can help you calculate your needs based on prior experience.
Disclaimer: I work for Rackspace Hosting which provides both of the above.
Note: This is not for unit testing or integration testing. This is for when the application is running.
I am working on a system which communicates to multiple back end systems, which can be grouped into three types
Relational database
SOAP or WCF service
File system (network share)
Due to the environment this will run in, there are no guarantees that any of those will be available at run time. In fact some of them seem pretty brittle and go down multiple times a day :(
The thinking is to have a small bit of test code which runs before the actual code. If there is a problem then persist the request and poll until the target system until it is available. Tests could possibly be rerun within the code to check it is still available at logical points. The ultimate goal is to have a very stable system, regardless of the stability (or lack thereof) of the systems it communicates to.
My questions around this design are:
Are there major issues with it? (small things like the fact it may fail between the test completing and the code running are understandable)
Are there better ways to implement this sort of design?
Would using traditional exception handling and/or transactions be better?
Updates
The system needs to talk to the back end systems in a coordinated way.
The system is very async in nature so using things like queuing technologies is fine.
The system must run even if one or more backend systems are down as others may be up and processing of some information is possible.
You will be needing that traditional exception handling no matter what, since as you point out there's always the chance that things'll fail between your last check and the actual request. So I really think any solution you find should try to interact smoothly with this.
You are not stating if these flaky resources need to interact in some kind of coordinated manner, which would indicate that you should probably be using a transaction manager of some sort to do this. I do not believe you want to get into the footwork of transaction management in application code for most needs.
Sometimes I have also seen people use AOP to encapsulate retry logic to back-end systems that fail (for instance due to time-out issues). Used sparingly this may be a decent solution.
In some cases you can also use message queuing technology to alleviate unstable back-ends. You could for instance commit to a message queue as part of a transaction, and only pop off the queue when successful. But this design is normally only possible when you're able to live with an asynchronous process.
And as always, real stability can only be achieved by attacking the root cause of the problem. I had a 25-year old bug fixed in a mainframe TCP/IP stack fixed because we were overrunning it, so it is possible.
The Microsoft Smartclient framework provides a ConnectionMonitor class. Should be easy to use or duplicate.
Our approach to this kind of issue was to run a really basic 'sanity tester' prior to bringing up our main application. This was thick client so we could run the test every time the app started. This sanity test would go out and check things like database availability, and external network (extranet) access, and it could have been extended to do webservices as well.
If there was a failure, the user was informed, and crucially an email was also sent to the support/dev team. These emails soon became unweildy as so many were being created, but we then setup filters, so we knew when somethings really bad was happening. Overall the approach worked pretty well, our biggest win was being able to tell users that the system was down, before they had entered data, and got part way through a long winded process. They absolutely loved it.
At a technica level the sanity was written in C#, it used exception handling in a conventional way not to find the problems it was looking for. The sanity program became a mini app in its own right, and it was standalone from the main app. If I were doing it again I'd using a logging framework to capture issues, which is more flexible then our hard coded approach.
We are using standard items such as Hard Disk and CPU ID to lock our software licenses to physical hardware. How can we reduce the risk of customers installing onto a virtual machine and then cloning the virtual machine, bypassing our licensing?
One approach is to have a licensing server. When you enter a license code into the client (on a VM), it contacts the server and sends it its license code and other information. It contacts it repeatedly (you define the interval -- maybe once every few hours) asking 'Am I still valid"? Along with this request, it sends a unique ID. The server replies 'Yes, you are valid', and sends a new unique ID back to the client. The client sends this unique ID back with its next request to the server. The server verifies this is the same ID it sent to the client for that license, the previous request.
If the VM is duplicated, the next time it asks the server 'Am I valid?', the unique ID will be incorrect either for it, or for the other VM. Both will not continue to work.
You will need to determine what to do if the server goes down, or the network goes down, such that the client cannot communicate with the server. Do you immediately disable your software? Bad idea! Don't make your customers angry. You'll want to give them a grace period. How long should this be? A few days? Weeks?
Let's say you give them a 1-month grace period. In theory, they could clone the parent VM just after entering the license key, then restore the other VMs to this clone just before their grace period runs out, disabling network access to them. This would be a hassle for your customers though, just to have pirated additional copies of your software. You have to determine what kind of grace period won't hassle your legitimate customers, while hopefully giving you the protection you seek.
Additional protection could be achieved by verifying that the VM's clock is set correctly. This would prevent the above approach to pirating.
Another consideration is that a savvy user could write their own licensing server to communicate with the VM instances, and tell them all 'you're good' -- so encrypting the communication could help deter this. How far you want to go here really depends on how much you think pirating really might be an issue with your customers. In the end you won't be able to stop true pirates who have time on their hands, but you can keep honest users honest.
License. Tell your users, they may not run unlicensed copies.
We are actually failing to buy a license for a software at the moment, because the vendor is scared of virtual machines: The infrastructure for our department is being moved to a centralized virtualized sollution and we have to fight the vendor to be allowed to buy a license for his software!
Don't be afraid of paying users.
People too cheep to buy licenses are going to look for another sollution and will be too much hassle anyway.
(good luck telling your boss that, though...)
There is no good reason to lock to a physical machine. Last I checked computers can break down, and then the user is probably going to be inconvenienced not only by a dead computer, but by having to call you to get the software locked to a new machine. If you must do draconian license management use a (local) management server and have running copies verify that they have a license every few minutes. Just realize that whatever you do if someone really wants to use your software without paying you they will find a way.
You need something outside the computer "hardware" to authenticate against. Most companies choose hardware keys (dongles) in for software with a high cost where users will put up with it.
Other companies use online methods - if more than one user with CPUID and other hardware is concurrently using a given license, then disallow another instantiation, or close the existing instantiation.
You have to choose protection according to your needs and the consumer's willingness to jump through your anti-piracy hoops.
-Adam
There's not a lot you can do AFAIK, except require periodic online activation.
We have problems with people Norton-ghosting physical machines. Apparently HDD serial numbers are ghosted too.
If your software runs under a VM, then it will run under any number of cloned VMs. Therefore, the only option seems to prevent it running under a VM at all. Here's an article about virtual machine detection: Detect if your program is running inside a Virtual Machine and one about thwarting it.
By the way, cloning a VM is usually enough of a hassle to deter casual users from bypassing your licensing and those hell bent on cracking will probably find a way to bypass it anyway.
"Don't bother" is the short version. It's non trivial enough for your clients to do it that if they are doing that, then either they won't pay for what they use no matter what (they will not use it unless they can get it for free) or you are just flat charging to much (as in you are gouging.)
The "real" customer will generally pay for the stuff. From what I've seen, places like businesses will generally consider it not worth the effort.
I know some virtual machine software (at least VMware) have features that allow software to detect virtualization. But there is no foolproof way, it's possible to patch such features away anyway. Mysteriously changing performance (due to CPU spikes in the host) could also be used, reliability is questionable. There is a plethora of "signs of being virtualized", but they tend to be not 100% reliable.
It is a problem, and any savvy user will be able to defeat pretty much anything you do about it. Unsavvy users might get caught by behaviors like VmWare's player that changes MAC and other IDs of the virtual machine when you move it, presumably in a nod to this kind of issue.
The best solution is likely to use a license server instead, since that server will count the number of active licenses. Node locking is easier to defeat, and using a server tends also to push responsibility onto an IT department that is more sensitive to not breaking license agreements compared to individual users who just want to get their job done as quickly as possible.
But in the end, I agree that it all falls back to proper license language and having customers you trust somewhat. If you think that people are making a fool of you in this way, you should not be selling your software to them in the first place...
If your software was required to under on a VM what about this concept:
on the host machine you create a compiled program that run eg. every half hour, which reads the Hard Disk and CPU ID, and then stores that together with the current timestamp in a file together with a salted hash of all that information.
you then require that the folder with the file is shared with the VM.
in your compiled software within the VM you can then read this file and check that the timestamp is recent and the hash is valid.
Or better yet, have the host program somehow communicate with the software in the VM directly.
Couldn't this be an okay solution? Not as secure as using a hardware key (like Yubikey) but you would have to be quite tech savvy to break it...?