Hi I am a developer who is studying VOLTTRON PLATFORM.
I have problem about connecting volttron and local cluster(or AWS).
In document, I couldn't found about part of connection volttron with local cluster(or AWS).
I think there three case about this problem solving.
use vip(volttron interconnect protocol)<-- but this solution should install volttron
use http protocol <-- but this is not good solution for big data processing
use socket programming
solution 1 make customize agent for big data framework(Spark ,etc) interface
solution 2~3 make customize agent for connect to cloud
Would you recommend the way to solve this problem? or give some other solution ?
Thank you!
Are you wanting to integrate with the message bus, or just connect to the historian to access data flowing out of the volttron platform?
Either are possible, and your use cases you layout are correct. I would avoid #3, as it's unnecessarily complex and has no benefit that I'm aware of.
I think we would need more context to make a specific recommendation for your use case.
If you only need access to read data out of your volttron deployment, integrating at the database level is probably ideal.
If you need to have input back into the volttron deployment, ie, operating actuators, you would need deeper integration at the VIP level, and I would recommend just writing an agent to accomplish what you want and running it within the volttron environment. You can easily deploy volttron to an EC2 host and access the other AWS resources from that agent.
Related
I would like to switch to a different VPS host. How would I transfer a DigitalOcean droplet to another host?
It's a pain. I use DigitalOcean, AWS, Google Cloud, and Vultr. It's hypothetically possible, mad the best explanation is here; however, as they point out, while it may be difficult for servers that have been active for a very long time, a fresh start is at least worthy of consideration, given the incompatibility of the file formats big cloud services use for snapshots.
Also, keep in mind that even if you can get the snapshot to boot on AWS or Azure or wherever you're going, it's likely the virtual network configuration will be totally different. You're going to be stuck doing network configuration and possibly new reverse proxies and the like. Probably only accessible through the slow browser shell your new provider offers. Do not recommend, from experience.
So the scenario is like this...
I have a number of different users in an organization. Each has his own session of an AngularJS app running in their browser. They share an internet connection over a local LAN.
I need them to continue working together (data, notifications, ... etc) even when they lose internet i.e. server side communication.
What is the best architecture for solving this?
Having clients communicate directly, without a server, requires peer-to-peer connections.
If your users are updating data that should be reflected in the database, then you will have to cache that data locally on the client until the server is available again. But if you want to first send that data to other peers, then you need to think carefully about which client will then update the database when the server comes back up (should it be the original client that made the edit - who may not be online anymore - or should it be the first client that establishes server connection). Lots to consider in your architecture.
To cope with this scenario you need angular service-worker library, which you can read about here.
If you just want the clients/users to communicate without persisting data in the database (eg. simple chat messages) then you don't have to worry about the above complexity.
Refer to this example which shows how to use simple-peer library with Angular2.
An assisting answer (doesn't fit in a comment) was provided here: https://github.com/amark/gun/issues/506
Here is it:
Since GUN can connect to multiple peers, you can have the browser connect to both outside/external servers AND peers running on your local area network. All you have to do is npm install gun and then npm start it on a few machines within your LAN and then hardcode/refresh/update their local IPs in the browser app (perhaps could even use GUN to do that, by storing/syncing a table of local IPs as the update/change)
Ideally we would all use WebRTC and have our browsers connect to each other directly. This is possible however has a big problem, WebRTC depends upon a relay/signal server every time the browser is refreshed. This is kinda stupid and is the browser/WebRTC's fault, not GUN (or other P2P systems). So either way, you'd have to also do (1) either way.
If you are on the same computer, in the same browser, in the same browser session, it is possible to relay changes (although I didn't bother to code for this, as it is kinda useless behavior) - it wouldn't work with other machines in your LAN.
Summary: As long as you are running some local peers within your network, and can access them locally, then you can do "offline" (where offline here is referencing external/outside network) sync with GUN.
GUN is also offline-first in that, even if 2 machines are truly disconnected, if they make local edits while they are offline, they will sync properly when the machines eventually come back online/reconnect.
I hope this helps.
Context:
We have a web application written in Java EE on Glassfish + MySQL with JDBC for storage + XMPP connection for GCM CCS upstream messaging + Quartz scheduler library. Those are the main components of the app.
We're wrapping up our app development stage and we're trying to figure out the best option for deploying it, considering future scalability, both in number of users and their geographical location (ex, multiple VMS on multiple continents, if needed). Currently we're using a single VM instance from DigitalOcean for both the application server and the MySQL server, which we can then scale up with some effort (not much, but probably more than GAE).
Questions:
1) From what I understood, Cloud SQL is replicated in multiple datacenters across the world, thus storage-wise, the geographical scalability is assured. So this is a bonus. However, we would need to update the DB interaction code in the app to make use of Cloud SQL structure, thus being locked-in to their platform. Has this been a problem for you ? (I haven't looked at their DB interaction code yet, so I might be off on this one)
2) Do GAE instances work pretty much like a normal VM would ? Are there any differences regarding this aspect ? I've understood the auto-scaling capability, but how are they geographically scalable ? Do you have to select a datacenter location for each individual instance ? Or how can you achieve multiple worldwide instance locations as Cloud SQL does right out of the box ?
3) How would the XMPP connection fare with multiple instances ? Considering each instance is a separate VM, that means each instance will have a unique XMPP connection to the GCM CCS server. That would cause multiple problems, ex if more than 10 instances are being opened, then the 10 simultaneous XMPP connections limit cap will be hit.
4) How would the Quartz scheduler fare with the instances ? Right now we're saving the scheduled jobs in the MySQL database and scheduling them at each server start. If there are multiple instances, that means the jobs will be scheduled on each instance; so if there are multiple instances, the jobs will be scheduled multiple times. We wouldn't want that.
So, if indeed problems 3 & 4 are like I described, what would be the solution to those 2 problems ? Having a single instance for the XMPP connection as well as a single instance for the Quartz scheduler ?
1) Although CloudSQL is a managed replicated RDBMS, you still need to chose a region when creating an instance. That said, you cannot expect the latency to be great seamlessly globally. You would still need to design a proper architecture to achieve that.
2) GAE isn't a VM in any sense. GAP is a PaaS and, therefore, a runtime environment. You should expect several differences: you are restricted to Java, PHP, GO and Python, the mechanisms GAE provide do you out-of-the-box and compatible third-parties libraries. You cannot install a middleware there, for example. On the other hand, you don't have to deal with anything from the infrastructure standpoint, having transparent high-availability and scalability.
3) XMPP is not my strong suit, but GAE offers a XMPP service, you should take a look at it. More info: https://developers.google.com/appengine/docs/java/xmpp/?hl=pt-br
4) GAE offers a cron service that works pretty well. You wouldn't need to use Quartz.
My last advice is: if you want to migrate an existing application, your best choice would probably be GCE + CloudSQL, not GAE.
Good luck!
Cheers!
I have an app that is running on Google App Engine and uses the Task Queue Api to do some of the heavier lifting in the background. Some of those tasks need to connect to Cloud SQL to do their work. At scale I get too many tasks attempting to connect to Cloud SQL at once. What I need is some sort of data service layer of a shared client so that the tasks themselves aren't making individual connections to Cloud SQL. If anyone has any ideas I'd love to hear them.
Yes, you can actually do this, but it will take a little bit of planning, coding, and configuration in your side.
One idea is to use a Pull Queue (instead of Push Queues). With a pull queue you can schedule your tasks and execute them in a separate module of your application. The hardware of that module can be configured separately from your main module, thus you can avoid too many instances serving requests which in turns will allow you to better use connection pooling.
Of course, depending on the traffic you are getting you might want to decide on how many concurrent backend instances you want running (and connecting to your DB) to avoid/minimize contention.
Easier said than done, but here are two resources that will help you out:
App Engine Modules - https://developers.google.com/appengine/docs/java/modules/
Pull Queues - https://developers.google.com/appengine/docs/python/taskqueue/overview-pull
I have a client-server application which use a firebird server 2.5 over internet.
I have met the problem of given a secure access to FB databases and as a first approch a tried to solve this problem by integrating a tunnel solution in the application (STunnel software more exactly). BUT, this approch suffer from many aspects :
- this add more resource consumption (CPU, memory, threads) at both client/server side,
- sotware deployment become a serious problem because STunnel software is writen as a WinNT Service, not a Dll or a Component (WinNT Service need administrator privileges for install)
and my client application need to run without installation !
SO, i decided to take the bull by the horn (or the bird by the feathers as we talk about Firebird). I have downloaded the Firebird 2.5 source code and injected secure tunnelization code directly in his low level communication layer (the INET socket layer).
NOW, encryption/decryption is done directly by the firebird engine for each TCP/IP packet.
What do you think about this approach vs external tunnelization ?
I would recommend to wrap data exchange in SSL/TLS stream, from both sides. This is proven standard.
While custom implementations, with static keys, can be insecure.
For instance, CTR mode with constant IV can reveal a lot of information, since it only encrypts incremented vector and XORes it with data, so XORing two encrypted packets will show the xored version of unencrypted packets.
In general, my view of security critical code is this, "you want as many eyes on the code in question as possible and you do not want to be maintaining it yourself." The reason is that we all make mistakes and in a collaborative environment these are more likely to be caught. Additionally these are likely to be better tested.
In my view there are a few acceptable solutions here. All approaches do add some overhead but this overhead could, if you want, be handled on a separate server if that becomes necessary. Possibilities include:
stunnel
IPSec (one of my favorites). Note that with IPSec you can create tunnels, and these can then be forwarded on to other hosts, so you can move your VPN management onto a computer other than your db host. You can also do IPSec directly to the host.
PPTP
Cross-platform vpn software like tinc and the like.
Note here in security there is no free lunch and you need to review your requirements very carefully and make sure you thoroughly understand the solutions you are working with.
The stunnel suggestion is a good one, but, if that's not suitable, you can run a true trusted VPN of sorts, in a VM. (Try saying that a few times.) It's a bit strange, but it would work something like this:
Set up a VM on the firebird machine and give that VM two interfaces,
one which goes out to your external LAN (best if you can actually
bind a LAN card to it) and one that is a host-only LAN to firebird.
Load an openvpn server into that VM and use both client and server
certificates
Run your openvpn client on your clients
Strange, but it ensures the following:
Your clients don't get to connect to the server unless BOTH the
client and server agree on the certificates
Your firebird service only accepts connections over this trusted VPN
link.
Technically, local entities could still connect to the firebird
server outside of the VPN if you wanted it -- for example, a
developer console on the same local LAN.
The fastest way to get things done would not be to improve firebird, but improve your connection.
Get two firewall devices which can do SSL certificate authentication and put it in front of your DB server and your firebird device.
Let the firewall devices do the encryption/decryption, and have your DB server do its job without the hassle of meddling with every packet.