I was thinking to make use of Elastic Search and want to know all the possible loopholes in security for Elastic Search and how to take care of them. Also, what effect will this have in performance of Elastic Search?
Elasticsearch by default is not secure, means anybody who knows your ip can access it. But there are lot of ways to secure it.
In configuration you can set the value of network.bind_host to localhost or your intranet ip so that is is accessible only from that. For more details check out the doc.
You can simply restrict the port access(default is 9200) using iptables.
You can use nginx as a proxy so that you can have all the goodness and configurability of nginx. Read about it at playing http tricks with nginx.
Elastic also has a commercial security product called shield.
There are few other security plugins available on the net also. Though elasticsearch by default is not secured it is easy to setup a security around it.
Of all I personally prefers the nginx proxy as it is very easy to setup and gives me an added advantage of logging all request to elasticsearch via nginx access logs.
Lastly, the security additions will have no/negligible performance impact.
ElasticSearch is insecure by default, however I'd really hesitate to say thats any different than any other service. You shouldn't have your database connection public facing, right? You should really consider treating it like any other services that you wouldn't want publicly accessible. Elasticsearch does provide https and basic auth. So it has the capability to be secure as long as you make it so, but the same can be said about many services you deploy.
Related
I'm using AWS EC2 to run a database that supports search capabilities - similar to Elasticsearch. The database is only running in a single AWS region due to budgetary constraints.
The database is also running inside of a private subnet in a VPC. Currently there are no inbound or outbound connections that it can make.
I need to allow access to the database so that only my serverless functions can connect to it via HTTP. Users should not be allowed to access it directly from the client-side. Using Lambda is possible but is far from ideal due to long cold start times. Users will expect search results to appear very quickly, especially when the page first loads. So something else is required.
The plan is to replace Lambda with Cloudflare Workers. With faster start times and closer distance to end users all over the world, connecting to the database this way would probably give me the speed I need while still offering all the benefits of a serverless approach. However, I'm not sure how I can configure my VPC security group to allow connections only from a specific worker.
I know that my workers all have unique domains such as https://unique-example.myworkerdomain.com and they remain the same over time. So is there a way I can securly allow inbound connections from this domain while blocking everything else? Can/should this be done through configuring security groups, internet gateway, IAM role, something else entirely?
Thank you for any help and advice
There are a couple of options.
ECS
You can run an ECS cluster in the same VPC as your database, and run Fargate tasks, which have sub-second start times (maybe 100ms or less?). And you can run ECS tasks on hot cluster instances (but you then pay for them all the time), but perhaps a scale to/from zero approach with ECS would allow you to manage cost without compromising on most of user requests (the first request after a scale-to-zero event would get 100ms+ latency, but subsequent requests would get similar). Lambda actually does something similar to this under the hood, but with much more aggressive scale-down timelines. This doesn't restrict from a specific domain, but may solve your issue.
Self-Managed Proxy
Depending on how your database is accessed, you might be able to have a reverse proxy such as Nginx in a public subnet doing request validation to limit access to the database. This could control access by any request headers, but I'd recommend doing TLS client validation to ensure that only your functions can access the database through the proxy, and it might be possible to validate the domain this way (by limiting the trusted CA to an intermediate CA that only signs for that domain, alternatively, I think Nginx can allow a connection depending on traits of the client cert matches regexes such as domain name).
Route Through Your Corporate Network
Using a VPN, you can originate the function from within your network or somehow filter the request, then the database could still be in a private subnet with connectivity allowed from the corporate network through the VPN.
Use AWS WAF
You make a public ALB pointing at your database, and set up AWS WAF to block all requests that don't contain a specific header (such as an API key). Note: you may have to also set up Cloudfront, I forget off the top of my head whether you can apply WAF directly to an ELB or not. Also note: I don't particularly advise this, as I don't think WAF was designed with sensitive strings in the rules, so you may have to think about who has describerule / describewebacl permissions on WAF, also these rules may end up in logs because AWS doesn't expect the rules to be sensitive. But it might be possible for WAF to filter on something you find viable. I'm pretty sure you can filter on HTTP headers, but unless those headers are secret, anyone can connect by submitting a request with those headers. I don't think WAF can do client domain validation.
Problem. I'm looking for an agile way to shoot a docker container (stored on GCR.IO) to a managed service on GCP:
one docker container gcr.io/project/helloworld with private data (say, Cloud SQL backend) - can't face the real world.
a bunch of IPs I want to expose it to: say [ "1.2.3.4" , "2.3.4.0/24" ].
My ideal platform would be Cloud Run, but also GAE works.
I want to develop in agile way (say deploy with 2-3 lines of code), is it possible run my service secretly and yet super easily? We're not talking about a huge production project, we're talking about playing around and writing a POC you want to share securely over the internet to a few friends making sure the rest of the world gets a 403.
What I've tried so far.
The only think that works easily is a GCE vm with docker-friendly OS (like cos) where I can set up firewall rules. This works, but it's a lame docker app on a disposable VM. Machine runs forever and dies at reboot unless I stabilize it on cron/startup. Looks like I'm doing somebody else's job.
Everything else I've tried so far failed:
Cloud Run. Amazing but can't set up firewall rules on it, or Cloud Director, .. seems to work only with IAP which is painful to set up.
GAE. Works with multiple IPs and can't detach public IPs or firewall it. I managed to get the IP filtering within the app but seems a bit risky. I don't [want to] trust my coding skills :)
Cloud Armor. Only supports a HTTPS Load Balancer which I don't have. Nor I have MIGs to point to. I want simplicity.
Traffic Director and need a HTTP L7 balancer. But I have a docker container, on a single pod. Why do I need a LB?
GKE. Actually this seems to work: [1] but it's not fully managed (I need to create cluster, pods, ..)
Is this a product deficiency or am I looking at the wrong products? What's the simplest way to achieve what I want?
[1] how do I add a firewall rule to a gke service?
Please limit your question to one service. Not everyone is an expert on all Google Cloud services. You will have a better chance of a good answer for each service if they are separate questions.
In summary, if you want to use Google Cloud Security Groups to control IP based access you need to use a service that runs on Compute Engine as security groups are part of the VPC feature set. App Engine Standard and Cloud Run do not run within your project's VPC. This leaves you with App Engine Flex, Compute Engine, and Kubernetes.
I would change strategies and use Google Cloud Run managed by authentication. Access is controlled by Google Cloud IAM via OAuth tokens.
Cloud Run Authentication Overview
I have agreed with the John Hanley’s reply and I have up-voted his answer.
Also, I’ve learned that you are looking how to restrict access to your service through GCP.
By setting a firewall rules, You can limit access to your service by limiting the Source IP range as Allowed source, so that only this address will be allowed as source IP.
Please review another thread in Server Fault [1], stating how to “Restrict access to single IP only”.
https://serverfault.com/questions/901364/restrict-access-to-single-ip-only
You can do quite easily with a Serverless NEG for Cloud Run or GAE
If you're doing this in Terraform you can follow this article
I'm trying to get a proof of concept going for a multi-tenancy containerized ASP.NET MVC application in Service Fabric. The idea is that each customer would get 1+ instances of the application spread across the cluster. One thing I'm having trouble getting mapped out is routing.
Each app would be partitioned similar to this SO answer. The plan so far is to have an external load balancer route each request to the SF Reverse Proxy service.
So for instance:
tenant1.myapp.com would get routed to the reverse proxy at <SF cluster node>:19081/myapp/tenant1 (19081 is the default port for SF Reverse Proxy), tenant2.myapp.com -> <SF Cluster Node>:19081/myapp/tenant2, etc and then the proxy would route it to the correct node:port where an instance of the application is listening.
Since each application has to be mapped to a different port, the plan is for SF to dynamically assign a port on creation of each app. This doesn't seem entirely scaleable since we could theoretically hit a port limit (~65k).
My questions then are, is this a valid/suggested approach? Are there better approaches? Are there things I'm missing/overlooking? I'm new to SF so any help/insight would be appreciated!
I don't think the Ephemeral Port Limit will be an issue for you, is likely that you will consume all server resources (CPU + Memory) even before you consume half of these ports.
To do what you need is possible, but it will require you to create a script or an application that will be responsible to create and manage configuration for the service instances deployed.
I would not use the built-in reverse proxy, it is very limited and for what you want will just add extra configuration with no benefit.
At moment I see traefik as the most suitable solution. Traefik enables you to route specific domains to specific services, and it is exactly what you want.
Because you will use multiple domains, it will require a dynamic configuration that is not provided out of the box, this is why I suggested you to create a separate application to deploy these instances. A very high level steps would be:
You define your service with the traefik default rules as shown here
From your application manager, you deploy a new named service of this service for the new tenant
After the instance is deployed you configure it to listen in a specific domain, setting the rule traefik.frontend.rule=Host:tenant1.myapp.com to the correct tenant name
You might have to add some extra configurations, but this will lead you to the right path.
Regarding the cluster architecture, you could do it in many ways, for starting, I would recommend you keep it simple, one FrontEnd node type containing the traefik services and another BackEnd node type for your services, from there you can decide how to plan the cluster properly, there is already many SO answers on how to define the cluster.
Please see more info on the following links:
https://blog.techfabric.io/using-traefik-reverse-proxy-for-securing-microservices-on-azure-service-fabric/
https://docs.traefik.io/configuration/backends/servicefabric/
Assuming you don't need an instance on every node, you can have up to (nodecount * 65K) services, which would make it scalable again.
Have a look at Azure API management and Traefik, which have some SF integration options. This works a lot nicer than the limited built-in reverse proxy. For example, they offer routing rules.
I used Riak(http://basho.com/riak/) as rest service and Angular on client. When I try to use method "PUT" then first request is OPTION, but Riak doesn't know how to properly respond for this.
I found some clients but all of them are made to run on server, not sure about Node.js client like this http://riak-js.org/
Can I make it working from web client?
Maybe Riak was not meant to work with web clients directly, then I'll try something else.
I don't know about Riak, but the OPTIONS request suggests that you're trying to perform a cross-domain request (Angular running on domain "aaa.com", Riak on domain "bbb.com", although it can also even be just a different subdomain or port number).
My guess is that Riak doesn't support CORS, in which case you need to look for an alternative (a simple server-side proxy might be all that you need, although please consider the security impact of exposing Riak directly to browsers).
I have found js gui client for Riak https://github.com/basho/rekon, but it works directly from riak, that's not what i wanted but maybe i can use same solution also, or make proxy on server
For our e-commerce service running on AppEngine we would like to offer the option for customers to run the stores on their custom domains (eg: www.mystore.com instead of www.enstore.com/mystore).
From a user perspective, I'd like them to enter the domain name they want to use in their preference screen and tell them how to configure their dns.
I know how you normally add domains to an AppEngine instance (through Google Apps) but I'm not sure you can automate that. And even if that's possible they would be all (hundreds) listed on our google apps page.
Anyone know if this is possible/if there is a good way to do it?
I don't think there is a way to add domains "programatically" to an AppEngine instance. Apparently, domains can only be added by using the Google Apps method that you described. This is confirmed in this SO post: How do i get foo.somedomain.com get handled by myapp.appspot.com/foo on appengine
The only options that pop to mind are the following:
HTTP Redirection
Many DNS providers support HTTP Redirection. In this case, your clients would be able to set up mystore.com and www.mystore.com to redirect to www.enstore.com/mystore. There are some obvious disadvantages with this method that might not be acceptable. First of all, with 301 and 302 redirects, the users will still be forwarded to the registered AppEngine URL: www.enstore.com/mystore, and it will show in their browser. In addition, choosing between a 301 and 302 redirect can make SEO tricky, since you'd have to get into how search engines behave with these redirects. For example most search engines will not use the original URL as a source for keywords when you use a 301 redirect.
In addition to 301 and 302 redirects, some DNS providers (like DNS Made Easy) also provide what they call a "masked hidden-iframe redirect". The page will render inside a hidden iframe, so the URL does not change in the user's browsers. However this makes SEO even more tricky, and it will not allow users to bookmark internal pages, or to reference them easily.
As you can see, this option is less than ideal, but it is one option to consider in some situations. Also note that at the moment, HTTP Redirection using 301 redirects is the suggested workaround for the Naked Domain Issue 777 on the AppEngine issue tracker.
Reverse Proxy
Another option could be to set up a small server somewhere else, like a small Amazon EC2 Instance, and set up a simple reverse proxy. You would be able to set this up very easily, just by using Apache and mod_proxy (or various other alternatives). This would allow you to ask your clients to set up a normal A Record pointing to this instance, while the Apache HTTP server would be acting as a proxy to your AppEngine.
The fundamental configuration directive to set up a reverse proxy in mod_proxy is the ProxyPass. You would typically set it up with one line like these for each VirtualHost (for each client domain):
ProxyPass / http://www.enmystore.com/mystore/
The configuration of the remote proxy could be easily handled by your back-end software.
This is a neater solution which gives you plenty of control - but there are obviously some costs for these benefits. First of all, there is the expense to host the reverse proxy. You would also be adding another point of failure, so you have to add this to your high-availability plan. In addition, if you are serving some pages through SSL it can become quite complicated.
Another option is to have each customer sign up for google apps, and then add your appengine app to their app. That way they can manage the url. They will need to use a cname for this, so urls will be limited to something like 'store.customer.com' You will have to support the multitenancy off of the host-header, but that isn't hard to do given that you already have a way to support multitenancy already. You might want to do the setup for the first couple of clients yourself so you can document the easiest way to set it up.
The rietveld code review app does this as you can add it to your google apps domain. See http://code.google.com/p/rietveld/wiki/CodeReviewHelp#Using_Code_Reviews_with_Google_Apps for more detail.
The preferred option is probably to offer your solution through the Google Solutions Marketplace: http://www.google.com/enterprise/enterprise_marketplace/about.html
We did something similar to Daniel Vassallo second proposal.
We created a python app on the Heroku cloud
(there is no limit for connecting custom domains).
This app is using python requests 1.2.0 lib to get the correct page from your app engine application according to the request domain.
all you need to tell your clients is to put your Heroku app url as their CNAME
For naked domains you can always use wwwizer