Design: using a backend server to circumvent great firewall of china - angularjs

I have a front-end angular app using firebase to store user data.
I currently do not have a backend set up, such as a node.js server.
I would like to use the Google Docs API to upload files from my app.
Since the Great Firewall of China does not (or makes unstable) the use of Google services, is it possible to place those services on the backend server and still use them reliably?
Perhaps after they have uploaded the document to firebase, a backend script retrieves it, uploads it to google docs, and then removes the record from firebase? Just trying to see if Google or similar services are even feasible for this use case.
I suppose the crux of my question is whether or not the calling of the Google API would be taking place on the user's computer, in which case would it become unstable?
** Updates for clarity:
I am deciding whether my firebase-backed app needs a more traditional backend like a node server to do things like: upload images and documents, send mail via Mandrill, etc... It would be helpful to me if I knew whether, after putting in the time to create a server, some of the services I am after (aka APIs) are any more resilient to the GFW than they would be if they ran on the client side. So if any one has had success in such a task, I would like to know.
** Technical update:
So, for example, if I run the Google Maps API on the client side, if the user is in China and is not running a VPN, accessing the API calls will either lag or time out or (rarely) success in returning the scripts. If I was somehow able to able to process the map query "off-site" aka on the server, could I then return with a static image of the map to a Chinese user without fail?


If I was somehow able to able to process the map query "off-site" aka
on the server, could I then return with a static image of the map to a
Chinese user without fail?
Yes, of course. What you are going to miss this way is all the front-end interactive functionality Google Maps offers. But if that's ok in your use case, sure.
I have never tried it with the GCF, but what I would do is this:
Google Maps <-> Your Reverse proxy <-> User
So, instead of the user visitng the real google maps site, it will be visiting your maps.mydomain.com site, that will be sitting in between, proxying everything.
Nginx is an excellent choice for a reverse proxy. If you need more control, there are good node.js reverse proxying packages that you an use to rewrite the content extensively before serving it (perhaps to obfuscate it in case the GCF blacklists content based on pattern matching, or to change the script names/links again to avoid pattern matching).


You are misunderstanding about the great firewall of China. I consulted for a couple of Chinese companies after the dot com crash so I can say this from personal experience, not hearsay.
It is mostly high-end Cisco hardware behind gateways behind their government telecom infrastructure. Nowadays they knock off what hardware they can, every chance they can, and spend money on specialized hardware to monitor cell phones systems.
There was a brief mention of the street-level surveillance hardware on 20/20 before the crash if you are interested in looking it up.
Not to discourage you, but I say set up whatever open servers you want with whatever frontends or backends you want, but the reality is the traffic is not going to be there.
That is why they call it an oppressive regime, they do not get to decide for themselves, remember?

Related

How to protect my own api from hacking and make it private, so no body can access it exept from the android app?

I have written my own api and I want to upload it to the server but I want to secure it so noone can access it except from my app, I have tried slim-basic-auth library but it didn't work, I don not know why...
any help with that please ?
$app->add(new Tuupola\Middleware\HttpBasicAuthentication([
"secure"=>false,
"users" => [
"userher##" => "passhere##" ]
]));

I think you will never be able to hide the api url being called by the app (an attacker with rooted android device can intercept traffic easily ), If your api response user specific data, you can add auth header in request and same verify this header at server. You can also use Cross-Origin Resource Sharing-CORS and SSL for extra layer of security.

YOUR QUESTIONS
In a glance you want to have a private API, lock your Android App to it, and solve your code issue in the API server. Let's address each one in the order I mentioned them.
PRIVATE APIs
How to protect my own api from hacking and make it private
Well I have a cruel truth to reveal to you, no such thing as a private API exists, unless you don't expose it to the Internet, aka only have it accessible inside a private network, via programs that themselves are not expose also to the internet. By other words your API can only be private if its is air gaped from the internet.
So no matter if an API doesn't have public accessible documentation or if is is protected by any kind of secret or authentication mechanisms, once is accessible from the Internet is not private any-more, and you even have tooling to help with discover them.
You can read more about it in this article section, and I extract to here some bits:
Now just because the documentation for your API is not public or doesn’t even exist, it is still discoverable by anyone having access to the applications that query your API.
Interested parties just need to set up a proxy between your application and the API to watch for all requests being made and their responses in order to build a profile of your API and understand how it works.
A proxy tool:
MiTM Proxy
An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers.
So the lesson here is that from the moment you release a mobile app that uses your API, you can consider it to belong now to the public domain, because anyone can reverse engineer it and discover how your "private" API works, and use that to build automated attacks against it.
HOW TO LOCK AN API TO AN ANDROID APP
I have written my own api and I want to upload it to the server but I want to secure it so noone can access it except from my app
Well you bought yourself a very hard task to accomplish, that some may say that its impossible to do, but once you dig deep enough in the subject, you will be able to understand that you still have some paths to explore.
First you will need to understand the difference between WHO and WHAT is accessing your API server, followed by learning some of the most common techniques used to secure an API server, and finally you will learn that the Mobile App Attestation may be what your are looking for.
The Difference Between WHO and WHAT is Accessing the API Server
I wrote a series of articles around API and Mobile security, and in the article Why Does Your Mobile App Need An Api Key? you can read in detail the difference between WHO and WAHT is accessing your API server, but I will extract here the main take aways from it:
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
So think about the WHO as the user that your API server will be able to Authenticate and Authorize to access the data, and think about the WHAT as the software making that request in behalf of the user.
API Security Defenses
The Basic Defenses
Now that you understand the difference between WHO vs WHAT is accessing your API server you may want to go an read my article about the basic techniques to secure an API:
In this article we will explore the most common techniques used to protect an API, including how important it is to use HTTPS to protect the communication channel between mobile app and API, how API keys are used to identify the mobile app on each API request, how user agents, captchas and IP addresses are used for bot mitigation, and finally how user authentication is important for the mobile security and api security. We will discuss each of these techniques and discuss how they impact the business risk profile, i.e. how easy they are get around.
More Advanced Defenses
You can start by read this series of articles on Mobile API Security Techniques to understand how API keys, HMAC, OAUTH and certificate pinning can be used to enhance the security and at same time how they can be abused/defeated.
Afterwards and depending on your budget and resources you may employ an array of different approaches and techniques to defend your API server, and I will start to enumerate some of the most usual ones.
You can start with reCaptcha V3, followed by Web Application Firewall(WAF) and finally if you can afford it a User Behavior Analytics(UBA) solution.
Google reCAPTCHA V3:
reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease.
...helps you detect abusive traffic on your website without any user friction. It returns a score based on the interactions with your website and provides you more flexibility to take appropriate actions.
WAF - Web Application Firewall:
A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to and from a web application. A WAF is differentiated from a regular firewall in that a WAF is able to filter the content of specific web applications while regular firewalls serve as a safety gate between servers. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.
UBA - User Behavior Analytics:
User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.
All this solutions work based on a negative identification model, by other words they try their best to differentiate the bad from the good by identifying what is bad, not what is good, thus they are prone to false positives, despite of the advanced technology used by some of them, like machine learning and artificial intelligence.
So you may find yourself more often than not in having to relax how you block the access to the API server in order to not affect the good users. This also means that this solutions require constant monitoring to validate that the false positives are not blocking your legit users and that at same time they are properly keeping at bay the unauthorized ones.
Regarding APIs serving mobile apps a positive identification model can be used by implementing a Mobile App Attestation solution that attests the integrity of your mobile app and device its running on before any request is made to the API server.
Mobile App attestation
Finally if you have the resources you can go even further to defend your API server and Mobile App, by building your own Mobile APP Attestation, and you can read in this article section about the overall concept of it, from where I extracted this:
The role of a Mobile App Attestation service is to authenticate what is sending the requests, thus only responding to requests coming from genuine mobile app instances and rejecting all other requests from unauthorized sources.
In order to know what is sending the requests to the API server, a Mobile App Attestation service, at run-time, will identify with high confidence that your mobile app is present, has not been tampered/repackaged, is not running in a rooted device, has not been hooked into by an instrumentation framework(Frida, xPosed, Cydia, etc.), and is not the object of a Man in the Middle Attack (MitM). This is achieved by running an SDK in the background that will communicate with a service running in the cloud to attest the integrity of the mobile app and device it is running on.
On a successful attestation of the mobile app integrity, a short time lived JWT token is issued and signed with a secret that only the API server and the Mobile App Attestation service in the cloud know. In the case that attestation fails the JWT token is signed with an incorrect secret. Since the secret used by the Mobile App Attestation service is not known by the mobile app, it is not possible to reverse engineer it at run-time even when the app has been tampered with, is running in a rooted device or communicating over a connection that is the target of a MitM attack.
The mobile app must send the JWT token in the header of every API request. This allows the API server to only serve requests when it can verify that the JWT token was signed with the shared secret and that it has not expired. All other requests will be refused. In other words a valid JWT token tells the API server that what is making the request is the genuine mobile app uploaded to the Google or Apple store, while an invalid or missing JWT token means that what is making the request is not authorized to do so, because it may be a bot, a repackaged app or an attacker making a MitM attack.
A great benefit of using a Mobile App Attestation service is its proactive and positive authentication model, which does not create false positives, and thus does not block legitimate users while it keeps the bad guys at bay.
So a Mobile App Attestation will allow your API server to identify, with a very high degree of confidence, that the request is coming from WHAT you expect, the original and unmodified APK you have uploaded to the Google Play store.
THE API SERVER CODE ISSUE
I have tried slim-basic-auth library but it didn't work, I don not know why
I am not familiar at all with the Tuupola project, but from a look into the README.md for the Slim API Skeleton, specially the section for how to get a token and then how to use it. The related code that generates the token can be found at routes/token.php, and to use the token to protect an API route you can find an example at routes/todos.php. This is all configured in the config file config/middleware.php. But I have to say that I am not impressed, security wise, with their posture, and this is because they encourage the exposure of server sensitive data via API endpoints, as we can see at routes/token.php, thus I strongly advise you to immediately delete all this endpoints if they are present in your project.
SUMMARY
In my opinion the best solution is defense in depth, by applying as many layers as you can, so that you increase the time, effort and skill-set necessary to by pass all your security layers, thus keeping at bay the script kids and occasionally hackers from abusing your API server and Mobile App.
So you should employ has much techniques as possible in both sides of the equation, mobile app and API, like the ones you have learned when reading the articles I have linked: HTTPS, API keys, User Agents, Captchas, Rate Limiting, OAuth, HMAC, Certificate Pinning, Code Obfuscation, JNI/NDK to hide secretes, WAF, UBA, etc.
In the end, the solution to use in order to protect your API server and Mobile App must be chosen in accordance with the value of what you are trying to protect and the legal requirements for that type of data, like the GDPR regulations in Europe.
GOING THE EXTRA MILE
I would strongly recommend you, to also take a look into the OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.

The thing you need is to add a Variable from your client app passing to your server application. Like APP_KEY or CLIENT_ID, which allows your app connecting to the server. You can add encryption so that your server application can only decrypt it and identify the request coming from your client app.
If your app is a web application and hosted in another server, you can implement IP whitelisting in your server.
But if your app is Mobile, you need to pass like a secret_key from mobile to your server.

Best/Correct way to create a client-server constant listener

I am creating an app that involves sending and receiving settings... The desktop application is constantly sending information to a hosted MySQL database, and the Android app will query this same information. It is something similar to the whatsapp web (but in this case, I'll be using a desktop app instead of webpages).
Until this part, everything is working as I need... but, this same Android app will be used to send settings to the desktop app, and the desktop will read and change its settings according to what was just sent.
If I need to constantly query the hosted MySQL database and check if there is any kind of changes sent from the Android, I believe that I'll have a performance drop... each time a query loop is finished, I would have to query, check for any modifications and so on.
Is there a better or correct way to do this kind integration between two apps? I've read something about WebSockets, but I don't have much technical information about this, neither examples that I can use in this case.
Thank you very much for your knowledge sharing.

Here are some useful sites on WebSocket:
http://websocket.org
http://blog.kaazing.com/ [some useful blog posts]
http://www.html5rocks.com/en/tutorials/websockets/basics/
https://goo.gl/5OaJff [mozilla site]
You may want to consider the Observer/Observables pattern. The MYSQL is the Observable and your desktop app and Android app are Observers (and you can add other Observers in the future). Its a common pattern with lots of examples out there. But you'll need a centralized WebSocket server and an Observer/Observable coordination subsystem. You can setup a pub/sub message broker that uses WebSocket with a nice JMS, MQTT, etc, API to make your life easier. ActiveMQ, IBM MQ Lite, Kaazing JMS Edition... lots of options.
full disclosure. I work for Kaazing.

traffic splitting in google app engine based on customer subdomain

When customers signup to my service (myservice.com), they get assigned a subdomain. So customer A would get a url to customerA.myservice.com, customer B would get a url like customerB.myservice.com etc.
I am working on a new version of the service, and would like to redirect some of the new customers to the new version to test the new version.
I know I can create different versions of the service, and split the traffic by ip and cookie between them. It is also possible to split traffic based on routing, but it seems, that in the end GAE still relies on either ip or cookie information to split the traffic: https://developers.google.com/appengine/docs/adminconsole/trafficsplitting#Routing. So if my customer accesses the my service on desktop, ipad and smartphone, he might sometimes get one version of my service on his desktop (at home) and another on his smartphone (on the road).
So my question is if it is possible to always direct customerA.myservice.com to one version of my service and customerB.myservice.com to another version, regardless of device, cookie or ip?
thanks
Thomas

It's not possible with Traffic Splitting feature of App Engine, but you can implement this logic yourself. When users hit your domain, you can choose which version of the app to serve them. There are three drawbacks:
It won't look pretty - the URLs pointing to individual versions may be confusing.
Users might bookmark the version URL, and later they will get an error when they hit it, if the version is no longer available.
You have to make sure that all URLs in your app are relative, if they point to your domain. Otherwise you might link across different versions.
Also, some versions may not be compatible due to the changes in the data model, i.e. you cannot run them in parallel.
Similar issues also arise with the App Engine's splitting - or any other traffic splitting. This is why I recommend to have a trusted group of your most enthusiastic users. Send them a new version as a "preview" and ask for their feedback. They will appreciate you reaching out to them, and they will not be confused if they hit a different version on a different device.

Track application usage - security

Similar questions have been asked before, but this a one is a little different. I created a REST API to send an XML document with `POST. I send data from my (Windows) application to the servers, which includes: open time, operating system, version, etc.
I have one problem though. How can I make sure people can't use the REST API? How do I know that the information sent to the server is from an application and not from someone who knows the URL? How do analytic software companies solve this problem?
Thank you.
Update
I would like users to use my application without having to log in. I am pretty sure that companies that create apps that do not force you to log in are able to see whatever you are doing.

Well there are several way to secure your service.
You can always setup authentication & authorization for the service - this way the service will be available only to registered/known users.
Here are links few links for more details:
Best Practices for securing a REST API / web service
http://www.stormpath.com/blog/secure-your-rest-api-right-way
Also there are less sophisticated ways such as setting firewall rules to allow connections only from certain places -- I don't think it is a recommended approach.

Web servers and web hosting for mobile client/server application

I would like to build a mobile application with the following requirements:
The mobile client applications should request and recieve data from a database on a server.
In the future I will probably want to build a web application for the same database.
For communication between the clients and the server I would like to use Google Protocol Buffers.
So I have the following questions:
How does one set up a server to take request and respond with anything other than html. I think that using RPC sounds nice, but I have no clue how to set it up on a server.
I need to find a good web hosting service which will allow me to set up a database and a server that can serve both Google Protocol Buffers and regular web pages with data.
Before I get to making the web app, is there any more lightweight solution that might be better just for communicating with the clients (maybe even a home made tiny server), and how hard would it be to do it with a full scale web server from the start?
Please point me in the right direction so I know what to read up on.
I'm not necessarily looking for specific names of web hosting services but rather an idea what kind of services are available that might meet my needs. I've worked a little bit with django, Spring and Java EE so if there's any solution involving those that would be great, however I'm not afraid of learning something new.
Thanks in advance
Simon

if you still mean http, that is pretty trivial - you simply set an appropriate content-type, and write your data to the response stream. The exact how depends on your web framework and tools, but this is no different to (say) serving generated images on the fly. HTTP requests function fine for messaging scenarios - as simple as making an http request (typically POST) with a protobuf (etc) body, and processing the response in the same way.
can't comment
a web app can be lightweight; certainly more-so than having to configure a non-http service/daemon. The "lightweight" option would be raw sockets, but that is harder to deploy, and you'll have to be more picky choosing a host. Unless you absolutely need this level of terseness (i.e. dropping the http headers and writing your own transport to get close to the wire), just stick with http - it'll be a lot easier to get going and maintain
For info, I have a blog post on doing this with ASP.NET MVC; this isn't intended to mean "do this" (heck, use whatever tools help you) - simply, it is meant to show the kind of thing necessary.

Resources