How to configure NGINX with websocket for React app hosted in Kubernetes - reactjs

Question

The websocket (using #stomp/stompjs library) connection works ok on local development bench. Running the app on Azure AKS platform using an NGINX proxy for serving the React app (with also NGINX ingress controller) also works fine. The challenge, however, is the websocket connection in the latter environment.
Firefox browser returns the following error when trying to connect:
Uncaught (in promise) DOMException: An invalid or illegal string was specified ...
i stomp-handler.ts:31
(Async: promise callback)
s stomp-handler.ts:31
da stomp-handler.ts:31
da stomp-handler.ts:31
value client.ts:404
value client.ts:401
In React, the websocket configuration is:
const server = {
name: process.env.REACT_APP_BASE_URL_SOCKET,
}
...
const client = new Client();
client.configure({
brokerURL: server.name,
...
The environment variable is:
REACT_APP_BASE_URL_SOCKET=/ws
NGINX configuration for React app is specified as:
...
http {
...
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
# listen on port 80
listen 80;
server_name foo.com;
gzip off;
proxy_max_temp_file_size 0;
# save logs here
access_log /var/log/nginx/access.log compression;
root /usr/share/nginx/html;
index index.html index.htm;
proxy_set_header HOST $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
try_files $uri $uri/ /index.html =404;
}
location /ws {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass http://bar-api-service.default.svc.cluster.local:8000;
}
...
}
}
With Docker file as:
FROM nginx:alpine
COPY build/ /usr/share/nginx/html
# Copy our custom nginx config
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
ENTRYPOINT ["nginx", "-g", "daemon off;"]
Meanwhile, the Kubernetes NGINX Ingress controller is configured as:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
cert-manager.io/cluster-issuer: letsencrypt
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/websocket-services: bar-ui-service
nginx.org/websocket-services: bar-ui-service
spec:
tls:
- hosts:
- foo.com
secretName: tls-secret
rules:
- host: foo.com
http:
paths:
- pathType: Prefix
backend:
service:
name: bar-ui-service
port:
number: 80
path: /
With "bar-ui-service" as:
apiVersion: v1
kind: Service
metadata:
name: bar-ui-service
spec:
type: NodePort
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
selector:
app: bar-ui
And "bar-api-service" as:
apiVersion: v1
kind: Service
metadata:
name: bar-api-service
spec:
selector:
app: bar-api
tier: backend
ports:
port: 8000
targetPort: 8000
In the Spring Boot API serving the websocket, Spring Security is used, with OAuth2 resource server. Configuration of HttpSecurity is:
#Override
protected void configure(HttpSecurity http) throws Exception {
http.httpBasic().disable()
.formLogin().disable()
.and()
.cors()
.and()
.authorizeRequests()
.antMatchers("/ws").permitAll()
And websocket broker config as:
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketBrokerConfig implements WebSocketMessageBrokerConfigurer {
#Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.enableSimpleBroker("/queue", "/topic");
registry.setApplicationDestinationPrefixes("/app");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/ws")
.setAllowedOrigins("foo.com");
}
}
How do I get the websocket to work in k8s?

Answer 1

To solve this challenge in Kubernetes, I had to change the WebSocket library. Simply put, the '#stomp/stompjs' library (v6.1.2) cannot upgrade from http to ws. After trying out several libraries (yes, JavaScript WebSocket libraries seem to be a dime a dozen out there!), I settled on react-stomp-hooks (v2.1.0).
For those who might be curious about the fuller solution, read on.
In React (TS), the 'react-stomp-hook' library comes with a convenient Provider component for connecting to the server. This may be added to App.tsx like so:
...
import {
StompSessionProvider,
} from "react-stomp-hooks";
import WebSocketSubscriptions from './shared/WebSocketSubscription';
...
const socketHeaders = {
login: token.userId,
passcode: <can be anything>
}
return (
<React.Fragment>
{token &&
<StompSessionProvider
url={server.name}
connectHeaders={socketHeaders}>
<WebSocketSubscriptions />
</StompSessionProvider>}
...
The url ('server.name') is from an environment variable defined as:
REACT_APP_BASE_URL_SOCKET=/ws
for Kubernetes deployment, and:
REACT_APP_BASE_URL_SOCKET=http://localhost:8000/ws
in the local development environment.
In order to work, the web socket's connection header requires to pass down a user identifier to match with Spring Security's user authentication (shown below). In my example, this is provided by the OAuth 2.0 JWT token. However, the "passcode", although a required parameter, can be anything. Note the Provider enables subscriptions to be defined in a separate component (or components) using a useSubscription hook, for example:
import {
useSubscription,
} from "react-stomp-hooks";
const WebSocketSubscriptions: React.FC = () => {
useSubscription("/app/subscribe", (message) => {
console.log(`..subscribed ${message}`);
});
...
The React-serving NGINX proxy configuration is as follows (you can see the fuller implementation in my question above):
...
location /ws {
proxy_pass http://bar-api-service.default.svc.cluster.local:8000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
}
...
Meanwhile, the Kubernetes NGINX ingress controller need only deal with secure termination of traffic, like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- foo.com
secretName: tls-secret
rules:
- host: foo.com
...
The heavy-lifting now comes in the Spring Boot (v2.7.3) API where the WebSocket server resides. Addition of Spring Security is what makes the solution hairy.
At least the following dependencies are required:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-messaging</artifactId>
<version>5.6.2</version>
</dependency>
First let's look at the websocket's support files:
The AuthChannelInterceptorAdaptor class authenticates the user:
#Component
public class AuthChannelInterceptorAdaptor implements ChannelInterceptor {
private static final String USERNAME_HEADER = "login";
private static final String PASSWORD_HEADER = "passcode";
private final WebSocketAuthenticatorService webSocketAuthenticatorService;
#Autowired
public AuthChannelInterceptorAdaptor(final WebSocketAuthenticatorService webSocketAuthenticatorService) {
this.webSocketAuthenticatorService = webSocketAuthenticatorService;
}
#Override
public Message<?> preSend(final Message<?> message, final MessageChannel channel)
throws AuthenticationException {
final StompHeaderAccessor accessor = MessageHeaderAccessor.getAccessor(message, StompHeaderAccessor.class);
if (StompCommand.CONNECT == accessor.getCommand()) {
final String username = accessor.getFirstNativeHeader(USERNAME_HEADER);
final String password = accessor.getFirstNativeHeader(PASSWORD_HEADER);
final UsernamePasswordAuthenticationToken user =
webSocketAuthenticatorService.getAuthenticatedOrFail(username, password);
accessor.setUser(user);
}
return message;
}
}
As mentioned earlier, for an API server using the OAuth 2.0 Resource library, only the username need be provided; password can be anything. Spring Boot Security simply authenticates the user id (from the websocket's header login) and accepts the connection.
The authentication is carried out in the WebSocketAuthenticatorService class:
/*
courtesy: Anthony Raymond: https://stackoverflow.com/questions/45405332/websocket-authentication-and-authorization-in-spring
*/
#Component
public class WebSocketAuthenticatorService {
public UsernamePasswordAuthenticationToken getAuthenticatedOrFail(final String username,
final String password)
throws AuthenticationException {
if (username == null || username.trim().isEmpty()) {
throw new AuthenticationCredentialsNotFoundException("Username was null or empty.");
}
if (password == null || password.trim().isEmpty()) {
throw new AuthenticationCredentialsNotFoundException("Password was null or empty.");
}
// You can add your own logic for retrieving user from, say, db
//if (fetchUserFromDb(username, password) == null) {
// throw new BadCredentialsException("Bad credentials for user " + username);
// Null credentials, we do not pass the password along
return new UsernamePasswordAuthenticationToken(
username,
null,
Collections.singleton((GrantedAuthority) () -> "USER") // MUST provide at least one role
);
}
}
The WebSocketAuthorizationSecurityConfig class extends AbstractSecurityWebSocketMessageBrokerConfigurer, allowing configuration of inbound messages:
#Configuration
public class WebSocketAuthorizationSecurityConfig extends AbstractSecurityWebSocketMessageBrokerConfigurer {
#Override
protected void configureInbound(final MessageSecurityMetadataSourceRegistry messages) {
// You can customize your authorization mapping here.
messages.anyMessage().authenticated();
}
#Override
protected boolean sameOriginDisabled() {
return false;
}
}
Then we come to the WebSocketBrokerConfig class, which implements WebSocketMessageBrokerConfigurer:
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketBrokerConfig implements WebSocketMessageBrokerConfigurer {
#Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.enableSimpleBroker("/queue", "/topic");
registry.setApplicationDestinationPrefixes("/app");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/ws")
.setAllowedOrigins("http://localhost:3000", "foo.com")
.withSockJS();
}
}
This in-memory broker is good for training and POCs. In production, however, a more robust provider like RabbitMQ or Kafka is called for.
Another implementation of the WebSocketMessageBrokerConfigurer is required for configuring the inbound channel. This is implemented in the class WebSocketSecurityConfig:
#Configuration
#Order(Ordered.HIGHEST_PRECEDENCE + 99)
public class WebSocketSecurityConfig implements WebSocketMessageBrokerConfigurer {
#Autowired
private AuthChannelInterceptorAdaptor authChannelInterceptorAdapter;
#Override
public void registerStompEndpoints(final StompEndpointRegistry registry) {
// Endpoints are already registered on WebSocketConfig, no need to add more.
}
#Override
public void configureClientInboundChannel(final ChannelRegistration registration) {
registration.setInterceptors(authChannelInterceptorAdapter);
}
}
Note the #Order annotation, which configures a high precedence for the channel.
The React app client move from a ws-based websocket connection to an http-brokered one also requires a CORS filter to work. The example filter looks like:
/*
courtesy of https://stackoverflow.com/users/3669624/cнŝdk
*/
#Component
public class CORSFilter implements Filter {
private final List<String> allowedOrigins = Arrays.asList(
"http://localhost:3000", "foo.com");
public void destroy() {}
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {
// Lets make sure that we are working with HTTP (that is, against HttpServletRequest and HttpServletResponse objects)
if (req instanceof HttpServletRequest && res instanceof HttpServletResponse) {
HttpServletRequest request = (HttpServletRequest) req;
HttpServletResponse response = (HttpServletResponse) res;
// Access-Control-Allow-Origin
String origin = request.getHeader("Origin");
response.setHeader("Access-Control-Allow-Origin", allowedOrigins.contains(origin) ? origin : "");
response.setHeader("Vary", "Origin");
// Access-Control-Max-Age
response.setHeader("Access-Control-Max-Age", "3600");
// Access-Control-Allow-Credentials
response.setHeader("Access-Control-Allow-Credentials", "true");
// Access-Control-Allow-Methods
response.setHeader("Access-Control-Allow-Methods", "POST, GET, OPTIONS, DELETE");
// Access-Control-Allow-Headers
response.setHeader("Access-Control-Allow-Headers",
"Origin, X-Requested-With, Content-Type, Accept, " + "X-CSRF-TOKEN");
}
chain.doFilter(req, res);
}
public void init(FilterConfig filterConfig) {
}
}
The WebSocketController class simply provides an endpoint for the websocket's subscriptions and support for discrete '/queues'.
#Controller
public class WebSocketController {
#MessageMapping("/subscribe")
#SendToUser("/queue/notification")
public String replyToAccountFromClient(#Payload String message,
Principal user) {
return String.format("hello: %s", message);
}
#MessageExceptionHandler
#SendTo("/queue/errors")
public String handleException(Throwable exception) {
return exception.getMessage();
}
}
Finally, the HttpSecurity configuration looks like the following:
#Override
protected void configure(HttpSecurity http) throws Exception {
http.httpBasic().disable()
.formLogin().disable()
.addFilterAt(corsFilter, BasicAuthenticationFilter.class)
.csrf().ignoringAntMatchers(API_URL_PREFIX)
.and()
.cors()
.and()
.authorizeRequests()
.antMatchers("/ws/**").permitAll()
...
.oauth2ResourceServer(oauth2ResourceServer ->
oauth2ResourceServer.jwt(jwt ->
jwt.jwtAuthenticationConverter(
getJwtAuthenticationConverter())))
.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
}