Motivation

When I joined the company, their existing system primarily utilized Amazon Cognito. However, several limitations come with this approach:

1. Inadequate Documentation: The available resources for understanding and implementing Cognito were not up to par, making it challenging to leverage its full potential.

2. Limited Customization: We faced constraints in customizing the frontend experience and the JWT (JSON Web Token) features, restricting our ability to tailor the system to our specific needs.

3. Dependency on Additional Libraries: The necessity for extra libraries from AWS for both frontend and backend integration resulted in a bloated architecture.

4. Tight Coupling with AWS: Our reliance on AWS-specific libraries meant that our system was heavily dependent on AWS services, reducing our flexibility.

5. Outdated Libraries: We encountered issues with libraries being outdated, which posed risks and limitations for our system.

6. Suboptimal Regional Management: The management of resources across different regions was not as efficient as required, leading to potential scalability and performance issues.

Since I overhauled the backend using Kubernetes, I thought it would make sense to handle this on a gateway level and use an SSO solution that better integrates with the frontend. I wanted to learn and explore seamless solutions moving forward and potentially abstract away auth operations from both frontend and backend.

Backend

The way our frontend interacts with the backend utilizes the following technologies: REST, GraphQL, and SSE (Server-Sent Events), which are all served over HTTPS as endpoints.

Even though I replaced WebSocket with SSE for real-time updates — since SSE doesn't require a second connection and our use case doesn’t necessitate real-time publishing from users — WebSocket could also utilize the same authentication mechanism. Websocket handshakes use HTTP, which can be used to pass the same authentication information.

We didn't have issues with JWTs (JSON Web Tokens) as they’re stateless and inherently scalable. We didn't see the need for a different authentication and authorization mechanism. It would also be the case that a different approach, like service side cookies or sessions, leads to more issues:

Issues with Server-Side Sessions

• Storage Requirements: Server-side sessions need to be stored, which could result in database overhead or necessitate the implementation of caching solutions.

• Service Dependency for Retrieval: These sessions must be fetched by a service, requiring extra parsing. This approach often doesn’t scale efficiently in microservice architectures.

• Implementation Complexity: Setting up server-side sessions involves significant implementation overhead, adding complexity to the system.

• Synchronization Challenges: If caching is employed in distributed systems, additional synchronization is required, which can complicate the architecture and increase the risk of data inconsistencies.

Gateway & Reverse Proxy

Since a reverse proxy manages all incoming requests on our clusters, transitioning to a gateway solution proved more effective. This approach handles the requests and then seamlessly directs them to the appropriate services and pods.

I tried to set up Kong Gateway first for this task, but their documentation and versions changed a lot, and their implementation wasn't as straightforward. I decided to stick to Istio for this task, which has a service mesh and excellent community support. Other existing solutions might also serve the task.

Istio

While Istio provides mechanisms for authorizations, it doesn't natively integrate with OAuth 2. In other words, it provides AuthorizationPolicy for access control based on request payload (when RequestAuthentication is configured):

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "iot-policy"
  namespace: dev-iot
spec:
  action: ALLOW
  selector:
    matchLabels:
      app: iot
  rules:
    - when:
        - key: request.auth.claims[permissions]
          values: ["read:sensor-data"]
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: "auth0"
  namespace: istio-system
spec:
  jwtRules:
    - issuer: "https://oauth2-issuer"
      jwksUri: "https://oauth2-jwks.json"
      forwardOriginalToken : true
      fromHeaders:
        - name: x-auth-request-access-token
        - name: some-custom-header
      outputPayloadToHeader: istio-jwt-payload

Istio doesn’t handle:

• Obtaining and renewing tokens
• Handling redirections after authentication
• Dealing with various OAuth 2 providers

Istio also serves as a reverse proxy with Gateway and VirtualService configuration:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: iot-gateway
  namespace: dev-iot-rest-api
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - dev.iot.domain
    - port:
        number: 443
        name: https-443
        protocol: HTTPS
      hosts:
        - dev.iot.domain
      tls:
        mode: SIMPLE # enables HTTPS on this port
        credentialName: iot-certs-dev
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: iot-virtual-service
  namespace: dev-iot-rest-api
spec:
  hosts:
    - "*"
  gateways:
    - iot-gateway
  http:
    - name: iot-http-route
      corsPolicy:
        allowOrigins:
          - exact: "*"
        allowMethods:
          - POST
          - GET
          - DELETE
          - PATCH
          - PUT
        allowCredentials: true
        allowHeaders:
          - content-type
      route:
        - destination:
            host: iot-svc
            port:
              number: 9080

OAuth2 Proxy

An existing and well-tested solution exists for this task: OAuth2 Proxy. I could simply redirect requests that require authentication to OAuth2 Proxy from Istio using their CUSTOM external authorization.

Here's an example demonstrating this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: iot-auth-policy-oauth-proxy
  namespace: dev-iot-rest-api
spec:
  selector:
    matchLabels:
      app: iot
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    # You can also replace this with sample-ext-authz-http to test the other external authorizer definition.
    name: oauth2-proxy
  rules:
    - to:
      - operation:
          paths: [ "/sensor" ]

One additional benefit of using OAuth2 Proxy is that it provides client and server-side sessions. Yes, I know it's contrary to the concept of JWT at first glance. The client-side session is still completely stateless.

We don’t store credentials in local/session storage as they’re less secure and may lead to XSS (Cross-Site Scripting) access.

The proxy signs and encrypts the cookies on the server side to prevent modification from the client side and provides higher security. It decodes the cookie once a request arrives and retrieves the JWT stored. Credentials are only stored in frontend.

OAuth2 Proxy abstracts the following tasks:

• Cookie encoding and decoding
• Cookie setting
• Obtaining and renewing tokens
• OAuth2 Redirection
• OAuth2 Provider Support

Invalid JWT/Cookies:

Valid JWT/Cookies:

OAuth2 Proxy deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: prod-oauth2-proxy
spec:
  replicas: 5
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
        - args:
            - --provider=oidc
            - --cookie-secure=true
            - --cookie-samesite=lax
            - --cookie-refresh=1h
            - --cookie-expire=4h
            - --cookie-name=_oauth2_proxy_istio_ingressgateway
            - --set-authorization-header=true
            - --email-domain=*
            - --http-address=0.0.0.0:4180
            - --upstream=static://200
            - --skip-provider-button=true
            - --whitelist-domain=.domain
            - --oidc-issuer-url=https://oauth2-issuer
            - --oidc-jwks-url=https://oauth2-jwks.json
            - --insecure-oidc-allow-unverified-email=true
            - --skip-jwt-bearer-tokens=true
            - --login-url=https://oauth2-issuer/authorize?audience=https://istio.domain
            - --redeem-url=https://oauth2-issuer/oauth/token
            - --skip-oidc-discovery=true
            - --oidc-extra-audience=https://istio.domain,https://oauth2-issuer/userinfo
            - --pass-authorization-header=true
            - --pass-access-token=true
            - --set-xauthrequest=true
            - --cookie-domain=.domain
            - --force-json-errors=true
          env:
            - name: OAUTH2_PROXY_CLIENT_ID
              value: ""
            - name: OAUTH2_PROXY_CLIENT_SECRET
              value: ""
            - name: OAUTH2_PROXY_COOKIE_SECRET
              value: "" # could be generated with python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'
          image: quay.io/oauth2-proxy/oauth2-proxy:v7.3.0
          imagePullPolicy: Always
          name: oauth2-proxy
          ports:
            - containerPort: 4180
              protocol: TCP

Auth0 & OAuth2 Proxy

Even though I could use other providers or implement our own, our goal was to manage our users and their permissions and roles.

Auth0 (by Okta) stands as a well-tested solution for such needs. I decided to migrate our existing database to Auth0 while redesigning the permission and roles for various endpoints we have. However, the solution above also works with any OAuth 2 providers.

The only task remaining is to integrate Auth0 into our system. I already mentioned the configuration part of an OAuth 2 provider with Istio and OAuth 2 Proxy.

Here's a diagram of how they would work together:

Authorization

We have handled user authentication, OAuth2 cookies setting and redirections, JWT encoding, and decoding. A request made by the user (if valid) now passes through OAuth 2 Proxy and gets redirected back to Istio Gateway again.

Istio Gateway first verifies the request against all remaining pre-configured AuthorizationPolicy based on their priorities to see if the request should be handed to their destinations or rejected:

Authorization requests are handled on Istio using AuthorizationPolicy:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: system-0-view-system-status
  namespace: dev-iot-rest-api
spec:
  action: ALLOW
  selector:
    matchLabels:
      app: iot
  rules:
    - to:
      - operation:
          paths:
            - "/action/system-0/info"
      when:
        - key: request.auth.claims[permissions]
          values:
            - "view:system-status"
        - key: request.auth.claims[https://domain/app_metadata][systems]
          values:
            - "system-0"

Considering easier integration with potentially other gateway systems and easier customization, I decided to perform role-based access control with two stages: generic initial filtering using Istio AuthorizationPolicy and fine-grained control on the backend service itself.

Backend services are unaware of the entire authentication procedure; they only need to understand JWTs passed as headers if they need to be used for fine-grained user action authorizations.

Example snippet on how to require a set of permissions using middleware and Echo:

func endpoints() {
    e := echo.New()
    e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
        AllowOrigins:     strings.Split(config.GlobalConfig.EchoCORS, ","),
        AllowCredentials: true,
    }), middleware.GzipWithConfig(middleware.DefaultGzipConfig))

    e.Validator = &CustomValidator{validator: validator.New()}
    // Custom validator used to validate fields using Go struct tags

    g.Use(decodeJWT, decodeClaimsTokensUser, requireUser)
    g.DELETE("/:id", dbredis.WithTransaction(deleteTicket), requirePermissions(permission.DeleteTicket))
}

func requirePermissions(permissions ...string) echo.MiddlewareFunc {
    return func(next echo.HandlerFunc) echo.HandlerFunc {
        return func(c echo.Context) error {
            user := c.Get("user").(*restdata.RESTUser)
            if !user.HasPermissions(permissions...) {
                return echo.NewHTTPError(403, "user does not have permission")
            }
            return next(c)
        }
    }
}

In the example above, decodeJWT and decodeClaimsTokensUser are middleware functions where I decode JWT and map it to a User memory object (struct) for later access. The response and status code are routed back to the user from backend service.

This approach ensures minimum code in our backend services.

[Optional] decodeJWT :

Even though JWTs can be decoded without being verified, and our JWTs are verified by both Istio Gateway and OAuth 2 Proxy when it reaches the backend services, it's still recommended to verify again using JWK, which is provided by the OAuth2 Proxy: https://oauth2-provider/jwks.json.

This step can be done using any JWT crypto libraries in your language of choice.

I can’t think of a way to bypass the gateway mechanism; it should be fairly safe without verifying again (unless the user has access to K8s internal service directly, which is a much more critical issue on its own).

Static Frontend

In a static site, such as a site generated by React and Svelte without server-side rendering, I send cookies along with requests we make in the frontend code (e.g., REST requests).

For instance, you may do this in React using axios.defaults.withCredentials = true for axios REST requests and new EventSource(url, {withCredentials: true}) for SSE events.

In this flow, the frontend never touches JWT or the authentication flow.

Sign In & Sign Out

Invalid JWT (401) → Sign In

If any backend request returns status codes that may indicate JWT expiration or no JWT present like 401, redirect the current page to SSO URL provided by OAuth2 providers. As mentioned, this redirects the user to OAuth2 Proxy URL, to OAuth2 Provider, and then back to the frontend when callback URLs have been passed correctly.

Sample code:

const errorInterceptor = (error: any) => {
	if (error.response.status === 401) {
		window.location.href = `https://${process.env.OAUTH2_HOSTNAME}/oauth2/start?rd=` +
					encodeURIComponent(window.location.href);
	}
	return Promise.reject(error);
}
// OAUTH2_HOSTNAME is the domain behind the Oauth2-Proxy as Oauth2-Proxy redirects to OAuth2 Provider then back to that current location.href.

If the user successfully signs in, a cookie will be present without any frontend logic (set by OAuth2 Proxy when the user is redirected). We don’t need to handle any sign-in logic.

Valid JWT (403) → No Permission

If JWT is valid but the user's not authorized, 403 will be returned by either Istio or our backend services. We properly handle 403 responses and display information for the user on frontend.

Sign out

Sign-out logic is similar to Sign-in, where the user is first redirected to OAuth2 Proxy sign-out URL and then Auth 0 sign-out URL. Both redirections clear corresponding cookies. User is redirected back to frontend URL using encoded callback parameters; this usually triggers another sign-in flow if it is implemented (See: Invalid JWT (401) -> Sign In).

Sample code:

export const signOut = () => {
	window.location.href = `https://${OAUTH2_HOSTNAME}/oauth2/sign_out?rd=` + encodeURIComponent(
		`https://${AUTH0_HOST}/v2/logout?returnTo=https://${window.location.host}`)
}

Retrieve & Modify user information

Sometimes, we may require user information for frontend (e.g., when we need to display certain interfaces based on user roles, or we simply need the user email address!).

We can’t decode JWTs on the frontend! As mentioned, we set up OAuth2 Proxy to encode and encrypt JWTs in cookies. Such cookies can only be decrypted on servers after it has hit the gateway.

Secure, adopted approach:

We simply implement a user info endpoint on one of our backend services or as a backend service. Since all requests are routed through the gateway and JWT is visible to the backend, we can use IdToken to populate user information and return it to the user. We may place user data in a redux or context for future reference. User data obtained this way doesn't contain any sensitive information used for authentication.

Optionally, we may also include access control data using AccessToken. This also ensures the user is signed in and JWT is valid when we return the response.

Similarly, we talk to Auth0's management API for user metadata updates in our backend using their management tokens and return the response to the user.

Server-side rendering

In any SPA, it's not possible to redirect users with a 302 response (or return any different response code using frontend implementation). I implemented our static site redirection using custom frontend components and JS redirects.

Why 302 response may be preferred:

• The user doesn’t need to load the frontend page before entering the redirect loop
• Faster response time

Low impact on site performance; Two ways to handle this:

• Place any static frontend behind Gateway and OAuth2-Proxy (i.e., like a backend service)
• Use server-side rendering