Over the past months, a wave of technical content has emerged—deep dives into the Agentcore runtime, gateway integrations, memory persistence models, tool management, and more. These resources are incredibly valuable for understanding how to use the platform. But as the adoption curve steepens, it’s equally important to step back and look at the bigger picture: we are heading toward a world where distributed, specialized AI systems become the norm.

Agentcore makes it easier to build these components, but it also increases the risk of teams assembling complex systems without a solid architectural foundation. Without thoughtful system design, standardization, and distributed-systems thinking, organizations may unintentionally create fragile, siloed, or overly coupled AI ecosystems. The challenge—and opportunity—now is to balance speed of development with robust design principles that ensure these emerging agent-based architectures remain scalable, observable, secure, and maintainable.

This article explores that broader context: what Agentcore’s GA really means for distributed system design, why standardization matters now more than ever, and how teams can avoid the architectural pitfalls that come with rapid innovation.

Distributed Design

Modern enterprise products are, at their core, distributed systems—whether teams consciously design them that way or simply evolve into them over time. As organizations scale, their applications naturally break into interconnected services, workflows, and data domains that must communicate reliably. In this environment, boundaries matter: clear ownership, well-defined responsibilities, and stable interfaces determine whether a system grows sustainably or accumulates silent complexities that surface only during failures.

Distributed systems introduce challenges that are fundamentally different from writing business logic. Engineers may thoroughly understand the domain, the models, and the rules they implement—yet still find themselves frustrated when the real difficulty emerges not from the logic itself, but from the interactions between components. Latency, partial failures, race conditions, version drift, inconsistent state, and unclear integration contracts can turn even well-designed services into unpredictable systems. This is often what makes distributed systems feel maddening: you can master the business logic and still get blindsided by emergent behavior rooted in architectural decisions.

These pitfalls are not signs of poor engineering but symptoms of systems where boundaries are blurred, responsibilities overlap, and integrations evolve informally rather than intentionally. Without deliberate system-design thinking—defining boundaries, standardizing communication patterns, enforcing contracts, and anticipating failure modes—distributed architectures tend to drift. Over time, they accumulate complexity that becomes harder to reason about, harder to extend, and harder to trust.

In today’s enterprise landscape, understanding distributed-systems principles is no longer optional. It is the foundation that ensures the entire product ecosystem remains resilient, observable, and adaptable as it grows.

Core Concepts of Distributed Systems

Distributed systems succeed or fail not because of any single component, but because of how those components interact, evolve, and respond to real-world conditions. Mastering a few foundational concepts helps teams design systems that stay reliable as they grow, adapt to change, and avoid the hidden complexity that often emerges at scale. The following principles offer a baseline for building resilient, predictable, and maintainable distributed architectures.

Communication: Communication is an unavoidable glue of distributed systems, and every interaction carries risks: latency, timeouts, and partial failures. Choosing clear patterns—sync, async, events—helps manage uncertainty and keep components predictable.

Boundaries: Boundaries create separation of concerns, ensuring each service has a clear purpose. Well-defined boundaries reduce coupling and prevent hidden dependencies that make systems fragile and hard to evolve.

Specialization and Ownership: As systems grow, components must specialize to solve focused problems. Strong ownership ensures each one is maintained, monitored, and improved consistently, preventing drift and conflicting assumptions.

Contracts: Contracts define how components interact—what they expect, provide, and guarantee. Strong, explicit contracts reduce integration surprises and make changes safer in evolving architectures.

Scalability: Scalability ensures systems can handle growth smoothly. Components should scale independently and avoid shared bottlenecks to prevent performance drops during spikes or expansion.

Unintelligent Distributed System

Traditional distributed systems are powerful but inherently unintelligent: every service is specialized, yet entirely dependent on human-designed logic, rules, and constraints. Their behavior reflects the limitations of the engineers who built them—their time, cognitive bandwidth, domain knowledge, and ability to anticipate edge cases. Even well-architected systems can only operate within the boundaries explicitly encoded into them. They cannot adapt, reason, or make decisions beyond their predefined paths. This creates a landscape where services are efficient at what they were built for but rigid to change, reliant on manual intervention, and limited by the expertise available at design time.

Doctor vs Pharmacist

A helpful way to understand knowledge limitations in decision-making is to compare a doctor and a pharmacist. A pharmacist has deep, specialized expertise in medications—their composition, interactions, and safe usage. But when asked to diagnose a condition, their view is constrained; they lack the broader medical context required to interpret symptoms, evaluate risks, or consider alternative explanations. A doctor, on the other hand, combines a wide range of medical knowledge with diagnostic reasoning, allowing them to form a comprehensive picture before making a treatment decision. Both professionals are highly skilled, but the scope of their knowledge determines the quality and safety of their decisions. In complex systems, just like in healthcare, narrow expertise without broader understanding increases risk, while wider knowledge enables deeper analysis and more reliable outcomes.

Amazon Bedrock Agentcore

Amazon Bedrock Agentcore introduces a standardized foundation for building intelligent, agent-driven systems within an enterprise ecosystem. The Agentcore runtime provides a unified execution environment where agents can reason, call tools, manage memory, and orchestrate multi-step workflows without developers having to reimplement these patterns for every use case. It abstracts away the complexity of handling context, chaining operations, and maintaining state, enabling teams to focus on the logic that defines their domain rather than the mechanics of agent interaction.

Complementing the runtime, the Agentcore gateway streamlines communication between agents, services, and external systems. It acts as a controlled interface layer—managing routing, authentication, rate limits, and standardized behavior across all agent interactions. This consistency reduces integration overhead and ensures that agents can be safely exposed within larger architectures.

Together, the runtime and gateway dramatically accelerate product development. Instead of building custom orchestration layers, memory managers, or tool-integration pipelines, teams can assemble specialized AI-powered services much faster. The result is a shift from manual glue code and bespoke patterns to a repeatable platform where intelligent, domain-specific components can be created, tested, and deployed with greater speed and reliability.

E-Commerce System

To illustrate how Agentcore can accelerate intelligent system design, consider a typical e-commerce ecosystem. At its core are specialized services:

• Product Service – manages catalog information, attributes, and inventory.

• Order Service – handles shopping carts, checkout, and payment processing.

• Shipping Service – coordinates delivery options, tracking, and logistics.

• Listing Page – serves as the customer-facing interface for product search and ordering.

In a traditional distributed setup, these services communicate via well-defined APIs. The Listing Page queries the Product Service to display items, calls the Order Service to process purchases, and interacts with the Shipping Service to show delivery estimates. Each service is specialized and efficient within its domain, but interactions are linear and rigid, and the customer experience is constrained by the explicit logic programmed into each component.

Agentcore introduces the potential for agentic orchestration across these services. Agents can dynamically coordinate tasks, enrich queries, and reason over multiple services simultaneously. For example, a conversational search agent could interact with the Product, Order, and Shipping services to provide contextual recommendations, personalized bundling, or delivery-aware product suggestions—all in a single, seamless interaction. This approach transforms rigid service chains into adaptive, intelligent workflows, enabling faster innovation and more engaging user experiences.

Adopting an Agentic Approach with Agentcore

Agentcore enables an agentic layer over existing services using its runtime and gateway components. Here’s how the e-commerce system can leverage them:

• Conversation Search Chat UI Component – Acts as the frontend interface for customers. It communicates with the Listing Agent via the Agentcore gateway, ensuring safe, standardized interaction with the agent layer without exposing backend complexity.

• Listing Agent (Runtime) – Maintains context and memory for the conversation, coordinates between agents, and manages multi-step workflows. It handles complex tasks like interpreting user queries, maintaining session state, and orchestrating data retrieval across the system.

• Ordering Agent (Runtime) – Interacts with both the Product and Shipping services to generate accurate, context-aware order recommendations. The runtime allows it to reason over inventory, pricing, and delivery constraints dynamically, providing qualified results to the Listing Agent.

• Product Gateway – Serves as a bridge between the Product Service and agents. Using OpenAPI-spec integration, the gateway ensures that requests from agents are routed correctly and consistently, without agents needing to manage low-level API details.

• Shipping Gateway – Functions similarly for the Shipping Service. It provides standardized access for agents, enabling safe queries and updates regarding delivery options, tracking, and logistics.

By combining Agentcore runtime for backend orchestration (Listing and Ordering Agents) with Agentcore gateways for safe, standardized access to existing APIs (Product and Shipping), the system gains a layer of intelligence on top of existing services. This setup enables conversational search, adaptive ordering workflows, and dynamic coordination between services—all without modifying the underlying APIs or services.

Traditional Service Integrations

Traditional service integrations rely heavily on the foundational concepts of distributed systems:

1. Communication – Teams coordinate API contracts, call patterns, and failure handling.

2. Boundaries – Clear separation of responsibilities between services and teams prevents accidental coupling.

3. Specialization and Ownership – Each team owns a service domain, ensuring consistent updates and reliability.

4. Contracts – Deterministic API specifications define exactly how services interact.

5. Scalability – Human planning ensures that services can grow sustainably, avoiding bottlenecks or cascading failures.

In these systems, half of the safety and stability comes from human oversight: collaboration, code reviews, discussions on edge cases, and shared understanding of trade-offs. Engineers act as the implicit safety net, catching inconsistencies, reasoning about system-wide effects, and maintaining trust between services.

Allowing agents to fully satisfy these integrations without careful supervision introduces significant risk. Unlike humans, agents may interpret, combine, or explore service interactions in unexpected ways, bypassing implicit constraints and assumptions. This can result in invalid requests, unpredictable workflows, or even cascading failures—putting the reliability and safety of the system at stake. Clear human-defined boundaries, specifications, and guardrails remain essential when introducing agentic layers into established service ecosystems.

Agentic Contract Pitfalls

In traditional distributed systems, contracts are deterministic: each service exposes a set of well-defined operations, with clear input and output formats. Consumers of these services rely on the documented capabilities, and trust comes from a shared, stable understanding of what each service can provide. Exploration and communication are tightly bound to this deterministic model, which minimizes surprises and ensures predictable behavior.

In an agentic system, the dynamics change. Agents—often powered by foundation models (LLMs)—attempt to reason, analyze, and construct the “best match” for a given task. This introduces uncertainty: even if the underlying services remain stable, the agent’s interpretation, reasoning, or creative combination of operations can produce unintended behaviors, which may propagate through the system. Without careful constraints, the agent itself becomes a potential single point of failure, capable of generating invalid requests, misusing service capabilities, or creating inconsistent workflows.

To avoid these pitfalls, well-defined service specifications are critical. Each service should provide:

• Clean summarization and description – a concise overview of what the service does and the intended use cases.

• Examples – illustrative input-output pairs showing correct usage.

• Data formats and types – clear definitions of fields, expected types, ranges, and constraints.

• Possible values and enumerations – to restrict agent actions to valid options.

By providing precise, structured, and machine-readable specifications, teams can guide agents to safely explore and reason over services, reducing the risk of unintentional behavior. Agentic contracts require both human clarity and machine interpretability to ensure that agents enhance workflows without introducing instability, while preserving the trust that traditional deterministic contracts provided.

Agentic Communication Pitfalls

In traditional distributed systems, communication is deterministic and predictable: services exchange well-defined requests and responses, and errors or retries are explicitly handled according to agreed protocols. Agents, however, introduce a fundamentally different mode of communication.

Agents can decide on their own how to handle situations, which may lead to unexpected behaviors. For example:

• Ignoring contract priorities – an agent may remove search criteria or skip fields without understanding their importance, violating implicit assumptions in the service contract.

• Infinite retries – in response to failures, an agent may loop endlessly, potentially overwhelming services or consuming unnecessary resources.

• Unintended actions – agents may execute operations that deviate from the original user request, misinterpreting intent or optimizing incorrectly.

These behaviors are non-deterministic and can propagate across the system, creating unpredictable workflows and system instability. Unlike human operators, agents lack an innate sense of context, risk, and priority unless explicitly constrained. Proper guardrails, monitoring, and well-defined service specifications are essential to prevent agentic communication from becoming a source of errors or cascading failures.

Agents Are Not Doctors

Agents, even when powered by advanced foundation models, cannot become truly specialized in a domain without a supporting agentic ecosystem. Unlike human experts, their reasoning depends entirely on the data and context they are provided. GenAI models come pre-trained on vast amounts of information—both accurate and misleading, relevant and irrelevant—which creates a noise-to-signal challenge. This information overload makes it difficult for agents to consistently focus on what truly matters, increasing the risk of misinterpretation, inconsistent decisions, or unsafe actions.

What is often missing is clean, structured, and meaningful data: well-defined decision rules, product and service specifications, clear contracts, and curated examples. Without these foundations, agents may produce plausible-sounding but incorrect results, overlook priorities, or misapply rules. In short, agents cannot become “doctors”: they lack the domain expertise and curated knowledge that human specialists acquire over years. Specialization requires not just intelligence, but a carefully constructed ecosystem that constrains, guides, and informs agent behavior—turning raw potential into reliable, context-aware expertise.

Conclusion

The adoption of an agentic approach is becoming increasingly evident, as enterprises look to leverage generative AI and intelligent agents to build more adaptive, responsive, and specialized systems. However, success depends on more than just deploying agents: it requires clean and optimized data, detailed product specifications, and precise service contracts. Teams must also focus on reusability and well-defined boundaries to maintain clarity, reduce coupling, and ensure predictable behavior. Rushing adoption without these foundations risks cascading, non-deterministic communication, which can compromise reliability and system integrity. Thoughtful design, structured data, and disciplined system practices remain essential to unlock the full potential of agentic architectures while maintaining control and trust.

This part focus on Asynchronous design and related patterns and practices

Real Scenarios

Before starting patterns and practices, let’s look at a real scenario where the producer rate could impact downstream services.

Migrating Provisioned DDB to OnDemand

Recently, AWS reduced the cost of DynamoDb onDemand by 50%. This announcement led to many migration decisions. When we moved to On-Demand mode for one of our critical and core services, we recognized a higher processing latency and error rate in one of downstream services. While the overall service’s functionality was not impacted ( an old serverless system ), the delays and latency introduced by a higher rate of errors had a real business impact.

The above diagram demonstrates a typical design in many companies and works perfectly at some scale. Still, it can be dangerous when the processing rate exceeds the predicted rate. This design introduced significant OpenSearch queries’ latencies by migrating Dynamodb table to OnDemand.

Some metrics during investigations showed a higher rate of successful write requests compared to the provisioned mode before migration. In Provisioned mode, requests were throttled but succeeded after retries and an exponential backoff strategy for errors allowed balancing the instant peak to a wider time window by delaying it.

Fortunately ( or unfortunately ), experimenting with throttling and retrying for UpdateB allowed a decrease in the processing rate at a given time, helping downstream to continue processing or by giving downstream the chance of recovery. Migrating to OnDemand gave more capacity at initial migration to the producer (4 partitions 12000 RCU, 4000 WCU).

Too many changes increased the load on the downstream service, which in turn led to degraded performance of the downstream service using OpenSearch, the impact was due to the increased number of queries in OS internal buffer and, consequently, increased latency. These latencies lead to increased error rates for Lambda triggers. While not a real-world issue, using Bisect and a large batch size does increase the load on the downstream side.

Design Problems

• Distributing many changes at the same time

• Introducing the database changes instead of Concrete events representing the final and viable state at a given time (processing time)

• Introducing Coupling between two specific services DDB stream and OpenSearch

• Propagating Errors

• Big BatchSize with Bisect On Failure

High-Level Async Design

In a Timely-Balanced scenario, the Count of instantaneous occurrences and Speed are reduced. Concurrency can consequently be improved but needs to be controlled at a layer based on possibilities.

In an Instantaneous scenario, the Count, speed, and concurrency are increased, which leads to more capacity needed on the downstream side.

While at some scale, Time-Balanced and Instantaneous scenarios produce some close functional results, at a larger scale, more challenges can be considered and refined intentionally. The following high-level diagram can better represent the challenge.

In Part 1 of this series related to Rate Controlling, four attributes were explored: Time Window, Concurrency, Processing Speed, and Count, but another important attribute, State was also under focus.

Producer & Consumer Integrations

While AWS provides many services that allow and simplify integrations in an Event-drive design, each service is purpose-built and has dedicated pros and cons. The following diagram showcases some available integrations. (The Lambdas Presented in this diagram are just to demonstrate a computing resource, but this can be a Fargate container or any other service with computing/processing capability)

The “Invocation Models and Service Integrations” topic is deeply explained in Chapter 3 of Mastering Serverless Computing with AWS Lambda

In above diagram, three category of integrations are discussed:

• Streams

• Pub/Sub

• Queues

Streams

Streams such as DynamoDb streams or kinesis are ideal for near realtime data streaming even in high throughput scenarios with massive volumes of data. They guarantee ordering, provide means to manage highest Write throughput requirements, and support handling large amount of data.

Pub/Sub

Services providing Publ/Sub such as Event-Brdige or SNS have the capacity to distribute any single messages to 1 or many subscribers near realtime and offer high throughput allowing to manages high number of distinct messages, per design they operate in an fully asynchronous way which means they dont track the consumer’s processing status.

Queues

Queue systems such as SQS guarantee delivery and provide mechanisms for resiliency, load leveling, and decoupling. They provide simplified means to prevent message loss such as keeping messages in the queue till expirations and redrive policies.

General Recommendations

This section provides general guideline to better control the rate and load on downstreams and not focusing on design practices but adopting some level of practices to develop some safeguards.

Batching allows to reduce the rate of statelessness by providing related changes side by side that let the software code have more context to take decisions. Reducing the concurrency can be achieved in different way for each category but at the end a strict concurrency configuration such as ConcurrencyLimit with SQS, ReservedConcurrency with SNS/EB , or Partitioning with Streams allows reducing the execution rate and consequently the risk of overloading the downstream. Synthesizing messages allows being able to produce the final state at that given time by looking at a time ordered batch of related records.

Source Code

SQS integration

Describing how Amazon SQS works internally is out of scope, but at a high level the polling process of AWS lambda with SQS can be illustrated by following diagram

Aws lambda Asks for Messages and SQS returns available messages from sample nodes, and this is the SQS default behavior, By configuring a longer wait time than zero (default value for short polling), long polling will be applied and SQS will do a best effort to gather more messages from all nodes.

Batching: if a BatchSize specified this will improve the chance of proper batching, allowing more messages to be grouped and allows lambda to have more knowledge about state, while this state is yet distributed and not 100% localized, it will be useful and improve processing quality at some level. Using maxConcurrency option alongside batching allows to have more localized state by increasing the chance of having messages in the same batch. This can be configured using CDK as below.

const queue = new Queue(this, 'Queue', {
   receiveMessageWaitTime: Duration.seconds(20),
   deadLetterQueue: {
     maxReceiveCount: 3,
     queue: new Queue(this, 'DeadLetterQueue')
   }
});

lambdaFunction.addEventSource(new SqsEventSource(queue, {
   batchSize: 10,
   maxBatchingWindow: Duration.seconds(60),
   maxConcurrency: 2,
}));

But Lambda Event Source Mapping provides another configuration option maxBatchingWindow that can be used allowing the lambda service to defer invocation up to 5 minutes while asking for more and more messages form SQS and a function is invoked when one of the following conditions is met as payload size reaches 6MB, Max Batching Window reaches its maximum value, or the Batch Size reaches its maximum value.

The lambda now has more context and can apply more control and synthesize messages. The following snippet shows how in typescript this can be achieved.

const records = (event.Records as SQSRecord[])
  .sort((a, b) => { return a.attributes.ApproximateFirstReceiveTimestamp.localeCompare(b.attributes.ApproximateFirstReceiveTimestamp); })
  .map((record) => { return { ...record, body: JSON.parse(record.body) }});

const grouped = Object.groupBy(records, (currentValue: recordType) => currentValue.body.subject);

const results: SQSRecord[] = [];

grouped.forEach((entry) => {
    const entryValues = entry?.[1] as any[];
    let firstEvent = entryValues?.splice(0,1)?.[0];
    const initialType = firstEvent.body.type;

    entryValues.forEach((current: any) => {
        const currentData = current.body.data;
        const currentType = current.body.type;

        if(initialType == 'user.signedup') {
          if(currentType == 'user.profile_updated') {
            firstEvent.body.data.profile = {
              ...firstEvent.body.data.profile, 
              age: currentData.age,
              nickname : currentData.nickname,
              prefered_channels: currentData.prefered_channels
            };
          }
          else if(currentType == 'user.unsubscribed') {
              firstEvent = null;
          }
        }
    });
    results.push(firstEvent);
});

Sending two messages related to the same id will group them and by synthesizing will avoid distributing or doing unnecessary processes. The following json example represents two distinct events’ payload sent to SQS.

{
  "id": "dmPvPSlwFZOTOjefpu8m2",
  "time": "2025-03-23T17:13:32+00:00",
  "source": "user.management",
  "subject": "omid_unique_user_id",
  "data": {
    "name": "omid",
    "profile": {
       "age": 40
    }
  },
  "type": "user.signedup"
},
{
  "id": "TAF5aEQeAh7GX1q8Cvam3",
  "time": "2025-03-23T17:14:32+00:00",
  "source": "user.management",
  "subject": "omid_unique_user_id",
  "data": {
    "name": "omid",
    "age": 43,
    "nickname": "xaaxaax",
    "prefered_channels": [ "EMAIL" ]
  },
  "type": "user.profile_updated"
}

The Example will distribute a final and single user.signedup event representing a combination of both events.

{
  "id": "dmPvPSlwFZOTOjefpu8m2",
  "time": "2025-03-23T17:13:32+00:00",
  "source": "user.management",
  "subject": "omid_unique_user_id",
  "data": {
     "name": "omid",
     "profile": {
        "age": 43,
        "nickname": "xaaxaax",
        "prefered_channels": [ "EMAIL" ]
     }
  },
  "type": "user.signedup"
}

While the above example provide some details to better control the rate and reducing the risk of overloading downstreams but this approach has no guarantee of control on top of single entities even if in above example the goal was achieved, this means it is possible to have two above messages in two distinct invocations while each invocation receives the respective batch but random entities.

Partitioning: While Batching can be useful but at some points two related messages can be presented in two distinct invocation ( execution environment ) even they are close in terms of time, Grouping related entities and guarantee that they will reach the same invocation (exp. user.signedup and user.profile_updated events related to same user). This can be configured using CDK as below.

const queue = new Queue(this, 'Queue', {
   receiveMessageWaitTime: Duration.seconds(20),
   fifo: true,
   deadLetterQueue: {
     maxReceiveCount: 3,
     queue: new Queue(this, 'DeadLetterQueue', { fifo: true })
   }
});

lambdaFunction.addEventSource(new SqsEventSource(queue, {
   batchSize: 10,
}));

Sending the previous message payloads presented in batching section, following SQS messages requests must be send to SQS, the presence of MessageGroupId guarantees that at a given time all messages for the same user ( MessageGroupId ) will be strictly received together.

 {
    "Id": "Test-0001-2015-09-16T140731Z",
    "MessageGroupId": "omid_user_id",
    "MessageDeduplicationId": "dmPvPSlwFZOTOjefpu8m2",
    "MessageBody": "{\r\n  \"id\":\"dmPvPSlwFZOTOjefpu8m2\",\r\n  \"subject\":\"omid_user_id\",\r\n     \"time\":\"2025-03-23T17:13:32+00:00\",\r\n      \"data\":{\r\n         \"name\":\"omid\",\r\n         \"profile\":{\r\n            \"age\":34,\r\n            \"nickname\":\"omid\"\r\n         }\r\n      },\r\n      \"type\":\"user.signedup\"\r\n   }"
 },
 {
    "Id": "Test-0002-2015-09-16T140930Z",
    "MessageGroupId": "omid_user_id",
    "MessageDeduplicationId": "2dokCjxmQvUk2brjLuus1",
    "MessageBody": "{\r\n  \"id\":\"2dokCjxmQvUk2brjLuus1\",\r\n  \"subject\":\"omid_user_id\",\r\n    \"time\":\"2025-03-23T17:14:32+00:00\",\r\n      \"data\":{\r\n         \"name\":\"omid\",\r\n         \"age\":43,\r\n         \"nickname\":\"xaaxaax\",\r\n     \"prefered_channels\": [ \"EMAIL\" ]},\r\n      \"type\":\"user.profile_updated\"\r\n   }"
  }

This example apply a mix of Partitioning and Batching allowing to give more change to lambda have more consistent state which is managed by SQS internally.

DynamoDb / Kinesis Stream Integration

Before deep diving into Streams integrations, this section elaborate on how the integration works. The following diagram illustrates how Items, Item Collections, and stream records are organized.

Aws lambda Asks for List of shards ( aka. Partition in Dynamodb ), Now Lambda handle each shard process in isolation, and starts by getting the Shard Iterator ( aka. start pointer in streaming concept ), giving the iterator the stream returns the items in response. Knowing that each item collection ( items related to same PartitionKey ) is always located in a single Shard the lambda already benefit from having all related data as by default a single lambda execution is dedicated to each shard at a given time which means at least for a single PartitionKey there is a single running invocation.

sequenceDiagram
    participant A as Event Source Mapping
    participant B as Stream
    participant C as Shard
    A->>B: Ask for list of Shards
    par Shard 1
        A->>C: ask for Shard Iterator
        A->>C: Get Records in a Shard using Iterator
        C->>A: Respond back batch of messages
    and Shard 2
        A->>C: ask for Shard Iterator
        A->>C: Get Records in a Shard using Iterator
        C->>A: Respond back batch of messages
    end

Using AWS CDK, the integration can be refined to give more flexibility to make concrete decisions. The following snippet shows how the event source mapping can be configured.

lambdaFunction.addEventSource(new DynamoEventSource(table, {
      startingPosition: StartingPosition.LATEST,
      batchSize: 10,
      maxBatchingWindow: Duration.seconds(60),
      bisectBatchOnError: true,
      retryAttempts: 3,
      maxRecordAge: Duration.minutes(15),
      parallelizationFactor: 1,
}));

A batch of 10 records and a 60 seconds batching window is configured, allowing reception of more related records that consequently will allow to reduce the overload on downstreams.

Sending a new item to DynamoDb table and updating it

// New Item
{
    "PutRequest": {
        "Item": {
            "pk": {"S": "omid"},
            "sk": {"S": "t0000"},
            "nickname": {"S": "Omid"},
            "age": {"N": "30"},
            "prefered_channels": {"S": "EMAIL"},
            "status": {"S": "ACTIVE"},
        }
    }
}

// Update Item
{
    "PutRequest": {
        "Item": {
            "pk": {"S": "omid"},
            "sk": {"S": "t0000"},
            "nickname": {"S": "XaaXaaX"},
            "age": {"N": "43"},
            "prefered_channels": {"S": "EMAIL"},
            "status": {"S": "MDIFIED"},
        }
    }
}

Looking at handler code the grouping is done per PK and verification based on eventName being INSERT , MODIFY , or REMOVE.

  const grouped = Object.entries(
    Object.groupBy(records, (currentValue: DynamoDBRecord) => currentValue.dynamodb?.Keys?.pk.S || '')
  );

  const results: DynamoDbInternalRecord[] = [];

  grouped.forEach((entry) => {
    let key = entry?.[0];
    let entryValues = entry?.[1] as DynamoDBRecord[];
    let firstEvent = entryValues?.splice(0,1)?.[0];
    const initialType = firstEvent.eventName;

    let data: Record<string, any> = {};
    if(initialType == 'INSERT' || initialType == 'MODIFY')
      data = unmarshall(result.dynamodb?.NewImage as DynamoDbInternalRecord);
    else if(initialType == 'REMOVE')
      data = unmarshall(result.dynamodb?.OldImage as DynamoDbInternalRecord);

    entryValues.forEach((current: any) => {
      const currentType = current.eventName;
      let currentData: Record<string, any> = {};
      if(initialType == 'INSERT' || initialType == 'MODIFY')
        currentData = unmarshall(current.dynamodb?.NewImage as DynamoDbInternalRecord);
      else if(initialType == 'REMOVE')
         currentData = unmarshall(current.dynamodb?.OldImage as DynamoDbInternalRecord);

      if(initialType == 'INSERT') {
        if(currentType == 'MODIFY') {
          data.age = currentData.age;
          data.nickname = currentData.nickname;
          data.prefered_channels = currentData.prefered_channels;
        }
        else data = {};

      }
    });
    results.push(data);
  });

The printed result for the above records will be as below

{
    "pk": "omid",
    "sk": "t0000",
    "nickname": "XaaXaaX",
    "age": 43,
    "prefered_channels": "EMAIL",
    "status": "ACTIVE"
}

EventBridge / Sns Integration

SNS and EventBridge services will allow a Fanout design with multiple consumers (subscribers). SNS/EB, even though the event contract for SNS contains an array of records ( Records[] ), the lambda always receive a single message in the array ( ref in FAQ: here ). There is no possibility of refined batch processing and custom batching configuration that derives a higher level of concurrent execution of lambdas and a load without the possibility of control. The distribution is an asynchronous call, This means the Channel will not be aware of processing results in consumer side.

• Reserved Concurrency: In cases of direct integration with SNS or Eventbridge, the solution to limit downstream impact is to apply reserved concurrency.

• OnFailure Destination: Integrate a DLQ. The default behavior is 3 retries with default intervals between each invocation retry (unchangeable):

  • 1st Retry: 1 minute

  • 2nd: 2 minutes

• Load Leveling: Use an SQS as an intermediate layer to facilitate load balancing.

    const lambdaFunction = new NodejsFunction(this, 'LambdaZipFunction', {
      reservedConcurrentExecutions: 1,
      ...
    });

    const topic = new Topic(this, 'Topic');

    lambdaFunction.addEventSource(new SnsEventSource(topic, {
      deadLetterQueue: new Queue(this, 'DeadLetterQueue'),
    }));

Conclusion

Driving design decisions often lies on top of scalability and High-Throughput, While this is an ideal approach, this can become tricky and will impact the business process when boundaries are not well defined and communications are not well thought ( Which is often the case ). Keeping track of surroundings of a service while making such decisions is a must to have and will save time and effort in long term. While crossing out-of-control boundaries ( external system, partners, Saas, etc. ) is not a major case but when exists will cost companies effort, downtime and disruption.

Asynchronous designs give more power and control at communication boundaries but they come with some process oriented decisions by reducing concurrency as a safeguard but also speed for the sake of having more on the fly State consistency.

This part of series explored some details about typical async services and some integration details to control the load on downstream services. The next part of this series will focus on patterns and practices related to rate control in synchronous scenarios where the final product is highly coupled to an external service or downstream with limitations and constraints.

While focusing on core business is essential, external services often present challenges and highlight several considerations. Those services introduce quotas, limits, or constraints that vary based on Business plan, Architecture, or Infrastructure. Whatever the reason, any consumer must plan enough guardrails to isolate the internal process from external services, and those considerations differ based on the layer in the internal system that lives with that coupling.

Coupling Level

It is not avoidable to decouple all the external dependencies, and sometimes, a business needs to integrate some external service to gain complicated expertise such as payment, financing, insurance, etc. But at the same time, there is no way to persist all data for systems such as payment, and even for others such as insurance and financing, there is no possibility to predict all end-user interactions. The coupling can be established in two ways, Direct or Indirect.

Direct

Direct here does not mean Request / Response but more about how the end user product is coupled to the external systems. Coming back to the insurance example, the scenario can be described as:

• A Product is available for a user.

• The Product is eligible for N years’ insurance.

• The User can modify the insurance default details.

• The insurance price is computed per modified details.

• The user makes the decision.

When a user changes the insurance duration, this results in a price change and the system needs to interact with the insurance partner service, which, brings the risk of introducing a new point of failure in the system.

Indirect

The above scenario mentions the default value for any single product, While these defaults are essential for the end user, they are often close to being static for a while. However, those values don’t relate to the product but to the product category and characteristics. As an example, a Combined Refrigerator ( Frigo/Freezer) with an E-level energy class can have a higher insurance price than an A-class ( just assumptions ), Maybe that price can be lower for well-manufactured and popular brands like SIMENS, AEG, and BOSCH.

The above diagram illustrates a backend service interacting with the partner for product Creation/Modifications. This helps reduce the process coupling level to partner for all product detail page hits by persisting the defaults related to any product Category projected to all available products.

Coupling Layer

Focusing again on the article’s topic, defining a rate-controlling strategy depends on the layer's position interacting with the external system, which drives the corresponding design and implementation patterns. Each layer offers distinct possibilities to solve the problem.

Client Side

When a client-side application, such as a Web or Mobile application, interacts with an external system directly or via a gateway proxy, any external system behavior will be replicated in the client app. This can be any latency, throttling, or internal failure problems. In case of rate issues, the client app will receive a 429 HTTP status code, indicating “Too Many Requests”. The client can retry the call and hope for a successful response. The problem with the client side is that there is no visibility about the overall rate of errors at a given time, and each client app runs with a local state.

Having millions of users means many users can find a single product details page, so they all need to interact with external insurance partners directly or via a gateway proxy. However, the important takeaway is that no single user knows about the overall load on the systems and how many retries are inflight to the external system.

Edge Side

If the process layer interacting with the external system is an edge layer, an edge-level state can help to keep track of the state, this allows to lead better the client apps and also avoids external system calls by applying some level of caching. But this layer adds a challenge of How zonal distributed edge locations can answer the regional rate challenge, while the rate is dedicated to a single regional communication boundary, many edge locations can send requests. However, this problem is many times less important than the client-side problem.

Earlier about the edge, the paragraph mentioned that the edge locations yet follow the same distributed state problems. While this is true, in the real world, Geographic insurance and pricing are definitively different, and this can be an unrealistic challenge except for some rare use cases.

Another option when using content delivery networks ( CDN ) is the “Request Collapsing” feature; it stands for serving the same origin response for many inflight requests, so the first request reaches the origin while the same requests at a given time are paused while waiting for the response to the first request. this reduces the origin load and, consequently, the external service load. ( The topic of CDN Request Collapsing is deeply discussed in Chapter 9 of Mastering Serverless Computing with AWS Lambda )

Server Side

The server layer is where all the autonomy and power reside. The server-side layer has access to computing power and is closer to the external system. It can be the best for keeping track of the rate limit's actual state and monitoring the actual consumed threshold. Using the server layer helps reduce the challenge of statelessness. However, this may be a more difficult layer for applying retries and exponential backoff of throttled requests, as this will increase the rate of suspended processor resources. The server layer allows synchronous or asynchronous interactions with the external systems. The choice of async or sync varies based on different constraints, and the Coupling level that can be Direct or Indirect.

Direct coupling choice will be synchronous as the end user product waits for the response, this means the interaction with the external service must be done before sending back the response.

All these assumptions can be real when a single container serves the backend service. However, it will suffer from the Concurrency challenge that will be explored later in this article and distributed state if the service is designed for high availability and scalability.

In the case of indirect coupling, an asynchronous backend process is a more flexible solution for interacting with an external system. Often, the async processes are on top of managed services such as SQS, Event Bridge, etc., and react to changes via messaging. This can be an ideal design, but the scaling capacity of managed services can become a point of reflection when interacting with external systems. This can be a real concern if the internal system is designed on top of Function as a service, in which the risk of high concurrency becomes a real concern.

Although the invocations are driven by pollers, each execution environment can receive a batch of records, giving more control to the consumer at the container level. However, the fact of batching by default is related to many factors such as speed, time window, and number of changes.

Processing Main Attributes

A variety of attributes can play a role in the rate of communication toward the external systems such as:

• Time Window

• Concurrency

• Speed

• Count

Time Window

A time window represents the time duration that a constraint applies, it is often a per-second metric, but there are services with per-minute or per-hour constraints. Whenever using a service with time window constraints, the base challenge will be, “How do we adapt the internal rate in a dedicated time window based on external service constraints?”

Concurrency

Concurrency is the ability to run many tasks simultaneously to increase some quality attributes such as Throughput and Scalability. A higher concurrency level can put more pressure on downstream, so crossing the rate constraints and thresholds. Controlling concurrency is a real challenge in distributed systems, and to achieve that control, a decrease in processing speed or throughput will be necessary. Another challenge in highly concurrent processes is state consistency, which can only be shared ( not done ) in a consistent way by context switching that is a complex process and, if achievable, will not allow the control of treatment but only sharing states by the complexity and cost of the real-time context switching process.

Speed

The speed represents how fast a system treats the demands. A higher speed means a higher level of statelessness, and achieving state means reducing the treatment speed. The system speed must be aligned with the overall business value, and there is no interest in driving fast while putting everyone in danger.

Count

The number of demands significantly impacts downstream services. Responding to a high number of demands is toward responsiveness, but how downstream services can handle that rate brings more design-related discussions. However, dealing with external systems becomes tricky because an external system has many more customers than our internal system. Sometimes, a subscription change and, sometimes, a move to another vendor can be the solution. Aligning two roadmaps is not achievable. Controlling the number of demands depends on the layer of coupling.

Trustable State

The rate controlling in distributed systems belongs to consistent, performant, and highly available persistent storage. Why those quality attributes are important?

Achieving a shared state consistently relates to many factors when dealing with distributed storage and will solve the most important challenge of controlling the State.

Availability: is important as unavailability in one node can lead to a stale state. so this will be important to fetch from the most consistent node or fetch and cache the state if any node is replaced. Yet, With all efforts, having a real-time trusted and available state is not an option if we talk about nano or milliseconds.

Performance: leads to lower latency, and lower latency means less overhead on storage, however, high latency will result in some shared state synchronization problems.

Consistency: The state must be consistent over different nodes in near real-time, updating the persisted state must be consistent for all subsequent reads.

DynamoDb, S3, and ElasticCache are some available options on AWS. Many other interesting options, such as Momento, can help achieve a distributed share state.

Conclusion

Designing distributed systems and business processes at scale leads to more inter-service communication needs, while in the best cases, the value is evident, there will be the worst cases that will lead to frustration because of the pressure a service puts on downstream services.

Some services add guardrails such as rate limiting to safeguard their infrastructure health, but this impacts the consumers and based on the presented coupling can impact the final products and users.

In this part of “Rate Controlling in Distributed Systems”, some concepts related to Rate Controlling, such as Coupling Level, Coupling Layer, Processing Main Attributes, and Trustable State, are explored to give an overview of how the rate is measured.

The new Part of the Series will explore some patterns and technical details to reduce the risk and control the rate in both Sync and Async designs.

Some typical dependencies are:

• Libraries and Frameworks

• Container Images

• Runtimes

Importance and necessity

It seems important, on paper, to keep everything up to date all the time, but in reality, the importance relies on the following axis:

• Criticality: How Critical is that update?

• Life Cycle: How frequently are new versions released?

• Effort: How many projects shall be changed?

Let’s explore a bit more. Some dependencies are used in many projects, such as @types/node, but it is used in the build phase, and the new version release rate is low enough. It is rarely released as a major version upgrade and follows the node runtime’s version, but minor and patch versions are released regularly. Another example is @aws-sdk\client-dynamodb, while the rate of releases is lower, this impacts the runtime, and once a change is definitively required, the whole organization will put lots of effort into updating. Another important note is how critical updates are rolled out for any package. Does it respond to any security issue or not?.

Enterprise Level Ecosystem

An ecosystem represents the overall leading system to any level of revenue. That system runs on top of many components that collaborate to achieve the final asset, and all those components communicate via defined interfaces called contracts. Not talking about contracts here, but the released assets that are settled on two sides of any single line.

At first glance, those boxes are some autonomous services communicating together, but what a service means to every engineer counts a lot. Some consider a service a dynamic component with a database residing on a separate isolated tier with some business logic, and some others consider a service can have different levels of granularity and flexibility. There are many options. It might be only some complex business logic without any database, protocol, or memory, such as a consumed library at runtime to help extract some metadata from a web URL path, or it can be a container running as a gateway proxy to some partner services.

Looking at a single box and imagining the potential dependencies it may have can be a fun part of the game. The following figure tries to explode a single box to as many pieces as possible.

The above list is not an exclusive list of all dependencies but ones I suffered in experience with. Let’s explore the challenges each one brings

Build time:

• Execution Container: These are the application Dockerfiles built from base images such as public.ecr.aws/docker/library/node:22.13.1-slim. The base images for runtime are less evolved than base images such as public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0, which evolve more frequently.

• CDK: The CDKs are used at development time and evolve frequently as they encapsulate various services and infrastructure. However, they also release versions if any security issue is reported.

• Delivery Dependencies: They evolve less frequently (Such as GitHub Actions) but can be hard to track and keep up to date as they are often declarative and out of sight.

• Compiler & Bundler: In reality, bundlers and compilers are often once configured and keep working for years, but the rate of Bundler releases is high enough that it can never be tracked to define the real upgrade necessity.

• Test Runners and Linters: Are always some packages/libraries that are at a normal change rate; they never impact the real environment. However, it is nice to keep them updated.

• L3 constructs: These are custom and internal CDK constructs developed internally, and they might have a higher or lower release rate based on the problem they solve. The tricky part is when they rely on official CDKs, so not keeping them updated gives some dependency version conflicts.

Run time:

• Execution Runtime: The release of new runtimes is not as often as other options, but EOL must be tracked, and necessary action and upgrades must be done.

• Libraries: Keeping them up-to-date in the JS ecosystem is a must. The JS ecosystem is big enough and often driven by the community, which means more people with different knowledge and culture can contribute, and hidden issues can be produced in terms of functionality or security. But the interesting point is that the resolution of those issues is often fast enough to cover the produced issue.

• SDKs: They provide some simple-to-use interfaces to abstract the real complexity. These are similar to libraries but are often maintained in a structured way and under an enterprise ownership. Often, the rate of releases is low.

• Frameworks: It is hard to name Frameworks as runtime dependencies as they are transpired to JS, however, they can impact the runtime environment.

• Database Engines: This category of engines is the hardest part per experience, as often the upgrades need a full regression testing. Some challenging Engines are PG-SQL, My-SQL, and OpenSearch. The rate of major releases is often low, such as once or twice yearly. The minor and patches are more frequently released. There are options to auto upgrade patch or minors while using managed services, but some instability can be experienced based on the engine and other factors.

GitHub Dependabot

One of the best tools to simplify the process and keep dependencies up to date is Dependabot. It is a free tool maintained by GitHub. It covers a variety of package managers and bundlers. For dependencies, it covers most known such as npm, pnpm, and yarn but also docker.

The configuration gives enough flexibility to manage a variety of scenarios such as single or mono-repo, private or public dependencies, development or production dependencies , etc.

Npm Dependencies

To manage npm dependencies the dependabot looks at repository and detect the package manager by presence of package-lock.json and looks at any single package version on registry to find if a new version available. Following example shows a simple dependabot config.

version: 2

updates:
  - package-ecosystem: 'npm'
    directory: '/'
    schedule:
      interval: 'weekly'
    commit-message:
      prefix: "chore"
      include: "scope"

Mono Repo dependencies

Dependabot will determine the pnpm process if finds the pnpm-lock.yaml , the package-ecosystem for pnpm has the same as npm , both must have the npm value. By default using the above config it will run into all modules and will raise pullrequests for upgrading packages. But if any grouping is configured those are applied correctly to root level package.json but all modules will have single PRs per dependency which is not ideal.

version: 2

updates:
  - package-ecosystem: 'npm'
    directories:
      - 'packages/**'
      - '/'
    schedule:
      interval: 'weekly'
    commit-message:
      prefix: "chore"
      include: "scope"

Grouping Dependencies

By default Dependabot creates a Pull Request per package that can become cumbersome and hard to manage and need to deal with conflicts , etc. Grouping allows to raise single Pull Request for a configured set of dependencies as shown below.

version: 2

updates:
  - package-ecosystem: 'npm'
    directories:
      - 'packages/**'
      - '/'
    schedule:
      interval: 'weekly'
    groups:
      dependencies:
        update-types:
          - 'patch'
          - 'minor'
          - 'major'
    commit-message:
      prefix: "chore"
      include: "scope"

Dev and Prod Dependencies

To separate the build and run dependencies, you can give those dependencies different grouping and have separate PR per group, this allows to better take actions and test based on the group PR.

version: 2

updates:
  - package-ecosystem: 'npm'
    directories:
      - 'packages/**'
      - '/'
    schedule:
      interval: 'weekly'
    groups:
      prod-dependencies:
        dependency-type: 'production'
      dev-dependencies:
        dependency-type: 'development'
    commit-message:
      prefix: "chore"
      include: "scope"

The above config group development ad production dependencies in different processes and raise separate PRs for each group.

Isolating Major, Minor and Patch versions

It is a good practice to separate the production dependencies ( those impacting runtime ) from development ones, however, in each category the Major versions are the most fearful and separating them from the minor and path version updates will save the time , energy and mental health.

version: 2

updates:
  - package-ecosystem: 'npm'
    directories:
      - 'packages/**'
      - '/'
    schedule:
      interval: 'weekly'
    groups:
      prod-dependencies:
        dependency-type: 'production'
        update-types:
          - "minor"
          - "patch"
      dev-dependencies:
        dependency-type: 'development'
        update-types:
          - "minor"
          - "patch"
    ignore:
      - dependency-name: "*"
        update-types: [ "version-update:semver-major" ]
    commit-message:
      prefix: "chore"
      include: "scope"

  - package-ecosystem: "npm"
    directories:
      - 'packages/**'
      - '/'
    target-branch: "prod"
    schedule:
      interval: "monthly"
    groups:
      prod-dependencies:
        dependency-type: "production"
        update-types:
          - "major"
      dev-dependencies:
        dependency-type: "development"
        update-types:
          - "major"
    ignore:
      - dependency-name: "*"
        update-types: [ 
          "version-update:semver-minor",
          "version-update:semver-patch"
        ]
    commit-message:
      prefix: "chore"
      include: "scope"

The above config ignores all semver major and only looks for minor and patch updates.

Docker Dependencies

Dependabot upgrade Dockerfiles by looking at stages ( FROM ) and apply updates as configured. In following example the major versions are excluded imperatively as the example relies on nodejs base image, but, a Major config ( excluding minor and patch ) can be added if the base image Major releases are desired.

version: 2

updates:
  - package-ecosystem: "docker"
    directories:
      - 'packages/**'
    schedule:
      interval: "weekly"
    ignore:
      - dependency-name: "*"
        update-types: [ "version-update:semver-major" ]
    commit-message:
      prefix: "chore"
      include: "scope"

Private and Public Registries

By default the public registers are verified such as https://registery.npmjs.com/ , but private registers can be considered by adding them in dependabot registries section. When using pnpm adding a .npmrc file helps to eliminates the dependabot confusions and looking at both public and private dependencies and applying the fallback pattern if package is not found.

version: 2

registries:
  npm-github:
    type: npm-registry
    url: https://npm.pkg.github.com
    token: ${{secrets.MY_GITHUB_TOKEN}}

updates:
  - package-ecosystem: 'npm'
    directories:
      - 'packages/**'
      - '/'
    schedule:
      interval: 'weekly'
    registries:
      - npm-github
    ignore:
      - dependency-name: "*"
        update-types: [ "version-update:semver-major" ]
    groups:
      prod-dependencies:
        dependency-type: 'production'
        update-types:
          - "minor"
          - "patch"
      dev-dependencies:
        dependency-type: 'development'
        update-types:
          - "minor"
          - "patch"
    commit-message:
      prefix: "chore"
      include: "scope"

The .npmrc file must be like the following example

registry=https://registry.npmjs.org
@xaaxaax:registry=https://npm.pkg.github.com

This helps to lead the Dependabot to recognize the scoped packages ( starting with @xaaxaax ) from GitHub packages and others from public registry.

Other Docker image registries

In scenarios when the base images are from other registries such as public.ecr.aws , this can be configured as a registry. Dependabot force having a username and password for docker registeries but some of public registries dont need that, however providing fake values for these two parameters is the solution.

version: 2

registries:
  ecr-publichub:
    type: "docker-registry"
    url: "https://gallery.ecr.aws"
    username: "fakeit"
    password: "fakeit"

updates:
  - package-ecosystem: "docker"
    directories:
      - 'packages/**'
    schedule:
      interval: "weekly"
    registries:
      - ecr-publichub
    ignore:
      - dependency-name: "*"
        update-types: [ "version-update:semver-major" ]
    commit-message:
      prefix: "chore"
      include: "scope"

Automating Pull Request Actions

If the project has enough barriers to validate the viability of dependencies’ upgrades some last parts of the process can be automated, such as approving, and merging pull requests. This can be achieved using the GitHub workflows, The process looks at pull request metadata, validates some details, approves, and merges the Pr.

name: Dependabot Automated Pull Requests
on: 
  pull_request:
    types: [opened, reopened]

permissions:
  pull-requests: write
  contents: write
  issues: write
  repository-projects: write

env:
  DEPS_SCOPE: 'production'
  MAJOR_UPDATE: 'false'

jobs:
  dependabot:
    runs-on: ubuntu-latest
    if: github.event.pull_request.user.login == 'dependabot[bot]'
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Install pnpm
        uses: pnpm/action-setup@v4
        with:
          version: latest
          run_install: false

      - name: Install dependencies
        run: pnpm install --no-frozen-lockfile

      - name: Build 
        run: pnpm run build:all

      - name: Synth
        run: pnpm run synth:all

      - name: Test
        run: pnpm run test:all

      - name: Dependabot metadata
        id: metadata
        uses: dependabot/fetch-metadata@v2
        with:
          github-token: "${{ secrets.GITHUB_TOKEN }}"

      - name: Define Dependencies scope
        run: |
          echo "DEPS_SCOPE=${{ steps.metadata.outputs.dependency-type == 'direct:development' && 'development' || 'production' }}" >> $GITHUB_ENV
          echo "MAJOR_UPDATE=${{ steps.metadata.outputs.update-type == 'version-update:semver-major' && 'true' || 'false' }}" >> $GITHUB_ENV

      - name: Create ${{ env.DEPS_SCOPE }} label
        continue-on-error: true
        run: gh label create ${{ env.DEPS_SCOPE }}
        env:
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

      - name: Add a label for all ${{ env.DEPS_SCOPE }} dependencies
        continue-on-error: true
        run: gh pr edit "$PR_URL" --add-label ${{ env.DEPS_SCOPE }}
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

      - name: Approve a PR
        run: gh pr review --approve "$PR_URL"
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

      - name: Enable auto-merge for Dependabot PRs
        if: ${{ env.MAJOR_UPDATE != 'true' }}
        run: gh pr merge --auto --rebase --delete-branch "$PR_URL"
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

The above workflow will be triggered each time a Pull Request is opened or closed by dependabot[bot] user. This is the Dependabot default user. The Pull Requests are build and tested and in case of not a major version bump , they are automatically merged.

Conclusion

While software development benefits from internally shared or community-driven dependencies, they can become an obstacle when an upgrade is required. The decision is based on many factors that lead to consider a dependency upgrade necessary or not.

While the factors are important, they often make things more complex than upgrading the dependencies frequently. Dependency upgrades just need to be categorized and automated as much as possible. Dependency upgrades can force some software changes or not; whatever the case, small changes are easier than looking at the whole implementation, so keeping track of versions and upgrading gives more chance to keep the minimum of effort and mental health.

This article explores two Serverless computing services and offers a straightforward method to centralize logging for Serverless applications:

• Cloudwatch Subscription Filters

• AWS Lambda Zip package with Extensions

• AWS Lambda Custom Image with Extensions

• AWS Lambda Web Adapter Image with Extensions

• AWS Fargate with sidecar

About Provided Source Code

The whole examples can be found in source code GitHub repository

The source code is designed as mono-repo using NX, and pnpm. The core package is a private lib that shares some central helpers with other modules. observability-core module is the prerequisite for all other modules and provides:

• Lambda Extension Layer

• ECR Repository

• Base Container Image with Extension

• Kinesis data Stream

• IAM managed policy.

The dependencies are configured via nx.json file in root of the repository for the cdk and build targets. This will force the build and deployment of prerequisite module before the other modules.

"targetDefaults": {
    "cdk": {
      "dependsOn": [
        {
          "projects": "@xaaxaax/observability-core",
          "target": "cdk",
          "params": "forward",
          "required": [ "projects", "target" ]
        }
      ]
    },
    "build": {
      "dependsOn": [
        {
          "projects": "@xaaxaax/observability-core",
          "target": "build",
          "params": "forward",
          "required": [ "projects", "target" ]
        }
      ]
    }
  },

The targets are defined in the root package.json file scripts section as below:

{
  ...
  "scripts": {
     "nx:build:all": "nx run-many --target=build --output-style static --skip-nx-cache",
     "nx:cdk:all": "nx run-many --target=cdk --output-style static --skip-nx-cache --require-approval never",
  },
  ...
}

For simplicity the created functions are configured with function url and can be triggered easily. The only hint is a function gets triggered two times when triggering from browser as an invocation will be for favicon.ico.

Cloudwatch Subscription Filters

AWS services like Lambda and Fargate have native integration with CloudWatch, but for critical workloads, the cost of log ingestion can become prohibitive. Depending on usage needs, CloudWatch logs can be utilized selectively, with different approaches available.

When using CloudWatch, Subscription Filters offer a way to forward logs to various destinations, including OpenSearch, Kinesis Data Streams, Amazon Data Firehose, or AWS Lambda.

In this section, CloudWatch Subscription Filters are used to stream logs to an Amazon Kinesis Data Stream for further processing and analysis.

The following code snippet showcases how to implement this log forwarding mechanism using AWS CDK:

    const logGropup = new LogGroup(this, 'LogGroup', {
      logGroupName: `/aws/lambda/${lambdaWithCloudwatch.functionName}`,
      retention: RetentionDays.ONE_DAY,
      removalPolicy: RemovalPolicy.DESTROY,
    });

    const logsDeliveryRole = new Role(this, `LogsDeliveryRole`, { 
      assumedBy: new ServicePrincipal('logs.amazonaws.com'),
    });

    logGropup.addSubscriptionFilter('SubscriptionFilter', {
      destination: new KinesisDestination(LogStream,{
        role: logsDeliveryRole
      }),
      filterPattern: {
        logPatternString: ' ', // this configure all logs to be filtered
      }
    })

    LogStream.grantWrite(logsDeliveryRole);

Invoking the lambda function will result sending log records in kinesis via cloudwatch subscription filters as shown is the following figure.

Lambda Telemetry Api

Lambda offers a Telemetry API, which will be excellent choice to capture function log records without using the cloudwatch logs. The logs received through the Telemetry API follow a straightforward format, as shown below.

{
    "time": "2025-01-30T00:00:00.000Z", 
    "type": "function", // function, extension, platform
    "record": {
       "timestamp": "2025-01-30T00:00:09.429Z",
       "level": "INFO",
       "requestId": "79b4f56e-95b1-4643-9700-2807f4e68189",
       "message": "Log Message HERE"
    }
}

If The lambda LogFormat is TEXT the received format will be as the following snippet.

{
    "time": "2025-01-30T00:00:00.000Z", 
    // function, extension, platform
    "type": "function", 
    //  Timestamp \t RequestId \t Type \t Message
    "record": "2025-01-30T00:00:09.429Z 79b4f56e-95b1-4643-9700-2807f4e68189 [INFO] Log Message HERE" 
    }
}

Lambda Extensions

For high-throughput applications, relying on CloudWatch Logs can lead to substantial costs. One way to mitigate this is by denying CloudWatch Logs permissions in the Lambda execution role. This prevents the Lambda service from sending logs to CloudWatch, and the consequent will be preventing the use of "Subscription Filters."

However, Lambda provides a Telemetry API that captures all logs, even when CloudWatch logging is disabled. By using the Extension API, you can subscribe to the Telemetry API and register for specific log categories, such as platform, function, or extension logs.

The Source code repository provides the extension module here, that will be used for Lambda Zip package, Custom Image and Web Adapter image sections

The execution of extensions, whether as a standard ZIP package or a custom image, is managed by the Lambda runtime. The Lambda service scans the /opt/extensions directory and automatically executes any extensions found in that location.

For ZIP package deployments, this attachment occurs during the Lambda initialization phase, where the extensions path is constructed by aggregating all attached layers. However, for custom images, this structure must be manually set up during the container image build process.

This project generates two final assets from the same extension source, along with other previously mentioned resources. The extension itself is built using esbuild and bundled as JavaScript, with a post-build script handling the folder structure setup.

The provided final assets are:

• A Lambda Layer

• A ECR Base Container Image

Lambda Layer

For the layer the build process do all necessary steps. The only required step is to use CDK for creating the layer. The following snippet demonstrates the way to create a layer using CDK.

   const extension = new LayerVersion(this, 'kinesis-telemetry-api-extension', {
      layerVersionName: `${props?.extensionName}`,
      code: Code.fromAsset(resolve(process.cwd(), `build`)),
      compatibleArchitectures: [
        Architecture.X86_64,
        Architecture.ARM_64
      ],
      compatibleRuntimes: [
        Runtime.NODEJS_20_X,
        Runtime.NODEJS_22_X,
      ],
      description: props?.extensionName
    });

    // Exporting the Layer Arn to parameter store
    new StringParameter(this, `LambdaExtensionArnParam`, {
      parameterName: `/${props.contextVariables.stage}/${props.contextVariables.context}/telemetry/kinesis/extension/arn`,
      stringValue: extension.layerVersionArn,
    });

The LayerVersion resource point to the build directory generated by build script, The underlying build folder structure is as below

- build
  - extensions
    - kinesis-telemetry-extension
  - kinesis-telemetry-extension
    - index.js

The kinesis-telemetry-extension file under extensions folder is an executable file that will be the entry point for lambda service to detect extension and execute it. The file name must be equal to the extension directory name for this example executable.

#!/bin/bash
set -euo pipefail

OWN_FILENAME="$(basename $0)"
LAMBDA_EXTENSION_NAME="$OWN_FILENAME"

echo "[extension:bash] launching ${LAMBDA_EXTENSION_NAME}"
exec "/opt/${LAMBDA_EXTENSION_NAME}/index.js"

Base Container Image

The base custom image with extension included is created using a Dockerfile. The Dockerfile simply use the built asset ( build folder ) contents and move it to the /opt/ directory in built image.

FROM node:22.13.1-slim

COPY build /opt/

WORKDIR /opt/extensions

The Image will be built and pushed to the ECR repository created via cdk which is a prerequisite for pushing the image to ECR. This is done using a post script.

{
  "name": "@xaaxaax/observability-core",
  ...
  "scripts": {
    "build:docker": "docker buildx build --platform linux/arm64 --no-cache -t $ECR_REPOSITORY:latest .",
    "postbuild:docker": "pnpm run build:docker:login && pnpm run build:docker:tag && pnpm run build:docker:push",
    "build:docker:login": "aws ecr get-login-password --region $REGION --profile admin@dev | docker login --username AWS --password-stdin $ECR_URI",
    "build:docker:tag": "docker tag $ECR_REPOSITORY:latest $ECR_URI/$ECR_REPOSITORY:latest",
    "build:docker:push": "docker push $ECR_URI/$ECR_REPOSITORY:latest",
    "cdk": "cdk --profile admin@dev --app 'tsx ./cdk/bin/app.ts' -c env=dev",
    "postcdk": "cross-env REGION=eu-west-1 ECR_URI=904233108557.dkr.ecr.eu-west-1.amazonaws.com ECR_REPOSITORY=lambda-telemetry-image pnpm run build:docker"
  },
  ...
}

Lambda Zip package with Extensions

Dealing with Zip lambda package is the simplest option by attaching the layer to the lambda function. but also giving the required permissions to the associated role for underlying infrastructure that the extension need to interact with that is kinesis data stream in the provided example.

The following CDK represents how the extension layer can be attached to the function and what are the required permissions.

    const extensionArn = StringParameter.fromStringParameterName(
      this, 'extensionId', `/${props.contextVariables.stage}/logs-collector-lambda-extension/telemetry/kinesis/extension/arn`).stringValue;

    const managedPolicyArn = StringParameter.fromStringParameterName(
      this, 'policyName', `/${props.contextVariables.stage}/logs-collector-lambda-extension/telemetry/kinesis/runtime/policy/arn`).stringValue;

    const functionRole = new Role(this, 'LambdaFunctionRole', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole'),
        ManagedPolicy.fromManagedPolicyArn(this, 'managed-policy', managedPolicyArn)
      ]
    });

    const lambdaFunction = new NodejsFunction(this, 'LambdaZipFunction', {
      entry: resolve(process.cwd(), 'src/handler.ts'),
      ...
      bundling: {
        ...
      },
      layers: [ 
        LayerVersion.fromLayerVersionArn(this, 'ExtensionArn', extensionArn) 
      ],
    });

Lambda Custom Image With Extensions

The example custom images build a docker image base function from a Dockerfile based on provided base image with extension included.

The Dockerfile is based on both extension image and lambda nodejs 22 image provided by aws. The interesting part of AWS provided image is that it can be run locally and can be invoked for example using a curl command.

FROM 904233108557.dkr.ecr.eu-west-1.amazonaws.com/lambda-telemetry-image:latest AS extensions
FROM public.ecr.aws/lambda/nodejs:22

WORKDIR ${LAMBDA_TASK_ROOT}

COPY dist/* ./
COPY --from=extensions ./opt/ /opt/

CMD ["index.handler"]

The built asset will be copied to the /var/task path that can be accessed using LAMBDA_TASK_ROOT env variable and finally the CMD layer point to the handler inside index.js file.

In the example the base image imageUri is hardcoded in Dockerfile but it can be parametrized using parameter store and passing as docker build ARGs,

Lambda Web Adapter Image With Extensions

While Lambda Adapter provides a custom runtime, it brings some particularity to the way the Dockerfile shall be used. As per LWA documentation and examples, the base image used is public.ecr.aws/docker/library/node:22.9.0-slim and not public.ecr.aws/lambda/nodejs:22 which means the lambda api interface no more can be used for local invocation, e.g using curl.

The example Dockerfile uses multiples stages

FROM 904233108557.dkr.ecr.eu-west-1.amazonaws.com/lambda-telemetry-image:latest AS extensions
FROM public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0-aarch64 AS webadapter
FROM public.ecr.aws/docker/library/node:22.9.0-slim

WORKDIR ${LAMBDA_TASK_ROOT}

COPY dist/* ./
COPY --from=extensions ./opt/ /opt/
COPY --from=webadapter /lambda-adapter /opt/extensions/lambda-adapter

CMD ["node", "index.js"]

The image will use the extension base image alongside the lambda web adapter base image and use the contents of ./opt folder related to each extension. Also it uses the built asset of function code ( here a http server on default port of LWA 8080 ).

The particularity behavior for LWA in our example is the way function logs are received in extension. Only the function logs are under this unfortunate behavior and are not formatted as a valid json object but are treated as text event while the LambdaLogFormat is set as JSON. so the above official format is not working and element.record.message will result an undefined value. The following shows how the record is received which is a representation of Javascript object surrounded by double quotes.

{
   "time":"2025-01-29T21:24:33@.665Z",
   "type":"function",
   "record":"{ name: 'omid' }"
}

To resolve the problem, the extension is adopted to look at element.record if the element.record.message is value. But event the change is not sufficient as the received record is a double quoted JS object. So the log data must be formatted using JSON.stringify().

console.log(JSON.stringify( logObject ));

Fargate with Firelens sidecar

Fargate, as a serverless solution for running containers on demand, supports both short-lived and long-running tasks. Regardless of the use case, enabling containers to communicate and complement each other’s capabilities is essential for building scalable and efficient architectures.

In line with the examples in this article, this section demonstrates how to forward container logs to a central Kinesis Data Stream. To achieve this, the Fargate task can include a sidecar container responsible for collecting logs and forwarding them to the data stream.

The application container provides a Dockerfile as below

FROM --platform=linux/arm64 public.ecr.aws/docker/library/node:22-slim

COPY dist/* ./

CMD ["node", "index.js"]

But as mentioned above, there will be another Dockerfile for the log forwarder container that uses the FluentBit image provided by aws.

FROM amazon/aws-for-fluent-bit:latest

ADD container.conf /container.conf
ADD parsers.conf /parsers.conf

As shown in the Dockerfile there are two configuration files for Parsing and Container specific configurations such as Filtering , etc. the content of both files are as below

// parsers.conf file
[PARSER]
    Name    log_json
    Format  json

// container.conf file
[SERVICE]
    Parsers_File    parsers.conf

[FILTER]
    Name            parser
    Match           *
    Key_Name        log
    Parser          log_json

[FILTER]
    Name            grep
    Match           *
    Regex           app_name fargate-example-app

Let see how these are deployed and resources are created. AWS Cdk provides the L2 constructs that can be used to simplify the infra as code steps. The example uses the FargateTaskDefinition and FargateService constructs.

   const jobDefinition = new FargateTaskDefinition(this, 'JobDefinition', {
      cpu: 256,
      memoryLimitMiB: 512,
      runtimePlatform: {
        cpuArchitecture: CpuArchitecture.ARM64,
        operatingSystemFamily: OperatingSystemFamily.LINUX,
      },
      taskRole: jobTaskRole,
      executionRole: jobTaskExecutionRole,
    });

After creating the base TaskDefinition, the app container and firelens router will be added as below

    jobDefinition.addContainer('Container', {
      image: ContainerImage.fromAsset(join(process.cwd())),
      logging: LogDrivers.firelens({
        options: {
          Name: 'kinesis_streams',
          region,
          stream: props.streamName,
        },
      }),
    });

    jobDefinition.addFirelensLogRouter('LoggingContainer', {
      image: ContainerImage.fromAsset(join(process.cwd(), 'fluent-bit')),
      logging: LogDrivers.awsLogs({
        streamPrefix: 'logging',
        logGroup: new LogGroup(this, 'FireLensLogGroup', {
          logGroupName: `/ecs/${props.contextVariables.context}`,
          retention: RetentionDays.ONE_DAY,
          removalPolicy: RemovalPolicy.DESTROY,
        }),
      }),
      environment: { FLB_LOG_LEVEL: 'info' },
      firelensConfig: {
        type: FirelensLogRouterType.FLUENTBIT,
        options: {
          configFileType: FirelensConfigFileType.FILE,
          configFileValue: '/container.conf',
        },
      },
    });

A service shall be created to encapsulate a task consisting of two side by side containers. The is simple and straightforward.

   const service = new FargateService(this, 'Service', {
      cluster,
      capacityProviderStrategies: capacityStrategy,
      desiredCount: 1,
      platformVersion: FargatePlatformVersion.VERSION1_4,
      propagateTags: PropagatedTagSource.TASK_DEFINITION,
      taskDefinition: jobDefinition,
      assignPublicIp: true,
      vpcSubnets: { subnets: vpc.publicSubnets },
      securityGroups: [ taskSecurityGroup ],
    });

For simplicity, the example allow assigning a public ip address to the task and put the service in public subnets, the reason that this is required is the FargatePlatformVersion.VERSION1_4 is under managed awsvpc and this is the simplest way to let fargate pull images from ECR. This is not recommended for production cases.

The Task role must have the kinesis PutRecords action permissions. Here the observability-core stack provides a managed policy that can be attached to the role.

  const managedPolicyArn = StringParameter.fromStringParameterName(
      this, 
      'ObservabilityManagedPolicy', `/${props.contextVariables.stage}/logs-collector-observability-core/telemetry/kinesis/runtime/policy/arn`).stringValue;

  const jobTaskRole = new Role(this, 'JobTaskRole', {
      assumedBy: new ServicePrincipal('ecs-tasks.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromManagedPolicyArn(this, 'TaskRoleManagedPolicy', managedPolicyArn),
      ],
  });

  const jobTaskExecutionRole = new Role(this, 'JobTaskExecutionRole', {
      assumedBy: new ServicePrincipal('ecs-tasks.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonECSTaskExecutionRolePolicy'),
      ],
  });

After deploying attached IP to the created ENI can be used over `http` protocol. The logs will be sent to the Kinesis DS as shown in following screenshot.

The Firelens has the same problem as LWA mentioned before, The log metadata object must be stringyfied. If the JS object is directly logged the same behavior will be provided as Web Adapter.

Conclusion

While Serverless offers a wide range of managed services that scale per needs, It is important to forget the shared responsibility that forces engineering teams to be engaged on their side. As part of software development the use of Processors capacity and Memory is the part under engineering teams ownership. This is not far from traditional software principals but is somehow forgotten by the fascinating nature of managed services.

Using Lambda extensions, multi container, or background processes is the way to apply processing isolation and achieve more trustable software which is running as a foreground process.

The article focuses on log aggregation to represent how decouple the critical processing from non critical ones via isolation and provides some examples to showcase the implementation in different scenarios.

• Complications in billing management

• Inability to share credits

• Challenges with quota management

• Orphaned resources

• A $1 charge for each account created

• The need to create temporary emails and manage contact information

AWS Organization

AWS Organizations is a service from AWS designed to streamline the centralized management of accounts. It offers features such as account provisioning, centralized billing, access management, policy management, and enforcement of standards.

Organizational Unit

An Organizational Unit (OU) is a grouping of accounts that allows for the distribution of accounts based on specific contexts or needs. For example, you might create one OU for workload accounts (development, testing, production) and another for security or networking purposes.

This separation enables additional automation for the member accounts, such as bootstrapping or deploying infrastructure tailored to specific contexts.

AWS CDK and challenges

Onboarding a new organization using CDK initially appeared to be as straightforward as creating any other piece of infrastructure. However, as I began implementation, I encountered several gaps due to missing features in CloudFormation and the Dedicated Service API.

• The AWS Organizations service requires trusted access, which is only available through the Organizations Service API.

• Activating SSO with IAM Identity Center can only be done through a manual process in the console.

• Creating accounts necessitates unique email addresses, which makes it frustrating to set up multiple Gmail or whatever accounts.

• While setting up SES with Route 53 was relatively easy, the documentation was confusing and misleading.

Source Code

Phase 1: Configuration

Since setting up an organization involves more details than an application, it’s important to consider the relevant configuration sections. The following ConfigType outlines the structure of the stack configuration.

export type Config = {
  contextVariables: ContextVariables;
  dns: DNSConfig;
  org: OrgConfg;
  sso:  SSOConfig;
}

DNS Config

DNSConfig should account for either importing an external HostedZone or creating a new one. By separating the types, we can enhance type safety and maintain better control within the stack.

type ExternalZone = { isExternal: true; hostedZoneId: string; }
type InternalZone = { isExternal: false; domainName: string; }
export type DNSConfig = (ExternalZone | InternalZone) & {
  mailExchangeDomainName: string;
};

Org Config

OrgConfig includes configuration attributes related to account creation and bootstrapping, as well as the activation of trusted services and parameter sharing.

export type OrgConfg = {
  memebers: {
    bootstrap?: boolean;
    accounts: {
      accountName: string;
    }[];
  };
  trustedAWSServices?: string[];
  crossAccountParametersSharing?: boolean;
};

SSO Config

SSOConfig features two distinct types: Ready and NotReady. This distinction allows for the creation of SSO and identity store groups, along with permission sets, after implementing the Click-Ops solution in the management account.

export type SSONotReady = { isReadyToDeploy: false; }
export type SSOReady = { 
  isReadyToDeploy: true; 
  ssoInstanceArn: string;
  identityStoreId: string; 
}
export type SSOConfig = SSONotReady | SSOReady;

HostedZone and Emailing

The management account can either own an existing HostedZone or create a new one for a domain name, and also set up the necessary components to receive emails. This is crucial for having unique email addresses for each account while still allowing them to be forwarded to a personal account.

The process for importing or creating the hosted zone is outlined below:

    let hostedZone: IHostedZone;
    if(dnsConfig.isExternal)
      hostedZone = HostedZone.fromHostedZoneId(this, 'HostedZone', dnsConfig.hostedZoneId);
    else {
      hostedZone = new HostedZone(this, 'HostedZone', { zoneName: dnsConfig.domainName, comment: 'Managed by CDK' });

      new StringParameter(this, 'HostedZoneId', {
        parameterName: `/${this.ENV}/${this.CONTEXT}/${dnsConfig.domainName}/hostedzone/id`,
        stringValue: hostedZone.hostedZoneId,
      })
    }

To enable email reception through the HostedZone, an MXRecord must be added. Note that the mailExchangeDomainName configuration can be either the same as the domain name (e.g., example.com) or a subdomain (e.g., mail.example.com).

  new MxRecord(this, 'MXRecord', {
      zone: hostedZone,
      values: [{
        hostName: `inbound-smtp.${this.REGION}.amazonaws.com`,
        priority: 10,
      }],
      recordName: dnsConfig.mailExchangeDomainName,
      deleteExisting: true,
    });

To receive emails using SES, you need to extend its capabilities through ReceiptRuleSet actions, as SES does not natively provide a mailbox option. To integrate SES, you must create an EmailIdentity, which should be of the Domain type, accomplished by providing the HostedZone.

    new EmailIdentity(this, 'Identity', {
      identity: Identity.publicHostedZone(HOSTED_ZONE),
    });

The ReceiveRuleSet allows you to establish rules for receiving emails and define actions to be taken. The actions specified in a rule will be executed in sequence. In this example, each incoming email will be stored in an S3 bucket, and a Lambda function will be triggered to process the stored content and forward it to other email servers.

const receiptRuleSet = new ReceiptRuleSet(this, 'MailReceivedRuleSet', {
        dropSpam: false,
        rules: [{
          tlsPolicy: TlsPolicy.REQUIRE,
          scanEnabled: false,
          enabled: true,
          recipients: [ HOSTED_ZONE.zoneName ],
          actions: [
            new S3({ bucket: deliveryBucket }),
            new Lambda({ function: receivedMailLambda }),
          ],
        }],
    });

Deploying the stack outlined above will create all the necessary resources for a functional domain name with email reception. However, if you attempt to send an email from your address, you will receive a postmaster response indicating that the email sending has failed.

inbound-smtp.eu-west-1.amazonaws.com has generated this error :
Requested action not taken: mailbox unavailable

SES permits only one active ruleset at a time, so when you create a new ruleset, it will not be activated automatically.

To activate the ruleset, you can use AWS CDK custom resources with SDK calls, as shown below.

    const rulesetActivationSDKCall: AwsSdkCall = {
        service: 'SES',
        action: 'setActiveReceiptRuleSet',
        physicalResourceId: PhysicalResourceId.of('SesCustomResource'),
    };

    const setActiveReceiptRuleSetSdkCall: AwsSdkCall = {
      ...rulesetActivationSDKCall,
      parameters: { RuleSetName: receiptRuleSet.receiptRuleSetName }
    };
    const deleteReceiptRuleSetSdkCall: AwsSdkCall = rulesetActivationSDKCall;

    new AwsCustomResource(this, "setActiveReceiptRuleSetCustomResource", {
      onCreate: setActiveReceiptRuleSetSdkCall,
      onUpdate: setActiveReceiptRuleSetSdkCall,
      onDelete: deleteReceiptRuleSetSdkCall,
      logRetention: RetentionDays.ONE_WEEK,
      policy: AwsCustomResourcePolicy.fromStatements([
        new PolicyStatement({
          sid: 'SesCustomResourceSetActiveReceiptRuleSet',
          effect: Effect.ALLOW,
          actions: [
            'ses:SetActiveReceiptRuleSet',
            'ses:DeleteReceiptRuleSet',
          ],
          resources: ['*']
        }),
      ]),
    });

This solution lets to activate the created ruleset and the ses reception now works as expected.

The provided example only trigger a lambda function that logs an ses event but you can implement your email forwarding if needed.

Organization and Accounts

The CDK for AWS Organizations only offers L1 constructs, but I find this approach simple and straightforward. In my opinion, adding L2 constructs might be unnecessary over-engineering, so I’m fine with this CDK decision for now.

To create an Organization and an OU, the only parameters required are the FeatureSet of the Organization, which can be either CONSOLIDATED_BILLING or ALL.

    const orga =new CfnOrganization(this, 'Organization', { featureSet: 'ALL' });

    const orgUnit = new CfnOrganizationalUnit(this, 'OrganitationUnit', {
      name: `workloads${tempSuffix}`,
      parentId: orga.attrRootId
    });

    orgUnit.addDependency(orga);

The following snippets illustrate how to create accounts. In this example repository, the account list is provided through configuration, meaning the accounts parameter will be an array of objects in the format { accountName: string }. The created account IDs will be stored in the parameter store, although this may not be necessary for a personal organization setup.

ACCOUNTS.forEach((account: { accountName: string }) => {
      const awsAccount =new CfnAccount(this, `${account.accountName}Account`, {      
        accountName: `${account.accountName}${tempSuffix}`,
        email: `${account.accountName}${tempSuffix}@${DOMAIN_NAME}`,
        parentIds: [orgUnit.attrId],
      });

      const param = new StringParameter(this, `${account.accountName}AccountIdParam`, {
        stringValue: awsAccount.attrAccountId,
        description: `Account ID for ${awsAccount.accountName}`,
        parameterName: `/${this.ENV}/${this.CONTEXT}/${awsAccount.accountName}/account/id`,
      })
    });

To set up SSO using IAM Identity Center, it's essential to enable AWS Organization trusted access. Unfortunately, there’s no way to activate this feature using CDK or CloudFormation, so we will once again rely on a Custom Resource. In this example, all services required for trusted access are specified through configuration (e.g., sso.amazonaws.com and servicequota.amazonaws.com).

    trustedServices.forEach((service: string) => {

      const identifier = service.replace('.', '');
      const enable: AwsSdkCall = {
        service: 'organizations',
        action: 'enableAWSServiceAccess',
        physicalResourceId: PhysicalResourceId.of(`OrgCustomResource${identifier}`),
        parameters: { ServicePrincipal: service },
      };

      const disable: AwsSdkCall = {
        service: 'organizations',
        action: 'disableAWSServiceAccess',
        physicalResourceId: PhysicalResourceId.of(`OrgCustomResource${identifier}`),
        parameters: { ServicePrincipal: service },
      };

      new AwsCustomResource(this, `AWSServiceAccessActivation${identifier}CustomResource`, {
        onCreate: enable,
        onUpdate: enable,
        onDelete: disable,
        logRetention: RetentionDays.ONE_WEEK,
        policy: AwsCustomResourcePolicy.fromStatements([
          new PolicyStatement({
            sid: 'OrgCustomResourceSetOrgAWSServiceActivation',
            effect: Effect.ALLOW,
            actions: [
              'organizations:enableAWSServiceAccess',
              'organizations:disableAWSServiceAccess',
            ],
            resources: ['*']
          }),
        ]),
      });
    })

SSO Setup

The provided example configuration uses a NotReady state by setting isReadyToDeploy = false, which prevents the CDK deploy from generating the SSO configuration, as the SSO instance has not yet been created. As mentioned earlier, this cannot be accomplished through automation or API calls; the only available API call for CreateInstance works solely for standalone accounts, not for Organization Management accounts.

Before changing the flag to true, you must go to the AWS Console in the management account, navigate to IAM Identity Center, and click the Enable button. After activation, you’ll need the SsoInstanceArn and IdentityStoreId, which should be set in the stack configuration file along with isReadyToDeploy=true.

Once these steps are completed, the CDK deploy will proceed to deploy the stack along with all associated groups and permission sets.

    const group = new CfnGroup(this, id, {
      displayName: `${id}`,
      description: `${id} Group`,
      identityStoreId,
    });

    const permissionSet = new CfnPermissionSet(this, `${id}PermissionSet`, {
      name: `${id}@${ENV}`,
      description: `${id}@${ENV}`,
      instanceArn: ssoInstanceArn,
      managedPolicies: managedPolicies,
      inlinePolicy: undefined,
      sessionDuration: Duration.hours(12).toIsoString(),
    });

    accounts.forEach((account) => {
      new CfnAssignment(this, `${id}Assignment`, {
        instanceArn: ssoInstanceArn,
        permissionSetArn: permissionSet.attrPermissionSetArn,
        principalId: group.attrGroupId,
        principalType: 'GROUP',
        targetId: account,
        targetType: 'AWS_ACCOUNT',
      });
    })

In the CDK snippet above, a group is created in the Identity Store, and a permission set is established for the SSO Instance. This group is assigned to each of the accounts created earlier. The code illustrates the Group Construct, which is utilized as shown below.

    // Org Accounts
    const developmentAccount = StringParameter.fromStringParameterName(this, 'AccountSecurity', `/${this.ENV}/${this.CONTEXT}/security_b/account/id`).stringValue; 

    //Managed Policies
    const adminManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('AdministratorAccess');
    const poweredUserManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('PowerUserAccess');
    const readonlyManagedPolicy = ManagedPolicy.fromAwsManagedPolicyName('ReadOnlyAccess');

    new Group(this, 'Admin', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ adminManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

    new Group(this, 'PowerUser', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ poweredUserManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

    new Group(this, 'Developer', { 
      contextVariables: this.CONTEXT_VARIABLES,
      ssoInstanceArn: SSO_INSTANCE_ARN,
      identityStoreId: IDENTITY_STORE_ID,
      managedPolicies: [ readonlyManagedPolicy.managedPolicyArn ],
      accounts: [ developmentAccount ]
    });

You can now create a user in IAM Identity Center and assign it to one or more groups. Next, navigate to the AWS Access Portal (accessible via the link from the Identity Center dashboard: https://d-123456788.awsapps.com/start), which will prompt you for login credentials.

Once logged in, the application page will allow you to select the appropriate group role and access it with the corresponding permissions.

Account Boostrap

It was an effective way to automate account creation and implement varying levels of security. However, an empty account requires several repetitive tasks before it can be considered usable. The example includes setting up GitHub OIDC and CDK Bootstrap to prepare the member accounts.

The bootstrap can be deactivated through configuration, and this will be verified in the organization stack as shown below.

   if( BOOTSTRAP ) {
      new Bootstrap(this, 'Bootstrap', { 
        contextVariables: props.contextVariables,
        regions: [ this.REGION ],
        organizationUnits: [ orgUnit ],
        types: {
          [BootstrapTypes.CDK]: { FileAssetsBucketKmsKeyId: 'AWS_MANAGED_KEY' },
          [BootstrapTypes.GitHub]: { Owner: gitHubConfig.owner, Repo: '*' }  
        }, 
      })
    }

The bootstrap construct creates a set of CloudFormation StackSets to allow the management account to bootstrap the member accounts, which will be triggered when accounts are created or updated. Unfortunately, the CDK does not offer a straightforward way to use CDK stacks for StackSets. After researching online, I found many solutions to be overly complicated, so I opted to stick with the readily available YAML templates found across the web (even though I probably won’t look at or modify them). I'm fine with this approach.

    export enum BootstrapTypes {
        GitHub = 'oidc-github.yml',
        CDK = 'cdk-bootstrap-template.yml',
     }

    const { contextVariables: { stage: ENV, context: CONTEXT }, types: TYPES } = props;
    const tags =  Stack.of(this).tags.renderTags();

    Object.keys(TYPES).forEach((value: string, index: number) => {
      const typeIdentifier = value.replace('.yml', '').replace(/[^a-zA-Z]/g, '');
      const cfnParams = Object.entries(TYPES[value as unknown as BootstrapTypes])
        .map(([key, value]) => (
          { parameterKey: key, parameterValue: value } as CfnStackSet.ParameterProperty
        )); 

      new CfnStackSet(this, `BootstrapStackSet${typeIdentifier}`, {
        permissionModel: "SERVICE_MANAGED",
        stackSetName: `${CONTEXT}-bootstrap-${typeIdentifier}-${ENV}`,
        description: `Account bootstrap StackSet ${typeIdentifier}`,
        autoDeployment: { enabled: true, retainStacksOnAccountRemoval: false },
        capabilities: ["CAPABILITY_NAMED_IAM"],
        templateBody: readFileSync(join(process.cwd(), `/cdk/lib/orga/bootstrap/${value}`), 'utf8'),
        parameters: cfnParams,
        tags,
        operationPreferences: { failureToleranceCount: 1, maxConcurrentCount: 1 },
        stackInstancesGroup: [{
          regions: props.regions,
          deploymentTargets: {
            organizationalUnitIds: props.organizationUnits.map((ou: { attrId: string }) => ou.attrId), 
          },
        }],
      });
    });

Conslusion

For a long time, I had been trying to set up an organization, but since I couldn't find a working piece of code, I decided to dive into it myself. It was an exciting experience, tackling different challenges and solving them along the way. Kudos to AWS CDK for its flexibility!

Having an organization is a great way to experiment and easily tear things down afterward. When closing accounts, you’ll incur charges for 90 days after they’ve been removed from the organization. However, after this 90-day period, the accounts will be permanently deleted. During this time, you can still recover the account, access it with limited permissions, and perform certain actions.

AWS CDK, as an abstraction layer, offers more flexibility using code by overcoming the difficulties of structured cloud formation templating. However, this allowance of autonomy and flexibility can become an obstacle in the long term. Often, this becomes frustrating when applying some conventions at the enterprise level, when teams must change all stacks to respect some standard or convention. Also, facing changes in AWS services or deprecations are cases that need some extra effort and modification in all stacks which is theoretically not possible or takes a lot of time.

Configuration

When dealing with IaC, configurable software practices must be adopted to help have a central and easy-to-change stack. There are many ways to use configuration such as configuration files, ParameterStore, or CloudFormation Outputs. The choice of configuration depends on the nature of values, lifecycle, and dependencies.

Configuration File

A configuration file can be ideal for some internal and local configuration elements, the following snippet shows an example of a configuration file in Typescript, including some naming and static config values.

import { ContextVariables, EnvVariable } from "@type";

export type Config = {
  contextVariables: ContextVariables;
  org: { accounts: { accountName: string;}[];};
}

const defaultConfig: Config = {
  contextVariables: {
    context: `my-application`,
    stage: 'dev', 
    owner: 'operations',
    usage: 'EPHEMERAL',
  }
}

const getFinalConfig = (config: Partial<Config>): Config => {
  return {
    ...defaultConfig,
    contextVariables: {
      ...defaultConfig.contextVariables,
      ...config.contextVariables,
    },
    ...config
  }
}

export const getConfig = (stage: EnvVariable): Config => {
  switch (stage) {
    case 'test':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables, 
        stage: 'test', usage: 'PRODUCTION' } 
      });
    case 'prod':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables,
        stage: 'prod', usage: 'PRODUCTION' 
      }});
    case 'dev':
      return getFinalConfig({ contextVariables: { 
        ...defaultConfig.contextVariables, 
        stage: 'dev', usage: 'EPHEMERAL' 
      }});
    case 'sandbox':
      return getFinalConfig({ contextVariables: {
         ...defaultConfig.contextVariables,
         stage: 'sandbox', usage: 'POC' 
      }});
    default:
      return getFinalConfig({});
  }
};

The config can be later easily fetched in CDK app entry as below

const app = new cdk.App();
const environment = getEnv(app);
const config = getConfig(environment);

Parameter Store

Parameter Store is advantageous when using external configurations such as Application Load Balancer Listener Arn, VPC Id, etc. It is a good practice to use parameters for central configurations or unpredictable cross-stack ones.

Parameter Store as a solution can lead to complexities if:

• Lack Of Conventional Naming for parameters.

• Consumer stacks spread param fetch in different places.

• Excessive use of Parameters

In the following snippet, the parameters fetch is done in the parent stack as passed to the nested stacks, either all nested stacks don’t need all parameters.

export class AwsGatewayUsingLoadbalancerStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const albSecurityGroupId = StringParameter.fromStringParameterName(this, 'https-listener-securitygroupe-id', '/alb/securityGroupId').stringValue;
    const albHttpsListenerArn = StringParameter.fromStringParameterName(this, 'https-listener-arn',  '/alb/httpsListenerArn').stringValue;

    const securityGroup = SecurityGroup.fromSecurityGroupId(this, 'alb-security-group', albSecurityGroupId);
    const albListener = ApplicationListener.fromApplicationListenerAttributes(this, 'alb-listener', {
      listenerArn: albHttpsListenerArn,
      securityGroup: securityGroup,
    });

    const lambda = new NodejsFunction(this, 'example-lambda', {
      entry: join(process.cwd(), '/src/example.ts'),
      handler: 'handler',
      ...LambdaConfiguration
    });

    const target = new ApplicationTargetGroup(this, 'example-target-group', { targets: [ new LambdaTarget(lambda) ] });

    albListener.addTargetGroups('dosomething-target', {
      priority: 1,
      conditions: [
        ListenerCondition.hostHeaders(['myservice.example.com']),
        ListenerCondition.pathPatterns(['/v1/dosomthing']),
        ListenerCondition.httpRequestMethods([
          HttpMethod.POST,
          HttpMethod.OPTIONS,
        ]),
      ],
      targetGroups: [target],
    });
  }
}

The example fetches the parameters early and passes through when you need it. The advantage of this approach is that the parameters are side by side and in a single place, simplifying future changes and evolutions. another approach is the parameter is fetched once and shared, this means if you need multiple times the same parameter, the Api call to parameter store is done in a single place and only once.

Validation

As part of the Infrastructure as code, companies often apply conventions that help simplify governance, such as Available Stages ( dev, test, staging, prod, sandbox, etc. ) or Tagging ( stage, context, project, application, or domain).

Discovering noncompliant resources is one way of doing so, and this is a correct way of analyzing and finding noncompliant resources, but often, these discovered problems take a long time to fix and cause a lot of frustration.

A better approach is to think as an enabler, making the present and future life easy and trying to simplify the way teams must apply the regulations and conventional aspects.

Using CDK, a validator will be executed soon during Synthesizer execution. The following example validates the allowed stage configurations

export class WorkloadEnvValidator implements IValidation {
  constructor(private readonly variables: ContextVariables) {}

  public validate(): string[] {
    const errors: string[] = [];
    if(!(isEnvValid(this.variables.stage))) {
      errors.push(`Provided Stage value is not a valid environment. Must be one of: ${JSON.stringify(AvailableEnvs)}.`);
    }

    if(
      !['dev', 'sandbox'].includes(this.variables.stage) &&
      this.variables.usage !== 'PRODUCTION'
    ){
      errors.push(`Provided Stage value is not eligible to run ephemeral or experimental stacks.`);
    }

    return errors;
  }
}


// IsEnvValid allows Dev, Test, Prod and sandbox
//export const isEnvValid = (env: EnvVariable): env is EnvVariable =>
//  IsDevelopmentEnv(env) || 
//  IsTestingEnv(env) ||
//  IsProductionEnv(env) ||
//  IsSandboxEnv(env);

Running the synth using an unknown stage name will result in the following messages.

Aspects

We can use aspects to apply conventions and compliance-related tasks in an automated way, such as Tagging resources, Conventional Naming, etc. The following example shows a simple way of applying parameter naming.

export class ApplyParameterStoreNamingPolicyAspect implements IAspect {
  constructor(private readonly variables: ContextVariables) { }
  public visit(node: IConstruct): void {
    if (node instanceof CfnParameter) {
      const inspector = new TreeInspector();
      node.inspect(inspector);

      const name = inspector.attributes['aws:cdk:cloudformation:props']['name'].toString();
      if(name.startsWith(`/${this.variables.stage}/${this.variables.context}/`)) return;
        const cleanedName = name
            .replace(`${this.variables.stage}`, '')
            .replace(`${this.variables.context}`, '')
            .replace('//', '');
      node.addPropertyOverride('Name', `/${this.variables.stage}/${this.variables.context}${name}` );
      Annotations.of(node).addWarningV2(`${name}`, `Parameter Name should start with /${this.variables.stage}/${this.variables.context}, A managed fix is applied by renaming the parameter name but this can have consequences per usage, please apply the correct naming convention` );
    }
  }

This sample Aspect looks at parameter resources, and transforms the parameter name, and shows a warning in the terminal.

Centralization

There are cases where providing a framework or extensions can be useful, letting the teams use them when necessary, but applying compliance brings a lot of small pieces that can become error-prone to ask teams to apply explicitly or per need.

The best way of achieving the goal but also keeping the rate of change and effort as minimal as possible will be giving abstractions that help teams to achieve the goal with simplicity, this can be achieved with constructs but again creating constructs and applying them everywhere and in all services is not the best choice.

However, providing an abstraction of type Stack can be simple enough to propagate at the enterprise level. The example below shows how can be achieved.

import { ArnFormat, Aspects, Duration, NestedStack, NestedStackProps, Stack, StackProps, Tag } from "aws-cdk-lib";
import { Construct } from "constructs";
import { ApplyDestroyPolicyAspect, ApplyParameterStoreNamingPolicyAspect, ApplyTagsAspect } from "./aspects";
import { ContextVariablesValidator, WorkloadEnvValidator } from "./validators";
import { ContextVariables } from "../types/Context";
import { Rule, Schedule } from "aws-cdk-lib/aws-events";
import { LambdaFunction } from "aws-cdk-lib/aws-events-targets";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { Runtime } from "aws-cdk-lib/aws-lambda";
import { join } from "path";
import { PolicyDocument, PolicyStatement, Role, ServicePrincipal } from "aws-cdk-lib/aws-iam";
export type EnforcedStackProps = StackProps & NestedStackProps & {
  contextVariables: ContextVariables;
}

export class EnforcedNestedStack extends NestedStack {
  protected readonly REGION: string;
  protected readonly ACCOUNT_ID: string;
  protected readonly ENV: string;
  protected readonly CONTEXT: string;
  protected readonly CONTEXT_VARIABLES: ContextVariables;

  constructor(scope: Construct, id: string, props: EnforcedStackProps) {
    super(scope, id, props);

    const { account: ACCOUNT_ID, region: REGION } = Stack.of(this);
    this.REGION = REGION;
    this.ACCOUNT_ID = ACCOUNT_ID;

    const { contextVariables: variables } = props;
    this.ENV = variables.stage;
    this.CONTEXT = variables.context;
    this.CONTEXT_VARIABLES = variables
  }
}
export class EnforcedStack extends Stack { 
  protected readonly REGION: string;
  protected readonly ACCOUNT_ID: string;
  protected readonly ENV: string;
  protected readonly CONTEXT: string;
  protected readonly CONTEXT_VARIABLES: ContextVariables;

  constructor(scope: Construct, id: string, props: EnforcedStackProps) {
    super(scope, id, props);

    const { account, region } = Stack.of(this);
    this.REGION = REGION;
    this.ACCOUNT_ID = ACCOUNT_ID;

    const { contextVariables } = props;
    this.ENV = contextVariables.stage;
    this.CONTEXT = contextVariables.context;
    this.CONTEXT_VARIABLES = contextVariables

    this.node.addValidation(new ContextVariablesValidator(contextVariables));
    this.node.addValidation(new WorkloadEnvValidator(contextVariables));
    Aspects.of(this).add(new AwsSolutionsChecks());

    if( contextVariables.usage !== 'PRODUCTION' ) 
      Aspects.of(this).add(new ApplyDestroyPolicyAspect());

    if( contextVariables.usage === 'EPHEMERAL' ) {

      const now = new Date(new Date().getTime() + 24 * 60 * 60 * 1000);

      const deleteFunction = new NodejsFunction(this, 'DeleteFunction', {
        handler: 'index.handler',
        runtime: Runtime.NODEJS_20_X,
        timeout: Duration.minutes(15),
        entry: join(process.cwd(), 'core/custom-resources', 'remove-stack.ts'),
        environment: {
          STACK_NAME: this.stackName,
        },
        role: new Role(this, 'DeleteFunctionRole', {
          assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
          inlinePolicies: {
            'CloudFormationPolicy': new PolicyDocument({
              statements: [
                new PolicyStatement({
                  actions: ['cloudformation:DeleteStack'],
                  resources: [
                    Stack.of(this).formatArn({
                      service: 'cloudformation',
                      resource: 'stack',
                      resourceName: `${this.stackName}/*`,
                      arnFormat: ArnFormat.SLASH_RESOURCE_NAME
                    }),
                  ]
                })
              ]
            })
          }
        })
      });

      new Rule(this, 'EphemeralRule', {
        schedule: Schedule.cron({
          minute: now.getMinutes().toString(),
          hour: now.getHours().toString(),
          day: now.getDate().toString(),
          month: now.getMonth().toString(),
          year:  now.getFullYear().toString(),
        }),
        targets: [ new LambdaFunction(deleteFunction)],
      });
    }

    Aspects.of(this).add(new ApplyTagsAspect({
      context: contextVariables.context,
      stage: contextVariables.stage,
      owner: contextVariables.owner,
      usage: contextVariables.usage,
    }));

    Aspects.of(this).add(new ApplyParameterStoreNamingPolicyAspect(contextVariables));

  }
}

The example applies all aspects and validations in a central place, also, as a specific detail, it uses one-time event bridge schedules to remove the ephemeral stacks after 24 hours.

Conclusion

Adopting IaC was a great evolution in how companies have been dealing with infrastructure, but as the rapidity of cloud infrastructure creation leads to a lot more amount of resources, this becomes important to enable teams to apply practices and help enterprises without becoming road-blockers or reducing the velocity of development teams.

AWS CDK is not only an object-oriented way of writing IaC but also provides many design patterns under the hood that can be extended to some requirements such as compliance, security, etc.

When working with Lambda, it's important to explore the intricacies of its execution environment and how it is managed. In this article, I will share my insights and observations, though some aspects may be open to debate.

Execution Environment

An execution environment is an isolated container (Micro-VM) that is launched on demand to handle incoming requests. Each environment processes one request at a time but can handle subsequent requests once the previous one is complete. If a new request arrives before the ongoing one finishes, an additional execution environment is created to manage it. The diagram below illustrates the lifecycle of an execution environment.

An Execution environment follows 3 phases: Initialization, Invocation, and shutdown

Init phase

The init includes initiating runtime, registering configured extensions, and downloading code packages which happen sequentially in respective order.

Invocation Phase

The Runtime, extensions, and function code will be invoked during the Invocation phase.

Shutdown Phase

The shutdown phase will shut down the runtime and send the shutdown signal to all extensions letting them clean up and finish the remaining work.

Reuse of Environment

An Execution Environment will be used for the subsequent requests, so the lambda will enter the shutdown phase if an execution environment does not receive any request for a period of time. The duration between the end of the invocation phase and the start of the shutdown phase is the idle duration when the Lambda service allows the reuse of that available environment.

AWS Lambda has a lot of interesting technical details. Lambda will not leave the execution environment running while there is no new demand, but the execution environment gets frozen and will be Thawed if it receives any new demand.

In the case of unintentional interruptions, the lambda will run the initialization as part of the next invocation, but this just seems to be a light init.

Code Execution

Let’s first see how the code is executed for any request. When the first request is received, the lambda initiates the environment by running the top-level code. How the init phase behaves depends on the programming language used and how the code is written. Typically, when using NodeJs what happens during the initialization phase is like running the following command

> node index.js

The lambda service will then execute the function handler, when the execution is finished, the execution environment gets frozen. Thawing the execution environment i the tricky part. When lambda Freezes and the execution environment all background processes will be frozen and will be executed by thawing back the execution environment. But what about if the execution environment gets no more requests? The execution environment gets shut down and everything will be lost.

Freeze

When the execution environment gets frozen under the hood, the container gets into kinda hibernate state, when all resources get in a sleep state, like when a PC goes into hibernate mode, The address space of all running processes will be registered, allowing reconstruction of same state next.

Thaw

The thawing stage is part of runtime invoke when a container is reused as part of the invocation phase; the lambda service invokes the runtime, but this theoretically must be behind the Frozen background process reconstruction.

I could not find a response to what happens when the execution environment wakes up, but this must be when the container gets awake and not the runtime.

Try it out

The following example will create a lambda-based Api with two endpoints, one for awaited and another for non-awaited tasks to observe how lambda will behave in real time.

Both functions have the same code except the first use the await and the second non.

import { LambdaFunctionURLEvent, LambdaFunctionURLResult } from "aws-lambda";

const delay = async (ms: number) => {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

const Task = async (req: string, name: string, sleep: number) => {
  await delay(sleep);
  console.log(`${name} : `, req );
  return { name: `${name}` };
}
export const handler = async (_event: LambdaFunctionURLEvent): Promise<LambdaFunctionURLResult> => {

  const resultA = await Task(_event.requestContext.requestId, "TaskA", 1000);
  const resultB = await Task(_event.requestContext.requestId, "TaskB", 2000);
  const resultC = await Task(_event.requestContext.requestId, "TaskC", 3000);

  const result = {
    resultA,
    resultB,
    resultC,
  };
  return {
    statusCode: 200,
    body: JSON.stringify(result, null, 2),
    headers: {
      "Content-Type": "application/json",
    },
  };
}

Running the awaited function will give a response time of 6.XX seconds accumulating 1000, 2000, and 3000 milliseconds.

The non-awaited function has the same code but calls TaskB and TaskC without waiting.

const resultA = await Task(_event.requestContext.requestId, "TaskA", 1000);
const resultB = Task(_event.requestContext.requestId, "TaskB", 2000);
const resultC = Task(_event.requestContext.requestId, "TaskC", 3000);

Running the first request gives the following logs, showing that only TaskA is terminated and the log is present.

Running a second request results in the logs as below, the previous execution remaining tasks are executed as part of subsequent invocations.

But the interesting part is how long they took to log. The TaskB and TaskC are executed at the same time and ended instantly. The following image shows the response time as 1105 ms, which is normal for new invocation TaskA.

Looking at the Non-Awaited code previously, the TaskB and TaskC must take around 5000 ms together that is not the case. Looking at Billing Duration the Billed duration corresponds TaskA execution.

Real scenario

To trust better the hypothesis ( i did not ) and validate that there will be no tricky side effect, we gonna send a message to an SQS queue just to prove the idea behind the observations is real.

By running the first request there will be nothing fancy except a new message in the queue for TaskA.

Now, let’s run another request and see what happens. Here, TaskB and TaskC just got included in the execution, and the new messages are present in the queue. the top 1 in the following screenshot is the first invocation message, and the three others are TaskB/TaskC of the first invocation + New Invocation TaskA.

In the above examples, the logs ingested were the logs after treatment. I was curious to see if I could observe how those function calls happen so i kept it simple by adding logs for the start and end of each treatment.

Here how Task method looks like

export const Task = async (req: string, name: string, sleep: number) => {
  console.log(`Starting ${name} : `, req );
  await delay(sleep);
  console.log(`Waking up ${name} : `, req );
  await client.send(new SendMessageCommand({
    QueueUrl: process.env.QUEUE_URL,
    MessageBody: JSON.stringify({ req, name }),
  }));
  console.log(`End ${name} : `, req );
  return { name: `${name}` };
}

The Task method has three levels of Logs indicating the Starting, WakingUp, and End. According to the following logs, the first represents an execution that the frozen tasks are executed before the current execution, and the second one, TaskB woke up at the same time as the current invocation start ( CloudWatch log ordering is defaulted by time ). based on these observations frozen tasks are not executed at the function handler invoke phase but at runtime invoke apparently.

These observations can prove that the frozen processes will be run as soon as the execution environment is thawed, and this seems like a kind of decoupling ( this is only a hypothesis ) to prove the coupling or decoupling running some failures for TaskB and TaskC will illustrate better the internal state.

Simulating Failures

The Task Method will be changed to cover failing explicitly per Task name as following code snippet.

export const Task = async (
    req: string,
    name: string,
    sleep: number,
    extendedProcess?: Function
) => {
  console.log(`Starting ${name} : `, req );
  await delay(sleep);
  console.log(`Waking up ${name} : `, req );
  if( extendedProcess ){ extendedProcess(); }
  await client.send(new SendMessageCommand({
    QueueUrl: process.env.QUEUE_URL,
    MessageBody: JSON.stringify({ req, name }),
  }));
  console.log(`End ${name} : `, req );
  return { name: `${name}` };
}

The extendedProcess is passed in TaskB initialization call as below

const extendedProcess = (name: string, req: string) => {
  console.log(`Failing ${name} : `, req );
  throw new Error('TaskB Failed');
}

export const handler = async (_event: LambdaFunctionURLEvent): Promise<LambdaFunctionURLResult> => {
  ...
  const resultB = Task(
     _event.requestContext.requestId,
     "TaskB",
     2000,
     extendedProcess("TaskB", _event.requestContext.requestId));
  ...
}

The first request will behave as before, but during the second execution, the previously frozen TaskB will result in the current execution interruption by throwing an UnresolvedPromiseRejection error. This proves that the Freeze/Thaw of incompleted tasks can be dangerous and breaks the AWS Lambda design principle of having isolated event-based processing. this approves a level of coupling that can become dangerous without careful implementation.

Tracking state

This is just a game play not a way to implement production ready solutions

This experimentation pushed me to think about how the lambda can behave like a stateful container and act based on the state. To experiment with state tracking, a use case can be a task that will provide some state in the execution environment, which can be done using a variable outside the handler, but this time, I want to try not to keep only the state but defer the actions and see if possible.

How the idea behaves can be demonstrated by the following sequence diagram

The idea is that the un-waited task will check the execution environment state and and modify it. What was achived in this test :

• Accumulting State in a Dictionary outside handler

• Pushing accumulated items to an SQS when count reaches 10 item

But what about undesired situation , during the time i played i had some sort of timeout occasionally, while deep diving i discovered the dictionary state got empty after a timeout as per following screenshot from cloudwatch logs.

Conclusion

While this is cool to try and fail, this article just was a game around how lambda execution environment behaves, and the behavior seems like an hibernate , when again awaked, the process will come back and resuming, did you ever tried a hibernate while coping a huge folder from one drive to other ? this is the same behavior.

The final note is, a controlled and imperative programming model is many times more efficient that behavioral programming, actually for this case a better approach is using the lambda layer to push the logs but again this was a fun and i thought it is worth to share with the community.

Enjoy reading

However, we often overlook the foundational principles of IaC, like recovery, fast deployment, resiliency, and minimizing time-to-market (TTM). For instance, if you decide to implement a multi-regional failover, can it be deployed effortlessly? How quickly can you recover if your production environment, region, or accounts go down?

Common Roadblocks

To navigate the complexities of IaC, vigilance and discipline are essential. Let’s explore some common roadblocks and how to address them effectively:

• Historical manual interventions

• Lack of configurable and parameterized code

• Lost secrets that cannot be restored

• Hard dependencies between stacks

• Circular dependencies

Human Actions

Human intervention is a regular part of our jobs—quick fixes to temporary issues often lead to future automation. Unfortunately, these "notes for later" sometimes get forgotten, turning into major pain points down the road. Identifying these recurring manual actions is key to saving time, effort, and frustration in the future.

Recommendations:

• Use tags to identify automated resources.

• Regularly explore untagged resources to detect those not yet automated.

• Review generated IaC templates to find missing tags.

• Foster a tech-driven culture within your team.

Configuration Shortcomings

One frequent issue in IaC is hardcoding variables in Stacks or Nested Stacks, which can complicate configuration management. Whether it's a queue name, topic name, or HTTP endpoint, manually searching through different IaC tools like CloudFormation or AWS CDK can slow you down. Centralizing all dependencies simplifies future changes—like shifting from "me.mycompany.com" to "me.mycompany.org"—by allowing you to quickly locate and update configurations.

Losing Secrets

Managing secrets securely is crucial. While storing sensitive information like API keys or credentials in a secret manager is helpful, what happens if your account gets lost? What about your partner’s lost secrets? What about the secrets you shared with your partners ? The solution is to maintain a backup of all necessary secrets outside the software environment, ideally in a dedicated vault—this is more of an organizational best practice.

Managing Dependencies

Dependencies in IaC can be categorized into three levels:

1. Light Dependencies: Passed to the environment variables (e.g., Lambda), these won’t break your deployment but could affect testing and runtime.

2. Soft Dependencies: Tied to infrastructure services but manageable—like subscribing to an SNS topic, though permission issues may arise from unautomated historical actions.

3. Hard Dependencies: These will prevent deployment if not properly handled. For example, an EventBridge rule may require an EventBus that isn’t yet deployed. The key here is identifying priority stacks and documenting these relationships, often using dependency graphs or architecture diagrams.

Circular Dependencies

Over time, as requirements evolve, stacks can develop circular dependencies. Imagine planning a production release only to find that it fails due to a circular dependency between two stacks. For instance, Stack A may require a CloudFront distribution that needs an upstream domain name for CORS, but the record set is managed in another stack—leading to a deadlock.

To avoid such issues, actively manage and mitigate circular dependencies. Divide stacks if needed or apply predictable naming conventions. For example, using "products.mycompany.com" instead of introducing direct dependencies between stacks can eliminate such problems.

By proactively addressing these common challenges, we can build more resilient, efficient, and scalable infrastructure, reducing downtime and increasing the speed of recovery when issues arise.

This article delves into different concepts and details regarding adopting micro-frontends, giving more clarity and vision about tradeoffs and the under-the-hood parts. To learn more about the impacts and goals of a MicroFrontend design, I highly recommend the Building Micro-Frontends book by Luca Mezzalira.

The conception at the root of the proposal applies to achieve the above ideas, but when it comes to design and implementation, this leads to a lot of discussions and details to take into account and consider. choosing the right approach depends on the tradeoffs and the priorities that business follows.

Micro-frontends are not far from microservice design approach, as they apply the same goals fundamentally, but at the same time brings exact or partial complexities and challenges the microservices introduce.

The Primer

At high level a micro-frontend architecture leads to answer achieving independent and autonomous collaboration to satisfy a final result asset within a distributed system.

In the above schema, the 3 services are deployed, maintained, and scale independently and will be composed together as a final result asset.

This is a simple, small and straight forward example but at higher scale we need to think about:

• Templating

• Discovery

• Routing

• Composition

Boundary Definition Challenges

One principal challenge when designing Micro-Frontends is how evaluate the boundaries of each MFE. Often having too many small distributed micro frontends seems to bring more flexibility and reusability opportunities, so a MFE can be composed in as many places as the client app needs. While having MFEs with small defined contexts bring the advantage of having self managed and autonomous MFEs but at the same time, adds more complexity at Discovery & Routing part, also adds the communication chattiness into MFEs by calling the downstream services multiple times while the result asset will be in the same context. In the other side, having too big MFEs can improve some challenges small MFEs brings but gives less flexibility and opportunity of reusability, and will introduce the rendering performance issues, complexities and performance degradation at client side.

Having MFEs with a logical size based on the context is a key to success, at the same time it brings more logic at MFE side but avoid adding complexity at other parts of the system.

Some factors to consider while defining a micro-frontend context

• Performance

• Lifecycle

• Co-existence

• Reusability

• Ownership

• Clarity & Vision

• Team Topology possibilities

Horizontal vs Vertical Splitting

The splitting refers to the fact of composing the MFEs on the horizontal or vertical axis.

In horizontal splitting the composition introduces a coupling like having a web app page dedicated to a micro-frontend in this case the components inside that given page have less opportunity to be reused in other contexts and pages. an example will be a widget that renders the active products for a given customer, while it can be shown in different pages like customer detail page or in the own customers admin page. Horizontal splitting leads to duplicated components, this seems ok but can be tricky when the number of reusable components raises or the standardisation and governance becomes a concern like applying design systems, complex business logic, etc.

In a vertical splitting the chance of reusability increases and a component has the chance of being reused in any other contexts and result assets. but it adds some layer of duplicated communication and data fetching but also at some points introduces useless decomposition.

When it comes to splitting a better approach is to pragmatically define the right boundary but also composition layer. often teams stands for applying MFE composition at a single layer being Web Server, Edge or shell. but the better approach seems to define right decomposition at server side while giving flexibility to future reuse but also right composition in shell ( kernel ).

In the above diagram having product catalog and insights at the same layer ( server side ) helps to reduce the unnecessary downstream service calls but adds some challenges such as cache configuration. So putting two Components in the same MFE must be considered when those components follow the same lifecycle.

This is important to think about different decision axis to adopt the right approach based on requirements and tradeoffs.

Rendering

The choice of rendering approach depends on functional requirements and technical possibilities. My journey began with front-end development using VBScript, where code ran on the client side. However, limitations such as database connectivity and security concerns led to a shift towards server-side development with ASP.NET, where interactions and rendering were handled on the server. As JavaScript technologies advanced, particularly with the introduction of Ajax, the drawbacks of server-side latency became apparent. This prompted a shift back to client-side rendering, allowing for partial or full application rendering on the client side, with DOM manipulation and updates occurring after API responses were received. This approach was well-suited to the era of limited server resources and networking constraints.

However, as server capabilities improved, particularly with the advent of cloud computing and rapid scaling, it became advantageous to offload processing back to the server. Today, server-side rendering (SSR) is a crucial front-end design decision, thanks to its ability to respond in milliseconds and scale efficiently. However, it's not always the optimal choice and should be considered within the context of the overall architecture.

Rendering choices are critical and involve several factors, including:

• User Experience

• Performance

• Cost

• Search Engine Optimization (SEO)

While other considerations like Separation of Concerns (SoC) and polyglot ecosystems are important, the ones listed above are the primary focus here. In the context of Micro-Frontends, various rendering strategies can be employed, such as Client-Side Rendering (CSR), Edge-Side Rendering (ESR), Server-Side Rendering (SSR), and Server-Side Generation (SSG), which is a variation of SSR.

Client Side Rendering

CSR applies rendering of HTML content in the browser using javascript dynamically, But with CSR a preliminary content is received from the server containing the javascript and CSS, further the execution of those javascript in client side will lead to content rendering by calling the required api or backend services. Before the script execution, the content will be appeared and will be interactive.

To see more how the CSR and SSR behave this video by Scott Hanselman is a nice one.

Server Side Rendering

SSR is all about rendering the HTML content at server side before returning to the client (browser), The server request can be done at the page load or behind any action with the ability to trigger a request. The server response contains the content but also all necessary scripts that lead to have interactive content on the client side.

SSR can be applied in a variety of ways but listening to experts like Luca Mezzalira helps to capture enough and this video is a rich and detailed one

Server Side Generation

SSG is the technique of generating HTML assets in server side and let the client use the pre rendered assets, This is really close to the fact of using static assets but in SSG the asset content is HTML. The SSG is great if the number of required assets are limited but this can be tricky when dealing with millions of items. when using SSG it s important to consider how long and till when that assets is really required and estimate the resulting costs.

Edge Side Rendering

ESR helps to tackle some SSR challenges such as latency and performance, The ESR helps to respond to client request at Edge and benefit from the global Point of presence close to the user, aside of latency improvements by removing the full load on origin, ESR also helps to remove the load of processing on client side being a browser, mobile app or etc.. to achieve a more device friendly approach.

Orchestration

The Micro-Frontend orchestration is a pattern that is used to apply composition of different micro frontends and makes a final result that represents a page or a representable and meaningful part of a UI. The orchestration can be done in different levels being Client side, Server side or Edge side. and also an intelligent mix of them can be applied.

This article fundamentally focuses on Server Side Rendering and demonstrates some challenges the teams face while designing SSR.

Kernel / Shell

As discussed earlier, while Micro Frontends (MFE) offer significant benefits to the overall system, they can also introduce cross-domain challenges and coordination issues among teams as they work to integrate distributed components into a cohesive solution. To address these complexities, a shell (or kernel) adds a layer of abstraction between the client and the distributed services, streamlining communication and simplifying the composition process, ultimately leading to a more flexible design.

Implementing a shell offers the advantage of simplifying the previously mentioned requirements. It facilitates easier discovery of services and their versions, resolves template details, routes requests to the appropriate service based on the resolved template, and ultimately handles the composition.

A shell (or kernel) is particularly valuable when managing various types of rendering—whether client-side, edge-side, server-side, or a combination of these approaches.

A shell can respond to the following requirements if applied:

• Service Discovery

• Template Discovery and rendering

• Service deployment strategy ( Canary, Rolling, All-At-Once )

• Error handling and fallbacks

• Instrumenting for Observability

Templating

A template serves as a predefined framework that outlines the structure of the result set required by a client application, whether it's a browser or a mobile app. Consequently, the templating module must identify the appropriate template based on the request information received from the client. For instance, a template represents a page that includes various components such as scripts, meta tags, CSS references, and more.

At the first glance maybe templating is not really interesting approach but it helps to resolve a lot of frontend cross cutting concerns like shared scripts, design systems , and etc, without changing and deploying all MFEs or client app.

Discovery & Routing

Discovery is the process of locating the appropriate services based on identifiers, versions, and other factors. It provides a more granular approach to integrating services and the fundamental aspects of a distributed design with reduced complexity.

A high-level implementation of discovery would look like this:

The routing module is responsible for directing external and public service requests to the appropriate internal or private distributed services. An effective routing module takes into account the deployment strategy and the type of composition, whether server-side, client-side, or edge-side.

The diagram illustrates the routing process when server-side composition is used.

Routing can be handled either by the Web Server or the Shell. When the Shell manages routing, it performs lightweight mapping based on the incoming request and the associated micro-frontend. The Shell directs traffic to the appropriate service and conducts basic contextual and generic checks. More detailed routing, which requires domain-specific knowledge, is managed within the micro-frontend itself. This includes scenarios such as applying a specific version for an individual customer or a particular list of products.

Shell Challenges

When designing a shell the most important thing to consider is to design a shell agnostic of business details, often the teams start light and brings incrementally the complexity into shell for a single reason, and this is the shell is already there and the easiest way for change. But at longterm this will lead to have a higher risk of changes and distributed knowledge. why this is risky ? a shell can be central and need to be evolved by a wide range of teams, this becomes tricky in long run.

What we gonna build

The following diagram shows the application we are building for this article, we will build a simple react application providing a client application that interact with server side resources via a shell ( Kernel ) that provides the discovery & routing, and templating and a light Api to help the micro-frontends communicate together. Each micro-frontend provides the html in response which will be shown in our Web App.

Source Code

The example related to this part of series is Part-01 branch in following Github repository.

Bookmarks

The Bookmarks Micro frontend stack has the responsibility of returning the bookmarks for a given userid, the bookmarks MFE validates the presence of UserId parameter and returns the results for the corresponding user.

In bookmarks service, the bookmarks list MFE is vertically sliced, but all downstream service calls can be managed if required.

The bookmarks list mfe returns all bookmarks for given userid and returns an 4xx error if the userid not provided. also it applies the filtering of user bookmarks per product reference and name if applied.

The caching is applied per query string parameters including userid, ref, name. the following snippet represents the CDK example for bookmark service and related configuration.

The bookmarks distribution owns the caching requirements of bookmarks service, this is related to the way that each service owns and master its proper requirements.

this.Distribution = new Distribution(this, 'BookmarkDistribution', {
    ...
    defaultBehavior: {
        origin: new FunctionUrlOrigin(props.DefaultOriginListBookmarksFunctionUrl, {
            connectionAttempts: 3,
            connectionTimeout: Duration.seconds(1),
            keepaliveTimeout: Duration.seconds(5),
        }),
        allowedMethods: AllowedMethods.ALLOW_ALL,
        cachedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
        cachePolicy: new CachePolicy(this, 'BookmarksCachePolicy', {
            queryStringBehavior: CacheQueryStringBehavior.allowList(
                'userid',
                'ref',
                'name'
            ),
            defaultTtl: Duration.hours(1),
            minTtl: Duration.hours(0),
            maxTtl: Duration.hours(24),
        }),
        viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
        originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
    },
});

const cfCfnDist = this.Distribution.node.defaultChild as CfnDistribution;

const bookmarksOriginAccessControl = new CfnOriginAccessControl(this, 'LambdaUrlOAC', {
    originAccessControlConfig: {
        name: `Bookmarks-Lambda-OAC`,
        originAccessControlOriginType: 'lambda',
        signingBehavior: 'no-override',
        signingProtocol: 'sigv4',
    }
});

cfCfnDist.addPropertyOverride(
    'DistributionConfig.Origins.0.OriginAccessControlId',
    bookmarksOriginAccessControl.getAtt('Id')
);

The cloudfront and function url integration is done using OAC Signature V4, the function url grants the permissions to let the calls only be authorized only from cloudfront.

this.FunctionUrl.grantInvokeUrl(new ServicePrincipal('cloudfront.amazonaws.com', {
     conditions: {
        ArnLike: {
            'aws:SourceArn': `arn:aws:cloudfront::${account}:distribution/XXXXXXXXX`,
        },
        StringEquals: { 'aws:SourceAccount': account},
      }
}));

Products

The Product service provides two distinct MFEs being catalog and Product Details Page ( PDP ), those are independent and isolated ones.

The product service follows same implementation and application architecture principles as bookmarks so vertically sliced and optimized for potential reusability needs. Here, the important note is possibility of sharing the repository layer, this is not an interest from programming perspective but more communication, as this let us to simply have multiple MFEs and apply composition for different components inside one MFE and reducing the number of network calls or Database calls.

The products service follows the same principals as bookmarks in terms of caching, security and uses also lambda function url for api invocations.

Web Application

The web application consists of Two pages , home page and product details page, the home page serves as a landing for bookmarks , and catalog micro frontends, while the PDP hosts only the product details micro frontend.

The Web app home page includes two MFEs , Product Catalog and bookmarks list, but this is a different case from the previous interests we had when optimizing to reduce network calls. The interest here applies more on organizational and bounded context.

• Each micro-frontend is owned and runs by different team

• Each one has its proper context so database or downstream service calls

• There is no domain context based relation between them.

In this diagram the react app calls both MFEs in parallel and use the received results as html assets that live side by side. The website build assets will be uploaded on a dedicated bucket that will be as the default behavior of web app cloudfront. The web app cloudfront applies the required caching for static web site assets, but for dynamic routes as the ones related to home page calls, bookmarks and catalog, and product details page applies no caching but only forward the requests to the downstream MFE servies.

Bundling

To bundle the website app, the build is done using react-scripts build command, the bundle results will be deployed to the web app s3 bucket using CDK BucketDeployment construct.

new BucketDeployment(this, 'BucketDeployment', {
   sources: [ Source.asset(join(process.cwd(), '/front-app/website/build')) ],
   cacheControl: [CacheControl.fromString('max-age=1800,must-revalidate')],
   destinationBucket: frontStack.Bucket,
   distribution: ditribution.Distribution,
   distributionPaths: ['/*'],
});

The use of BucketDeployment is for sake of demonstration but in real projects the deployment process must be under a dedicated cicd pipeline with multiple stages including CI ( linting, testing, analysis, etc.) and CD.

Caching

The cloudfront behavior for the dynamic content will use a dedicated CachePolicy and OriginAcessPolicy applying no caching but forwarding all request QueryString parameters.

const dynamicContentCachePolicy = new CachePolicy(this, 'DynamicContentCachePolicy', {
    headerBehavior: CacheHeaderBehavior.none(),
    cookieBehavior: CacheHeaderBehavior.none(),
    queryStringBehavior: CacheHeaderBehavior.none(),
    defaultTtl: Duration.seconds(0),
    minTtl: Duration.seconds(0),
});

const dynamicContentOriginRequestPolicy = new OriginRequestPolicy(this, 'DynamicContentOriginRequestPolicy', {
    queryStringBehavior: OriginRequestQueryStringBehavior.all(),
    headerBehavior: OriginRequestHeaderBehavior.none(),
    cookieBehavior: OriginRequestCookieBehavior.none()
});

this.Distribution.addBehavior('api/v1/bookmarks/*', new HttpOrigin(props.BookmarkServiceDomainName), {
    allowedMethods: AllowedMethods.ALLOW_ALL,
    cachePolicy: dynamicContentCachePolicy,
    viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
    originRequestPolicy: dynamicContentOriginRequestPolicy
});

Rendering

The underlying code for the example website will be a simple react app, using useEffects react hook to fetch the data on page load.

useEffect(() => {
    async function RenderMFEs() {
      const promiseProcessSuccess = 'fulfilled';
      await Promise.allSettled([
        fetchMfe(Mfes.BOOKMARKS_LIST),
        fetchMfe(Mfes.PRODUCT_CATALOG),
      ]).then((results) => {
        results.forEach((result) => {
          if (result.status === 'rejected') console.error('HP error :', result.reason) });
        if (results[0].status === promiseProcessSuccess) setBookmarks(results[0].value);
        if (results[1].status === promiseProcessSuccess) setCatalog(results[1].value);

      });
    };

    if (!bookmarks || !catalog) 
      RenderMFEs();

  }, [bookmarks, catalog]);

The following code demonstrates the FetchMfe() function, which is called in useEffect react hook.

export const enum Mfes {
  'PRODUCT_DETAILS',
  'BOOKMARKS_LIST',
  'PRODUCT_CATALOG',
}
export const Paths: Record<Mfes, string> = {
  [Mfes.PRODUCT_DETAILS]: '/api/v1/products/details/',
  [Mfes.PRODUCT_CATALOG]: '/api/v1/products/catalog/',
  [Mfes.BOOKMARKS_LIST]: '/api/v1/bookmarks/',
}

export const fetchMfe = async (MFE: Mfes, host?: string): Promise<string>  => {

  const hostDomain = `${window.location.protocol}//${window.location.host}/`;

  let urlPath = '';
  urlPath = urlPath.concat(Paths[MFE]);

  const url = new URL(urlPath, hostDomain);

  const queryParameters = new URLSearchParams(window.location.search);
  if(queryParameters) url.search = queryParameters.toString();

  const res = await fetch(`${url.href}`);
  return await res.text();
}

Redirection

As per requirement the redirection of certain URLs are important for SEO and brand trustworthy on the public, this is the case when certain links are no more valuable or has no corresponding result ( ex. when a product get out of stock or a url is deleted permanently).

The following CDK shows how to use a CloudFront Function to apply simple redirection for specific paths. The CloudFront functions use a lightweight version of Javascript. This way we can easily apply the redirection on top of specific urls under a distribution behavior and let the crawler to accumulate the old url score and brings them to the new redirected url ( this is how google indexation works when the permanent redirection is applied)

Thanks to David Behroozi sharing the great and cost effective solution for redirection. This sections use the mentioned solution for our dedicated purpose.

defaultBehavior: {
   allowedMethods: AllowedMethods.ALLOW_GET_HEAD,
   origin: webbucketOrigin,
   cachedMethods: AllowedMethods.ALLOW_GET_HEAD,
   cachePolicy: WebCachePolicy,
   viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
   functionAssociations: [
        {
           function: new Function(scope, `ProductRedirectFunctionViewerResponse`, {
                code: FunctionCode.fromFile({ filePath: `front-app/src/url-redirect.js`}),
                runtime: FunctionRuntime.JS_2_0 
           }),
           eventType: FunctionEventType.VIEWER_RESPONSE,
        },
   ]}

The CloudFront function will be triggered at viewer response CloudFront event source trigger stage and applies a 301 permanent redirection.

The solution uses the s3 user-defined metadata to register the redirection target link.

// @ts-ignore
function handler(event) {
  console.log(JSON.stringify(event, null, 2));
  const response = event.response,
        headers = response.headers,
        request = event.request;

  const header = 'x-amz-meta-location';

  if ( 
    'GET' == request.method &&  
    200 == response.statusCode && 
    headers[header] && 
    headers[header].value
  ) {
      headers.location = { value: headers[header].value };
      return {
        statusCode: 301,
        statusDescription: 'Moved Permanently',
        headers,
      };
  }
  return response;
}

The example ues BucketDeployment construct to upload a single file with corresponding metadata.

new BucketDeployment(this, 'RedirectionDeployment', {
   sources: [ Source.asset(join(process.cwd(), '/front-app/src/redirection-files')) ],
   metadata: {
      'location': `https://${ditribution.Distribution.distributionDomainName}/?category=ON_SOLD`,
   },
   destinationBucket: frontStack.Bucket,
   prune: false,
});

In this example all files in redirection-files folder will be uploaded with a location to the s3 bucket. but in real projects it can be any dedicated back-office generating files with corresponding metadata.

Shell

The shell is responsible for composing a template from different independent micro-frontends and returning the final result asset to the client app. The shell also applies some cross cutting concerns such as Authentication/Authorization, Graceful Degradation and Logging while acceding server side resources.

One of the most popular responsibilities of a shell is service (MFEs) discovery, a service discovery considers also the deployment strategy if applies, such as wighted , canary or Blue/Green.

A shell often impose an standard such as following example that can be integrated in a template as demonstrated below

<MfeTag
   src='...'
   error-handling='fallback'
   fallback='...'
   options='cors,auth,apikey,tls'
   timeout=1000
   passthrough='cookies=[...],query=[...],headers=[...]'
   strategy='canary'
/>

The above example represent a a simple custom HTML tag indicating under which constraints the MFEs communication shall be done.

in this example the shell will send a http call to the backend

In the next part of this series we will deep dive in the implementation details and design tradeoffs while considering a shell as a candidate to be user or not. The next part will focus on how to achieve a valuable shell and when consider applying a shell or adopt a simplified approach with less cognitive load but with a distributed responsibility at MFE sides per servie or Context.

Running the Example

To run the example locally, The backend MFEs must be ran on a local host server to let the local react website be functional. We use a simple way of running the typescript services locally over localhost( port 4242) to achieve local distributed MFEs and let the website communicate over localhost with MFEs.

The provided local server script is a simplified version of the script i used previously to run the backend lambda services with a minimum overhead and dependencies. A thanks to Zied Ben Tahar for giving the indications and helps to make this functional while we worked together at Aviv.

First, Lets deep a bit to see how this works. The script simply loop through an array of entrypoint and invoke the configured function which is exported.

const handleRoute = async (lambdaEntry: { entryPoint: string, handlerFn?: string, action?: (...args: any) => void }, req: Request, res: Response ) => {
   const module = await import(lambdaEntry.entryPoint);
   const lambdaFunctionHandler = module[lambdaEntry.handlerFn ?? "handler"];
   const result = await lambdaFunctionHandler(
       generateEvent(req), 
       createLambdaContextObjectFromContextPayload(req.body.context)
   );
   return res.send(result.body).end();
}

The entrypoints are the ts or tsx files, imported using dynamic import.

also the script uses the express to setup a local server using some defined middleware to use and the express router and provide all http verbs for each entrypoint.

const app = express();
app.use(cors());
app.use(express.json());
app.use(express.text());
app.use(express.raw());
app.use(express.urlencoded({ extended: true }));
const router = express.Router();

lambdasEntrypoints.forEach((lambdaEntry) => {
  router.all(`/${lambdaEntry.endpoint}/`, async (req: Request, res: Response) => {
    return handleRoute(lambdaEntry, req, res);
  });
});

The Entrypoints configs are configured as an array of entrypoints and are fetched using the glob package. The paths correspond to the corresponding MFEs real origin behavior to avoid changing the website code just for local testing.

export type ConfigSource = { path: string, source: string, handlerFn?: string, action?: (...args: any) => void };
const configs: ConfigSource[] = [
    { path: "api/v1/bookmarks/", source: 'micro-fronends/bookmarks/src/handlers/list/index.ts' },
    { path: "api/v1/products/catalog/", source: 'micro-fronends/products/src/handlers/catalog/index.ts' },
    { path: "api/v1/products/details/", source: 'micro-fronends/products/src/handlers/details/index.ts', action: (req: Request, res: Response) => { console.log(req); res.writeHead(302, {Location: `/api/v1/products/catalog/v1/?category=ON_SOLD`}).end();} },
];

export const lambdasEntrypoints = globSync(configs.map(src => src.source ?? './')  , { 
    ignore: [
        "**/**/node_modules/**",
        "**/**/*..test.ts",
        "**/**/*..spec.ts",
    ] }).map((entry: string) => {
        const config = configs.find(c => c.source.includes(entry));
        const entryPoint = join(process.cwd(), entry.split(path.sep).join(path.posix.sep)),
              lambdaName = entry
                .split(path.sep)
                .slice(-1)[0]
                .replace(".(ts|js)", ""),
              endpoint = config?.path,
              handlerFn = config?.handlerFn,
              action = config?.action;

        return { entryPoint, lambdaName, endpoint, handlerFn, action };
    });

The example local MFEs can be started using following comand

$ npm run local:start
------------ 
[Local λ debugger]: Local lambda invoke debug server is running at http://localhost:4242
[Local λ debugger]: Discovered 3 lambdas entrypoints
  [λ endpoint]: api/v1/bookmarks/
    [exported functions]: handler
  [λ endpoint]: api/v1/products/catalog/
    [exported functions]: handler
  [λ endpoint]: api/v1/products/details/
    [exported functions]: handler

The website can be ran using folling command

$ npm run local:website

The package.json local:website script passes the local url using env variable and the Shared FetchMFE function look at this environment variable to decide where to look, principal host or the local.

{
   "local:website": "REACT_APP_LOCAL_URL=http://localhost:4242 npm run start --prefix front-app/website"
}

Addressing the navigation to `http://localhost:3000/?userid=HJ-HnhYul_sm` url will fetch the user bookmarks and also the product catalog page as below

By adding the product reference as a query parameter the bookmarks and catalog will show only the corresponding references ( example http://localhost:3000/?userid=HJ-HnhYul_sm&ref=REF_2 )

The user auth will be as part of next article with shell implementation. the actual system pass all query string params to all the MFEs without considering the requirements of each MFE separately.

Conclusion

Applying a well architected Micro Frontend approach is highly similar to the micro services, it starts with curiosity, enthusiasm and rigor but can become complex and introduce the obstacles against the approach primer et goals.

Simplicity a principal to success but putting the right part in the right place and defining the responsibilities and boundaries are the crucial things to achieve a long term decision and design.

In this part of series, some simple and straightforward parts was discovered by a focus on representing the overall design goals and possibilities while adopting Micro Frontends.

The next part will deep dive in Shell implementation and all corresponding modules such as Discovery, Templating, Routing, and putting all pieces together for achieving run and build time advantages.

Eventbridge provides a highly available and scalable service bus that covers the Point to point and Fan-Out communication patterns, It integrates with a very wide range of AWS services and also provides a way of communicating with external systems and many well-known partners in the software industry.

Api Destinations

EventBridge API destinations allow you to send events to HTTPS endpoints., this means any public-facing and resolvable domain can be used as a destination to the Event Bridge Api Destination module.

Api destination supports all HTTP verbs except TRACE and CONNECT, so you can use all other verbs like GET, PUT, POST, PATCH, OPTIONS, and DELETE.

To use Api destination a Connection must be configured first, the connection is where we define the authorization mechanism.

The supported auth types:

• OAuth

• Api key

• Basic ( username/password)

The following representation shows my understanding of Api destination ( any feedback will be appreciated )

Rest Api façade: is the entry point of Api Destinations and is triggered by a rule or pipe for any matched event.

Rate Limiter: when creating an API destination and setting an Invocation Rate Limit Per Second value, the API destination explicitly controls that Api Destination incoming throughput per second.

How exactly works rate limit is a big point, and till now i find no article or blog representing it correctly event Serverless Land visuals for invocation rate

Connection: The connection validates the authorization mechanism, the connection validate and prepare the auth config.

I would like just to know if this component is responsible for OAuth call or not, for other types Apikey and Basic just adding a header is sufficient but for OAuth a call is required, i'm curious about that.

Target Invoker: This is the name given by me and i just tried to decompose the whole process. so to be clear here i mean the module calling the target via HTTP.

Overview

The Rest api receives the events over http and is used by Rule or pipe, the following diagram shows the Rule integration with api destination

The rule sends all matched events by calling Api destination synchronously and the rule will be acknowledged by the success of target.

Eventbridge default retry policy reattempts to send the event for a 24h period with a max retry count of 185 times. this way the eventbridge will do the best effort to have a chance for event delivery.

There is a possibility to add a Retry policy to customise the default retry configuration, the retry policy accepts maximum age of events to keep and the number of attempts in case of errors.

This is a good opportunity to avoid loose of events in case the target returns an error or has incapacity for reception of events.

Api Destination as part of request validation verifies the Rate limit if the throughput exceeds the configured rate limit the api destination will throttle the requests as shown in the following diagram.

EventBridge and Rules are the abstract concepts on top of a queueing system.

The event navigation flow follows as demonstrated below

• EventBridge event will be send to the rule

• Rule sends the request to the Api destination via api call

• The api validates the rate limiter status

• The connection manages the auth per configuration

• The target endpoint manage the external api endpoint via HTTP.

Source Code

Rate Limiting

This section is a brief recap of my own understanding after some discussions and research.

When you configure an API destination, you can specify a rate limit, which controls the maximum number of events per second that EventBridge will send to the destination endpoint. This rate limit helps prevent overwhelming the destination with a high volume of events.

• Rate limit is based on token bucket algorithm. The rate limit is represented by the size of a token bucket and the rate at which tokens are replenished. Each event arrival consumes a token.

• When the bucket becomes empty, the Api destination will throttle the requests that involves temporarily delaying or buffering some of the events until tokens become available again.

• This is not yet clear ( No documentation or reference found ), but per tokenization algorithm the tokens are added into the bucket at a fixed rate, corresponding to the specified rate limit.

• EventBridge continuously monitors the rate of incoming events for each API destination. It keeps track of the number of events received per second and compares it against the specified rate limit for that destination.

• A Backoff Mechanism will be applied if the rate of incoming events consistently exceeds the specified rate limit. This means that EventBridge will gradually decrease the rate at which it sends events to the destination in order to alleviate the overload. Once the rate of incoming events decreases and falls below the specified limit, EventBridge will gradually resume sending events at the normal rate.

• If EventBridge encounters errors while attempting to deliver events to the destination due to throttling or other issues, it may retry the delivery according to its retry policy. However, if the errors persist or if the destination consistently fails to handle the events, EventBridge may eventually stop attempting to deliver events to that destination and generate an error or warning notification.

Connection

A connection can be used as a Auth configuration blackbox, this means yous can choose your required auth type but Event Bridge will use a secret manager secret to register those credential securely.

Event bridge Basic and Api Key auth types are simple standards and the population of credentials and Http request header generation is managed.

The OAuth is based on client grant standard that is a standard to obtain credentials outside context of a user. When OAuth configured the Event bridge will communicate with OAuth base service providing the client_id and client_secret to obtain an access_token. As an access_token expires event bridge as per receiving an unauthorised response error being 401 or 407 will use the refresh_token to ask for a new access_token from OAuth server.

The practical usage

For the sake of demonstration, this section use two destinations to see in action how event bridge api destination behaves in action.

The example connection is simple and provides an Apikey auth type that will send the apikey under x-api-key header.

const connection = new Connection(this, Connection.name, {
    authorization: Authorization.apiKey(
        'x-api-key', 
         SecretValue.secretsManager(secret.secretArn)
    )
});

And the Connection will be used with Api destination

Please replace a new webhook.site url by navigating to the https://webhook.site and place it in cdk stack as a replacement for webhooksiteUrl const variable here and also here

const apiDestination = new ApiDestination(this, 'api-destination', {
    httpMethod: HttpMethod.POST,
    endpoint: props.apiUrl!,
    connection: connection,
    rateLimitPerSecond: 1,
});

The repository provides a fake event json file that let send the events to the bus as batch of events

npm run events:send

The api destination has a rate limit of 1 RPS, This means the api destination will receive the event from rule and in case of more than 1 request per second the request will be throttled at the Rule / Api Destination edge and this leads to prevent overwhelming the target.

Looking at the webhook.site and looking in reception time preciously, as per following figure the events are reaching target.

If the throttling errors cause the event to reach the DLQ the message attributes shows the RETRY_ATTEMPTS and ERROR_MESSAGE shows the Api Destination Message indicating the reason of failure including target response payload.

EventBridge guarantees the rate limit control at a 'Best Effort' and does not guarantee the exact, because rates are not globally shared across the fleet and asynchronously propagated. It s important to remind that Event bridge is a distributed service and keeping state consistency without having increased latency can be hard to achieve and that is why the Rate limits are not tracked correctly especially low TPS.

The hard part was imagining different prompts and it was a time-consuming task, so the next step was to simplify my prompting journey, I decided to try a simpler scenario and that was testing conversation. here is an example.

You are the secondary person in a conversation, you have a funny and sympathetic character, Omid is here to have a chat about some random topic, you must consider the logic, and facts but at the same time keep the conversation friendly. don't generate the response on his turn and just answer on your turn keeping responses short and moderate.

Omid says: {}

Here are my last attempts to discover the model. gave it up as this was a lot of thinking about phrasing and not a time-optimised discovery period.

Now the conversation needs to end at some time with a Goodbye phrase, let's say Goodbye.

You: Oh, it was nice talking to you, Omid. Have a great day!

But the conversation can end up with other words as we know the AI is behind let's say this time Stop

I ended up writing a lambda receiving the conversation and looking for keywords to avoid calling LLM, this was lots of if and else but was working. but there were gaps and saying 'I prefer to stop the conversation', the code was sending the request and that was reaching the LLM.

Lots of frustration leaving your goal and typing procedural codes around Yes ,No or Stop and end.

Finally i discovered my two echo devices in living room and YESSSSSS, Since smart devices and connected objects are propagating in our lives, in every residence I visit a kind of connected hub is present, Alexa, Siri, etc. That made me think about how we can benefit from their presence to remove simple tasks, so had a look around my journey and found behind any busy day there is enough tiredness and this has an impact on the way of following my son.

I want to have a simple and fun evening with him

Back in 2021, I had created an Alexa skill to play with the service but this time the idea was going toward no blueprint and a custom skill to have a customised skill that fit better my son's behaviors.

Why not only LLM?

Using GenAI is a fun part of the puzzle but it comes with a cost, the need of added complexity while prompting. Not considering the Alexa vocal and device capabilities, Alexa helps to simplify the conversational flow and creation of simple provisional intentions and manages it for us. With LLM you need to manage when and how to reiterate through the conversation, stop it, or defer it.

Refining the prompt could be simple to handle this challenge but again 'Does it make sens to call Bedrock?', below, example of prompt that could handle the situation.

You are the secondary person in a conversation, you have a funny and sympathetic character, Omid is here to have a chat about some random topic, you must consider the logic, and facts but at the same time keep the conversation friendly, End the conversation if he asks to stop such as stop or you feel any frustration in his response such as you gonna make me crazy, don't generate the response on his turn and just answer on your turn keeping responses short and moderate.

By looking at the default Alexa intents an intent has nothing special on its own and is just a wrapper around slots ( variables ) that can have many predefined words configured, these words in a custom intent can be such as 'run', 'execute', or 'open'. and this is more cost effective to use intents instead of calling Bedrock and being billed by token.

What we build

Building a simple Alexa skill was fun but was hard and time-consuming to list all required intentions and example phrases, there are a large number of situations to consider, and did not seem to me simple and achievable. we benefit from Alexa just for standard intents she provides but for conversation we put every thing to the backend and LLM.

Bedrock: Amazon Bedrock provides a simplified way of interacting with LLM models and this time Mistral 7B seemed to me a good one.

Step Functions: Orchestrating the Prompt generation and interacting with LLM was really fast using StepFunction.

Lambda: Lambda was a mandatory step as Alexa skill supports two kinds of backend endpoints, Http and Lambda trigger.

Alexa: Alexa helps to with minimum effort get into a vocal atmosphere.

Source Code

Alexa Skill

To start working with Alexa skills, the Alexa ASK Toolkit for VsCode gives all the required things to create, download, and interact with Skill.

Next Installing the Alexa Skill Kit cli.

npm install -g ask-cli

The walkthrough needs to create an Amazon Security Profile in the Developer Console, For further testing using the Alexa device use the same email address as the Alexa device account, this will provide the possibility to interact with skills from the real device without public distribution of skill.

After creating the security profile, Configure allowed redirect and allowed origin

Configuring the local is simple and fast, this will be using ask cli, when asked provide the ClientId and Client Secret from Security profile created ( client confirmation is where you put client secret )

ask configure

Proceed with the account linking process when asked if you prefer hosting the Alexa infrastructure on your account, for this article we leave the host of skill infrastructure on the Alexa side.

After redirect a default profile will be created in '<HOME_DIR>/.ask/cli_config'

Deploying the back end

Before deploying the skill manifest is already present as part of the source code the backend must be deployed.

The Mistral LLM model is not available in all regions, for this article i used us-west-2

To deploy the backend run the following command.

npm run cdk:app deploy

Deploy Skill

First we need to copie the SkillFunction Lambda Arn and use it in 'skill.json' file.

{
  "manifest": {
    "apis": {
      "custom": {
        "endpoint": {
          "uri": "arn:aws:lambda:us-west-2:11111111111:function:ConversationStack-SkillFunct-skillfunctionB016215E-dYmSe81VVXgY"
        },
        ....
      }
    },
    ....
  }
}

The Alexa assets are part of the article Source code under 'src/skill/WANTIT' path. to deploy the skill go to the path and run the following command

cd src/skill/WANTIT
ask deploy

The Skill will be Launched by 'Alexa open daddy omid' or 'Alexa ask daddy omid, ..........' , and skill will sent all remaining part of instruction to the backend. this help me to have the whole demand of my son in backend.

Deep dive in Backend

The entry-point of our backend is the lambda function listening to skill request. the lambda contains multiple request handler as

• LaunchRequestHandler (Alexa default)

• HelpIntentHandler (Alexa default)

• CancelAndStopIntentHandler (Alexa default)

• SessionEndedRequestHandler (Alexa default)

• YesIntent (Alexa default)

• NoIntent (Alexa default)

• AskWantItIntentHandler ( Custom )

The following snippet represents the AskWantItIntentHandler implementation

const AskWantItIntentHandler : RequestHandler = {
  canHandle(handlerInput : HandlerInput) : boolean {
    const request = handlerInput.requestEnvelope.request;  
    return request.type === 'IntentRequest' && request.intent.name === process.env.SKILL_NAME;
  },
  async handle(handlerInput : HandlerInput) : Promise<Response> {
    const item = { ... };
    const sfnresponse = await sfnClient.send(new StartSyncExecutionCommand({ ... }));

    const output = JSON.parse(sfnresponse.output ?? '{}');

    let speechText = JSON.parse(sfnresponse.output ?? '{}')?.Body?.outputs?.[0]?.text;

    return handlerInput.responseBuilder
      .speak(speechText)
      .reprompt('Are ok with that?')
      .withSimpleCard('You will get it.', speechText)
      .getResponse();
  },
};

As Alexa needs a direct response, the lambda send a StartSyncExecutionCommand and wait for the State Machine response.

The StartSyncExecutionCommand is only supported by express workflows and not standard*.*

The state machine definition is a simple workflow as illustrated below:

it simply retrieves the prompt from s3 bucket , prepare format by injecting the conversation into it, and finally invoke Mistral model.

Looking at statemachin executions the input received will be as bellow.

{
  "input": {
    "Id": "amzn1.echo-api.session.981ddf27-8725-438e-8022-eb6970bc769a",
    "timestamp": "2024-03-17T18:11:13.796Z",
    "message": "can I have a cup of coffee"
  },
  "inputDetails": {
    "truncated": false
  },
  "roleArn": "arn:aws:iam::11111111111111:role/ConversationStack-Convers-ConversationStateMachineR-cFO5YRWKRD91"
}

Testing

At this level we have two ways of testing, 1st in the Alexa skill developer console, which is nice during the development phase, the 2nd will be testing with a real Alexa device.

You can publish the skill publicly this way anyone can test the skill from a device, there is a possibility to use a user ID or device ID as part of policy conditions in Lambda Role to refuse any unintended use. I'm sure no one will distribute the skill in proof phase publicly. Alexa has a really interesting functionality that lets you use your device to trigger the skill, the only condition is connecting to the device with the same email address as the developer console account, this way the device easily discovers the skill and lets you interact with the skill as a real user.

Alexa skills are localized, you need to select a language and local like English(US), For people like me located in other regions ( France for me ), the skill will show an error indicating that you can not interact with the skill. a workaround will be changing the Kindle account address on your local Amazon site ( exp. amazon.fr ), this will create an account on amazon.com and make it possible to test the skill. You can later replace your Kindle account on amazon.com with your local site ( amazon.fr )

The following video shows the result with real device.

Next steps

As part of this article, due to the miss of clear documentation in AWS Docs there were difficulties in automating the skill deployment as part of CDK stack. The Alexa documentation refers to CloudFormation Docs (here) and CloudFormation Refers to Alexa Docs (here). That is why this article uses Alexa skill kit cli to deploy in this article.

As part of the next step, I will find a way to deploy the skill via CDK, also some demands can be based on

To use the CloudFormation AWS::ASK::Skill resource you need to provide the ClientId, Secret, and Refresh token, apart from the required sensitive information that seems a bit insecure, we need to use the Alexa skill kit cli to retrieve the refresh token, the cli command follows the allowed origin and allowed redirect (as mentioned above) in Alexa Skill section which is not documented well.

Improving the prompt to take into account other situations like time in conversation, an apple can be ok at 16:00 but not at 12:00 as it‘ ’s launch time.

A last part that I would like to try will be if the Alexa works with Response streaming, so I’ll give it a try.

Queue offering

The queuing services are used to decouple the software systems by their ability to act as an asynchronous transitional service to send messages from one system to another and let the producers and consumers be decoupled and independent.

In a queueing system, the producers write the messages to the queue, and the consumers fetch the messages from the queue process, and the message gets removed when no longer needed. by default, the queuing services keep the message as long as there is no removal of demand from consumers.

This design allows the producers and consumers to be independent and decoupled in terms of processing capacity, and manage better their internal state in an isolated way such as managing eventual failures, scaling, etc.

Event vs Job Consumer

Amazon SQS like any queuing service offers the basic functionality of message retrieval by consumers, this is often done by some sort of batch jobs or instances that periodically ask for messages. Any consumer can ask for removal at the end of processing. On the other hand, event-based consumption is more of a serverless concept that offers to send messages to the consumers per message availability, Amazon SQS being a serverless messaging service does not offer event-based message distribution unlike Amazon SNS and works with a job consumer approach.

Source Code

AWS Lambda / SQS

AWS lambda integrates with SQS and this integration offers smooth event-based consumption but this impression is by the excellent way that AWS lambda manages this integration. How it works behind the scenes is that the Lambda service asks for a batch of visible messages in the queue and will manage the consumption and message lifecycle on its own, for sure the principal management of messages inside the queue is under SQS ownership like visibility timeout, delays, etc.

The lambda poller receives the messages from SQS, giving the desired maximum number of messages and the maximum wait time to let the messages be gathered.

Consequently, in lambda event source mapping, the filtering will be applied followed by a process of batching to prepare the batch of records per function configuration before invoking the function.

Filtering

As the following diagram illustrates, the filtering will be applied on the Lambda service side, and if the configured filtering can be applied the record will be considered to be passed to the lambda function. But in the case that the configured filtering can not be applied to the record what happens is that the lambda service discards the message to be processed but also considers it as a message to be deleted and it will remove that message from SQS.

If you are a fan of reading documentation like me you already know this, but I had a mental challenge to see how the SQS would behave if I used the SQS as a central queue in my system, and my reason was that in some central part of the system we can offload the events to multiple lambda consumers.

Logically the idea was that if I applied the filtering as each lambda listens to a different event each message could have a single consumer and this would not be against the recommendation and that helps me have central control of my events without having a monolith function to handle a significant amount of process or doing some sort of lambda based orchestration. The following diagram demonstrates the exact scenario.

As part of my tests, I sent 3 different message with different payloads

{
  "isProductTransitEvent": "true",
  "productId": "123456",
  "deliveryId": "1"
}

{
  "isProductSynchroEvent": "true",
  "productId": "123456",
  "lotId": "12345"
}

{
  "isProductStockEvent": "true",
  "productId": "123456",
  "quantity": 10
}

Only the first function received the messages, but sending more messages for 3rd payload results in randomly receiving messages but not all.

Another hypothesis was sending a single payload with different values for filtered fields, this time none of the second and third functions received their messages per filtering. trying to send 20 messages resulted in the same behavior as we experienced previously.

Lambda integration with SQS using Event source mapping at a high level can be presented as illustrated in the following sequence diagram, and the discarded events will be treated the same as successful messages.

The results ensure that the first trigger receives most of the time a large part of events, but what about if the records fail? what happens to the discarded events being part of the same batch of messages passed through the first trigger?

The answer is simple, In case of failure the discarded messages are deleted like in the success scenario, and the failed messages are retried, but the failed messages also after experiencing the error become visible after visibility timeout and will be reflected for another time, this leads potentially to loss the message in retry phase. as per the tests I did for this article, I never experienced a second retry for 10 messages.

Hope this can be useful ;)

However, in straightforward software systems, the notion of security can be distinct into two phases: Authentication and Authorization. Authentication facilitates the identification of users seeking interaction with the available services. At the same time, Authorization determines granular access permissions, enabling the system to either permit or deny access to the resources owned by the user.

At a high level, this diagram illustrates the functioning of Authentication (AuthN) and Authorization (AuthZ)

This diagram illustrates the public entry points to the internal ecosystem: the Web App, Mobile App, and Gateway. The initial interaction step involves authentication or signing in for users. Users can be real individuals or machines seeking access to services within the internal ecosystem. In the case of mobile or web apps, users typically sign in using account credentials, employing a username and password mechanism to obtain an access token. Gateways utilize credential grant standards such as authorization code, implicit, client credentials, or resource password (defined by IETF Authorization grant).

The diagram below illustrates the flow of the Authorization Protocol Flow.

Json Web Token ( JWT )

The JWT is often utilized as a security measure to fulfill security needs. However, for the sake of clarity, let's assume that 'JWT is not inherently Secure'.

The JWT consists of three main parts: the header, payload, and signature.

• Header: Provides information about the token, including the algorithm used.

• Payload: Contains the claims, such as email, expiration time, and user unique identifier (sub).

• Signature: Utilizes a private key owned by the server to ensure the token's

  integrity and authenticity.

JWT Attacks:

• Signature Stripping: JWTs are signed and contain a signature as the third part of the token. Attackers attempt to recreate an unsigned token to gain unauthorized access. To counter this, the authorization server must validate the token signature as part of the validation process.

• CSRF (Cross-Site Request Forgery): Attackers obtain a signed-in user token and attempt to submit requests to the server from another site. To mitigate this attack, it is advisable to avoid using persisted tokens like cookies unnecessarily. If session persistence is necessary, using short-lived tokens can help. Additionally, incorporating extra meta information such as a unique header in requests, generated previously by the server, adds an extra layer of security.

• XSS (Cross-Site Scripting): This occurs when injected scripts reside in the browser and attempt to exploit and steal tokens from some legitimate storage, Often injecting using query strings or textboxes and sending requests using the user's logged-in session cookies or local storage. To mitigate XSS attacks, validating and sanitizing the received data on the server side is essential.

Learn about JWT and all related types like JWK, JWE here

OAuth and OIDC

Historically, OAuth 1 was introduced to apply a security mechanism to give access to parties for using resources on behalf of the resource owner without credential sharing, OAuth2 replaces the first protocol version and tries to safeguard server-side resources, either on behalf of the owner by initiating an access approval workflow or allowing the third party software gain access on behalf of the owner. However, a significant issue with OAuth was the sole responsibility of authorization and access control being with the authorization server. This resulted in a lack of fine-grained control on the application side, as the access token did not furnish adequate information for software to validate resource access and ownership.

OIDC, serving as an extended layer, sought to address this deficiency by introducing the ID token. Generated by the server, the ID token provides software with additional user information and metadata, thereby enhancing control and insight into user identities.

This is just a small part of history, learn more in this ebook by Okta: here

Amazon Cognito

Amazon Cognito stands out as an Identity and Access Management service offered by AWS, allowing you to incorporate a managed layer of security into your software. Amazon Cognito User Pools serve as the cornerstone for managing application-level security and adhering to the OAuth standard.

A user pool serves as a repository for application users (e.g., “eidivandi@live.com” as a user), with pricing based on Monthly Active Users (MAU). Within a user pool, one or more App Clients can be established. An app client represents an isolated client integration, not only ensuring application isolation but also managing Authentication (AuthN) and Authorization (AuthZ) flow isolation.

Cognito also introduces the concept of triggers, where specific actions within Cognito can invoke custom Lambda code, enhancing its extensibility. The diagram below outlines key actions and their corresponding triggers for any given action.

What we gonna build

In this article, our focus is on configuring an authorization server and implementing the impersonation feature to enable access to sub-accounts or customer resources for an already logged-in user. We'll walk through the setup process, following the structure outlined in the accompanying diagram.

The User authentication will be based on Email/Password credentials, allowing establishing a user session, and the impersonation will allow access to multiple tenant resources. This example will delve into the Password authentication flow and Custom authentication flow, demonstrating how they can work together synergistically. Additionally, it aims to offer insights into understanding Custom Authentication flows within Amazon Cognito more effectively.

Source Code

This article source code can be found on GitHub in the following link.

User Pool and App Client

Establishing a user pool and app client is straightforward, especially with the assistance of AWS CDK. The advantage of leveraging Cognito user pools lies in their simplicity for standard authentication and authorization procedures.

Cognito user pools offer various authentication flows, including Password, SRP, Admin Password, and Custom. In this article, we will use both the password flow for the Sign-In process and custom flow for impersonation and accessing resources on behalf of a tenant.

Below is a snippet demonstrating the CDK code to create the User pool, along with both password and custom flow based user pool clients:

    this.userPool = new UserPool(this, 'MultiTenantUserPool', {
      removalPolicy: RemovalPolicy.DESTROY, 
      accountRecovery: AccountRecovery.NONE,
      email: UserPoolEmail.withCognito('eidivandi@live.com'),
      selfSignUpEnabled: false,
      signInAliases: { email: true },
      autoVerify: { email: true },
      lambdaTriggers:{
        defineAuthChallenge: props.triggers.defineAuthChallenge,
        preTokenGeneration: props.triggers.preTokenGeneration,
        verifyAuthChallengeResponse: props.triggers.verifyAuthChallengeResponse,
      },
      customAttributes: {
        tenants: new StringAttribute({ mutable: true }),
      }
    });

    this.passwordAuthClient =  new UserPoolClient(this, 'MultiTenantPasswordAuthClient', {
      userPool: this.userPool,
      generateSecret: false,
      idTokenValidity: Duration.minutes(5),
      accessTokenValidity: Duration.minutes(5),
      refreshTokenValidity: Duration.days(1),
      authFlows: { userPassword: true }
    });

    this.secureAuthClient = new UserPoolClient(this, 'MultiTenantSecureAuthClient', {
      userPool: this.userPool,
      generateSecret: false,
      accessTokenValidity: Duration.minutes(5),
      refreshTokenValidity: Duration.hours(1),
      idTokenValidity: Duration.minutes(5),
      authFlows: { custom: true }
    });

The UserPool is a source of authentication process and the users source of trust. The two UserPoolClient are the isolated boundaries for the different required auth flows. Having two separate UserPoolClient is just a preference but a single app client can manage different auth flows.

authFlows: { 
    userPassword: true,
    custom: true
}

Sign Up / Sign In

The sign up is managed by a lambda function using AdminCreateUser command. The function creates a user by generating a temporary password and user attributes including some default and reserved ones but also a custom attribute presenting a list of allowed tenants this user can interact with ( use of custom attribute is for article simplicity and can be done using any other type of storage). The function also forces the verification of email, this is done just for the sake of simplicity and to avoid changing the password behind the first signin while testing.

 const UserAttributes = [
      { Name: 'family_name', Value: lastName },
      { Name: 'given_name', Value: firstName },
      { Name: 'email', Value: email },
      { Name: 'email_verified', Value: 'true' },
      { Name: 'name', Value: `${lastName} ${firstName}` },
      { Name: 'custom:tenants', Value: tenants?.join(',') }
    ]

    const adminCreateUserParams = {
      UserPoolId: process.env.COGNITO_USER_POOL_ID,
      Username: email,
      TemporaryPassword: generator.generate({
        length: 10,
        numbers: true,
        symbols: true,
        strict: true,
        exclude: '&%#?+:/;',
      }),
      DesiredDeliveryMediums: ['EMAIL'],
      UserAttributes,
      ClientMetadata: {
        step: 'SignUp_CreateUser',
      }
    } satisfies AdminCreateUserCommandInput;

After signing up, an email will be sent with the temporary password, we use this email to signins, the signin function will use InitiateAuth command to authenticate and force the change password challenge by keeping the temporary password ( This step is just for demo and often in production the user will change the password, so the NEW-PASSWORD_REQUIRED challenge will be used to force the user to change the password )

 const signInParams: InitiateAuthCommandInput = {
      AuthFlow: AuthFlowType.USER_PASSWORD_AUTH,
      ClientId: process.env.COGNITO_USER_POOL_CLIENT_ID,
      AuthParameters: {
        USERNAME: username,
        PASSWORD: password,
      },
      ClientMetadata: {
        step: 'Signin_InitAuth'
      }
    };

const signinResponse = await client.send(new InitiateAuthCommand(signInParams));

The InitiateAuth response has a session that must be used to trigger the challenge as following snippet demonstrates.

 if( signinResponse.ChallengeName === 'NEW_PASSWORD_REQUIRED' ) {
      const challengeParams: RespondToAuthChallengeCommandInput = {
        ChallengeName: 'NEW_PASSWORD_REQUIRED',
        ClientId: process.env.COGNITO_USER_POOL_CLIENT_ID,
        ChallengeResponses: {
          USERNAME: username,
          NEW_PASSWORD: password,
        },
        Session: signinResponse.Session,
        ClientMetadata: {
          step: 'Signin_Respond_Challenge'
        }
      };

      const challengeResponse = await client.send(new RespondToAuthChallengeCommand(challengeParams));
      authenticationResult = challengeResponse.AuthenticationResult;
    }

The challenge result will include the IdToken, AccessToken and RefreshToken and session that will be returned as the response.

Impersonation

The impersonation step is a custom Auth process including to step process, InitiateAuth and RespondToAuthChallenge command. before looking at how that behaves it is important to see how this custom flow works.

The user pool has 3 lambda trigger:

• DefineAuthChallenge

• VerifyAuthChallenge

• PreTokenGeneration

Triggering the challenge can be done as per AWS Documentation here , when using a Custom auth flow, both the InitiateAuth and RespondToChallenge will trigger the DefineAuthChallenge with VerifyAuthChallengeResponse triggered between two invocations after the challenge verification passes and if the second DefineAuthChallenge invocation response indicates the generation of token the PreTokenGenration trigger will be invoked.

const initiateAuthCommandInput = new InitiateAuthCommand({
      AuthFlow: AuthFlowType.CUSTOM_AUTH,
      ClientId: clientId,
      AuthParameters: {
        USERNAME: email,
      },
      ClientMetadata: {
        step: 'Impersonation_InitAuth',
        tenant,
      }
    });
    const response = await client.send(initiateAuthCommandInput);

    const challengeCommandInput = new RespondToAuthChallengeCommand({
      ChallengeName: 'CUSTOM_CHALLENGE',
      ClientId: clientId,
      ChallengeResponses: {
        USERNAME: response.ChallengeParameters?.USERNAME || '',
        ANSWER: 'impersonation',
      },
      ClientMetadata: {
        authFlow: 'impersonation',
        step: 'Impersonation_RespondToChallenge',
        tenant
      },
      Session: response.Session,
    });
    const challengeResponses = await client.send(challengeCommandInput);

Initiate Auth : DefineAuthChallenge

In the first step the DefineAuthChallenge trigger will receive an invocation with an event payload including the user attributes and an empty array of sessions. the sessions array represents the previous steps in auth challenge, for this first invocation the array is empty.This first invocation is triggered by InitiateAuth command.

{
    ...
    "triggerSource": "DefineAuthChallenge_Authentication",
    "request": {
        "userAttributes": {
            "sub": "e29524e4-f0a1-7018-7650-4bee0ca85f4b",
            "email_verified": "true",
            "cognito:user_status": "CONFIRMED",
            "name": "Hills Samir",
            "given_name": "Samir",
            "custom:tenants": "CUS-01,CUS-02",
            "family_name": "Hills",
            "email": "eidivandi@live.com"
        },
        "session": []
   },
   ...
}

The Lambda Function will orchestrates this steps as below and change the response elements.

  if ( event.request.session.length === 0 ) {
         event.response.issueTokens = false;
         event.response.failAuthentication = false;
         event.response.challengeName = 'IMPERSONATE_CHALLENGE';
  } else if (...) {
      ....
  } else {
    event.response.issueTokens = false;
    event.response.failAuthentication = true;
  }
  return event;

Respond Auth Challenge: VerifyAuthChallenge

The RespondToAuthChallenge Command will initiate the rest of flow, and as first step to verify the previously initiated challenge by invoking the VerifyAuthChallenge trigger, this is where the validation verifies if the ChallengeAnswer corresponds to the response provided. As part of the validation, the verification of tenant parameters with the user's custom tenants attributes validates if interacting with this tenant resources is authorized for this user.

 if (event.request.clientMetadata?.tenant &&
      event.request.userAttributes?.['custom:tenants']?.split(',')?.includes(event.request.clientMetadata?.tenant)
  ) {
      event.response.answerCorrect = event.request.challengeAnswer === 'impersonation';
  }
  return event;

Respond Auth Challenge: DefineAuthChallenge

In the next step the DefineAuthChallenge will be invoked for a second time but the event will be partially different, including more information about previous session initiated and some metadata.

{
   ...
   "triggerSource": "DefineAuthChallenge_Authentication",
   "request": {
        "userAttributes": {
            "sub": "e29524e4-f0a1-7018-7650-4bee0ca85f4b",
            "email_verified": "true",
            "cognito:user_status": "CONFIRMED",
            "name": "Hills Samir",
            "given_name": "Samir",
            "custom:tenants": "CUS-01,CUS-02",
            "family_name": "Hills",
            "email": "eidivandi@live.com"
        },
        "session": [
            {
               "challengeName": "CUSTOM_CHALLENGE",
               "challengeResult": true,
               "challengeMetadata": null
            }
        ],
        "clientMetadata": {
            "step": "Impersonation_RespondToChallenge",
            "authFlow": "impersonation",
            "tenant": "CUS-01"
        }
   },
   ...
}

Here the ClientMetadata is the only available option to pass the information between different triggers in the same challenge. this helps to share an ephemeral state between invocations. The function handler code will verify the session length and the challenge result has a truthy value.

  if ( event.request.session.length === 0 ) {
    ...
  } else if (
    event.request.session.length === 1 &&
    event.request.session[0].challengeResult === true
  ) {
       event.response.issueTokens = true;
       event.response.failAuthentication = false;
  } else {
    ...
  }
  return event;

As part of the event response, the issueTokens and failAuthentication indicate if a token can be generated or not, or even failing the authentication.

Respond Auth Challenge: PreTokenGeneration

The next step will be the PreTokenGeneration to apply customization of token claims. The default token customization can be applied only to the idToken but advanced security options if activated will give the possibility of access token customization. ( Advanced Security Options applies extra cost )

  let tenant;
  if( 
    event.triggerSource == 'TokenGeneration_Authentication' &&
    event.request.clientMetadata?.step == 'Impersonation_RespondToChallenge'
  ) {
    tenant = event.request.clientMetadata?.tenant;
  };

  event.response = {
    claimsOverrideDetails: {
      claimsToAddOrOverride: {
        tenant
      },
      claimsToSuppress: [],
      groupOverrideDetails: {
        groupsToOverride: [],
        iamRolesToOverride: [],
        preferredRole: "",
      },
    }
  };
  return event;

The function verifies the trigger source and custom metadata provided in impersonation handler and injects the tenant claim in the token.

Authorizer

As per the diagram shown in ‘What we gonna build’ section, the solution needs to have two layers of authorization. The first one is to validate the user sessions and authorize them to access protected endpoints, and the second one is to validate and authorize the impersonation token when trying to access the downstream services.

User Session

The first Authorizer is a CognitoAuthorizer attached to protected endpoints like impersonate endpoint. This can be achieved using aws CDK as below

const cognitoPasswordAuthorizer = new HttpUserPoolAuthorizer('CognitoPasswordUserPoolAuthorizer', props.cognito.userPool, {
   userPoolClients: [ props.cognito.passwordAuthClient ]
});

...

api.addRoutes({
    path: '/user/impersonate',
    methods: [ HttpMethod.POST ],
    integration: new HttpLambdaIntegration('ImpersonassionAuthFunctionIntegration', impersonassionAuthFunction),
    authorizer: cognitoPasswordAuthorizer
});

This authorizer will validate the signin token generated behind email and password authentication, to prevent unauthorized access to impersonation endpoint.

Impersonation

The impersonation authorier is a Lambda CustomAuthorizer integrating with ApiGateway and validates that the authorization token received is allowed to access the tenant resources. For this article's simplicity we use a single apigatway shared between downstream and authorization service but add a Lambda Authorizer attached at the route level for the downstream endpoint.

    const customAuthorizer =  new LambdaFunction(this, 'CustomAuthorizer', {
      entry: resolve(join(__dirname, '../../src/authorizer/handler.ts')),
      bundling: {
        banner: `import { createRequire } from 'module';const require = createRequire(import.meta.url);`,
      },
      environment: {
        COGNITO_USER_POOL_CLIENT_ID: props.cognito.secureAuthClient.userPoolClientId,
        COGNITO_USER_POOL_ID: props.cognito.userPool.userPoolId,
        TABLE_NAME: props.table.tableName
      }
    });

    const cognitoImpersonationAuthorizer = new HttpLambdaAuthorizer('CognitoImpersonationAuthorizer', customAuthorizer , {});

The authorizer validates the Token against cognito userpool and client app.

    const token = event.authorizationToken!.replace('Bearer ', '');
    const decodedToken = decodeAndGetToken(token);
    const user = await verifyToken(token, decodedToken.token_use, process.env.COGNITO_USER_POOL_CLIENT_ID!);

The verifyToken method uses the 'aws-jwt-verify' to validate the token using cognito userpool client.

  const verifierParams = {
    userPoolId: process.env.COGNITO_USER_POOL_ID!,
    tokenUse: use as CognitoVerifyProperties['tokenUse'],
    clientId: process.env.COGNITO_USER_POOL_CLIENT_ID!,
  };
  const verifier = CognitoJwtVerifier.create(verifierParams);

  return verifier.verify(token, verifierParams);

The authorizer returns a policy allowing or denying access to invoke api gateway

return {
      principalId: 'user',
      policyDocument: {
          Version: '2012-10-17',
          Statement: [
              {
                  Action: 'execute-api:Invoke',
                  Effect: 'Allow',
                  Resource: event.methodArn,
              },
          ],
      },
  };

The authorizer simply validates the sanity of the token and if it was originally owned by authorization server, but the fact of allowing that token to access tenant resources must be under the responsibility of downstream.

Let’s say, that omid@gmail.com signed in and asked to impersonate the tenant CUS-01 now the generated token will be validated even if the requested resource to downstream is owned by CUS-02, this is the responsibility of the downstream service to validate if the authorized token corresponds well the CUS-02 tenant or not.

In the following section, we will see this in practice and deploy the sample solution.

Running the authorization server

The source code provides a brief README to follow and deploy the solution, and a postman collection and environment are also provided to simplify the testing purpose.

To start and test the solution lets start by signing up, Amazon Cognito will send an email containing a temporary password.

After account creation, an email will be received containing the temporary password

Using this password the sign-in endpoint allows to establish a session using the password auth flow. Passing the received IdToken from the sign-in step as an authorization header lets the cognito authorizer identify the session and validate the access to impersonate api route.

The Impersonation token will be generated by custom flow and must be passed along the call to the downstream, the custom lambda authorizer will validate the generated token, the responsibility of validating the resource is by downstream service to check the asked resource and validate it against the authorizer context provided.

In this example the downstream verifies the requested resource against authorization token context.

  try {
    const eventBody = JSON.parse(event.body || '{}');
    if( eventBody.tenant !== event.requestContext.authorizer?.lambda.tenant ) {
      throw new Error('Tenant does not match');
    }
    return ActionResults.Success(eventBody);
  } catch (e) {
    console.error(e);
    return ActionResults.InternalServerError({ message: e.message || e.name });
  }

Conclusion

Security being part of our daily software development is not always simple such as providing an Api key or a JWT token, and when it gets complicated, this is important to think about shared responsibility. Understanding the Security foundation like AuthN, AuthZ and different components such as Client, Authorization server, and Resource Server will help to better give responsibility and scopes to each part of the communication flow.

Applying standards is not always free of effort and needs a higher level of deep understanding so looking at some resources like IETF helps to achieve a deeper vision and perspective.

Here some resources:

• JWT (rfc7519) : https://datatracker.ietf.org/doc/html/rfc7519#page-9

• OAuth2 (rfc6749) : https://datatracker.ietf.org/doc/html/rfc6749#page-7

Hope this article will be useful.

I have been using EventCatalog for 3 years now, but this time I tried to use the EventCatalog to simplify the service documentation transparently and automatically, and I had some deep thinking moments about how I can make it as suitable as it can be based on my own needs.

Here, the way i would like to manage the cataloging is as a central engine as illustrated below.

However, the way that EventCatalog was designed was about a central source of trust, and this was not the way I was looking for a cataloging solution as I have preferred always to keep the documentation as part of the service source control system and in the same repository service resides. This is the operational model of EventCatalog and seems OK but the enterprise ecosystem often has other constraints during adoption.

The actual operational model of the EventCatalog seems enough standard but I need to consider also the Software development practices. so I decided to stop waiting and make something operational based on our practices and using EventCatalog.

AsyncApi Extension

First, I tried to jump for a PR on EventCatalog AsyncApi Extension Github, but I just had the solution open for 1 week without finding time to start (As I am often in meetings ). EventCatalog AsyncApi plugin works on a generator concept in EventCatalog by letting have some simple config in the default js config file under eventcatalog.config.js name.

const path = require('path');

module.exports = {
  title: 'EventCatalog',
  tagline: 'Discover, Explore and Document your Event Driven Architectures',
  organizationName: 'Your Company',
  projectName: 'Event Catalog',
  ...
  generators: [
    [
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        pathToSpec: [
          path.join(__dirname, '../specs/Order/placement/1.0.0', 'asyncapi.yaml'),
          path.join(__dirname, '../specs/Order/Shipment/1.0.0', 'asyncapi.yaml')
        ],
        versionEvents: false,
        renderNodeGraph: true,
        renderMermaidDiagram: true,
        domainName: 'Order'
      },
    ],
    [
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        pathToSpec: [
          path.join(__dirname, '../specs/Product/stock/1.0.0', 'asyncapi.yaml')
        ],
        versionEvents: false,
        renderNodeGraph: true,
        renderMermaidDiagram: true,
        domainName: 'Product'
      },
    ],
  ]
}

Having the generators, the npm run generate command will read all specifications and generate the respective Domain, Events, and Services under domains folder.

Complexity

We can start asking any service owner team to do a Pull Request when new services are created or changes are necessary but this approach is against our learnings from the past when dealing with software that was managed by multiple teams, in the best case, this can work at some scale, but can become frustrating in the worst case.

Another challenge I encountered was, how big the EventCatalog config file can become when there are already hundreds of services in around 10 domains without counting internal modules communicating in an event-driven way, and this will be just a nightmare to live with.

Listing Desired State

Finally with all the blocking points and having a list of possibilities that the actual state of EventCatalog does not cover there was no starting point without listing what I need.

• Specification under the service ownership.

• No effort more than having updated specifications in the service repository.

• Each service is responsible for formatting and validating specifications.

• Specifications must follow the governance conventions.

• A catalog to represent all events and service specifications.

• The catalog must be in sync with the real service specification.

• The catalog integration must be automated and autonomously.

• Catalog must be reproduced rapidly.

Catalog Design

To define the catalog design, the considerations were as below

• For simplicity Static and bundled

• Respond to changes based on events

• Decoupled and without adding hard dependencies

• Autonomous

The final design is represented in the following diagram

Source Code

Build Configuration

It was simple to build the configuration dynamically by just adding a nodejs script as explained below.

The source code is available here on GitHub

The config generate is a simple script that does the following steps:

• Get all yaml files in specs folder

• Feed the generators array by creating a generator element per specification

• Merge the default config and generators

• Write the results and rewrite the eventcatalog.config.js if presents

const baseConfig = { ... }
const generatorDefaultConfig = { ... }
const createGenerators = (specsFolder = 'specs') => {
  const schemas = getDirectories(specsFolder)?.
    sort((a, b) => b.localeCompare(a) ).
    reverse().
    filter((fileName) => fileName.includes('.yaml'));

  if (!schemas) return [];

  let asyncApiGenerators = [];
  schemas.map((schemaName) => {
    asyncApiGenerators.push([
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        ...generatorDefaultConfig,
        domainName: schemaName.split('/')[2],
        pathToSpec: [ path.join(__dirname, `${schemaName}`) ]
      },
    ]);
  });

  return asyncApiGenerators
}

const generators = createGenerators('../specs');

fs.writeFileSync('./eventcatalog.config.js', 
  `module.exports = ${JSON.stringify({
    ...defaultConfig,
    generators
  }, null, 2)}`, 'utf8');

Running the script can be done by adding a new script in package.json that runs the node ./config.generator.js. The package json scripts section will be represented as below.

 {
  ...
  "scripts": {
    "start": "eventcatalog start",
    "dev": "eventcatalog dev",
    "build": "eventcatalog build",
    "pregenerate": "node ./config.generator.js",
    "generate": "eventcatalog generate"
  },

By prefixing the script name by pre the npm will run automatically the pregenerate script when we run npm run generate, this is handy to keep the cicd scripts un-touched and extending the capabilities.

Specifications

About Specifications there was two considerations, first each service takes ownership of its documentation and specification, second the catalog must be autonomous. by these facts i ended up 3 correlated notes.

• EventCatalog project available options be used, so the project will continue to use a local based spec source for generation to avoid any complicated personalisation in catalog project repository or in CICD pipeline.

    version: 0.2
    env:
      parameter-store:
        SPEC_BUCKET_NAME: /eventcatalog/bucket/specs/name
    phases:
      install:
        commands:
          - echo Installing dependencies...
          - npm cache clean --force
          - cd catalog && npm install --froce && cd ..
      pre_build:
        commands:
          - echo "Pre build command"
          - rm -rf specs
          - mkdir specs
          - aws s3 sync s3://$SPEC_BUCKET_NAME/ specs
      build:
        commands:
          - cd catalog
          - npm run generate
          - npm run build
    artifacts:
      files:
        - '**/*'
      base-directory: catalog/out
  

• The Local source of catalog must be updated and synced with all services by in a decoupled manner. using a GitHub action to sync any repository with catalog s3 on a change in the specs folder.

    name: AsyncApi Spec Sync
    on:
      push:
        branches: [ main ]
        paths: 
          - 'specs/**'
    env:
      AWS_REGION: eu-west-1
    jobs:
      sync-spec-to-s3:
        runs-on: ubuntu-latest
        steps:
        - name: Checkout
          uses: actions/checkout@v4
          with:
            fetch-depth: 0
        - name: Configure AWS Credentials
          uses: aws-actions/configure-aws-credentials@master
          with:
            aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
            aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            aws-region: ${{ env.AWS_REGION }}
        - name: Sync spec to S3
          run: |
            aws s3 sync ./spec s3://$(aws ssm get-parameter --name "/eventcatalog/bucket/specs/name" | jq -r '.Parameter.Value')/
  

• Each Service register documentation in a conventional way, being a root level specs folder, subfolder named by domain, and includes a folder representing the version inside domain.

  

Service integration

it will be simple by copying and pasting the workflow in all services, and after any changes in specs folder the specs folder will be synced by catalog specs s3 bucket,

The S3 changes will trigger the CodePipeline using EventBridge default bus. the pipeline will process the catalog and spec sync, the following diagram demonstrates the actual code pipeline workflow

Conclusion

This approach was one of the fastest and simpler approach to have a result with no complexity. but again i m looking to tackle some pain points to make EventCatalog more suitable.

I could add a wrapper around EventCatalog but i found this approach toward a wrong direction, the reason is that leads to extra complexity for a single place usage in the whole enterprise and hides a lot of intention of EventCatalog existence.

The last point for me to tackle is how to automate the Markdown consumers section fill up. as this approach focuses on documentation and specifications the gap of consumers was not the priority on my side but this is a thing i actively think of to achieve system landscapes from EventCatalog.

Enjoy Reading

Following the establishment of a standardization process incorporating elements such as Documentation, Versioning, Filtering, and Event Envelopes as detailed in the preceding articles, the next step is to implement these standards. This phase poses the greatest challenge, as effective collaboration and communication among software stakeholders play a crucial role in successful adoption.

Governance

Governance in IT has different definitions based on company culture, size, and the value it brings. For this article, Governance is defined as a means to identify the system's complexity, observe system behavior, and simplify the decision-making process.

When dealing with software, Often the source of knowledge is the development team per their detailed vision of implemented business complexity, software communication process, and potential risks behind any change. This is ok on its own but putting this knowledge in a local place being a team and adding some dependencies to people can become an obstacle when speed becomes an important pillar of success.

Software vs People Communication

Back in 2005-2010, Often software was designed using a monolithic approach, where all complexity was localized in a single place. That approach suffered from software complexity, availability, and scaling problems, But there was less network and communication burden being Software or People.

Distributed systems are a solution but in reality, they resolve monolithic suffering points, but adding new problems by distributing software and knowledge into multiple locations and adding a higher level of communication and alignment problems.

EDA is a distributed communication pattern applied at communication boundaries and resolves some distributed system problems by decoupling. But add some level of difficulties over the traditional distributed systems, by reversing the dependency direction. so any degradation will be identified at the subsequence layers behind producer service and not in preceding layers and this makes things harder to be identified.

Decisions

In a competitive business where competitors move by the fastest change rate to approve ideas and bring unique problem-solving ideas into the products, being able to make faster, cleaner, and more accurate decisions about a problem is important.

To this goal relying on data and having enough information become crucial to achieve rapidity and clarity while making decisions. Having detailed data is hard but an approach to deal with when the business needs to move at scale.

Event Cataloging

One of the useful approaches for observing the internal state of a system is cataloging, in the cataloging approach we gather information about the overall system communication like Documentation, Schemes, Versions, and Consumption.

A Catalog must offer a result set to cover the following details

• Producers - Who does introduce that change?

• Consumers - Who are the interested actors behind that change?

• Event Models - What are the event models transiting?

• Event Versions - What are the active and outdated event versions?

• Service Specifications - What does each producer offer to consumers?

• System Communication: How do the systems communicate together

Cloudevents

Cloud events is a new specification for event-driven design approach and helps to put some standards over how the event model must be done. Cloudevents bring the separation of context and data but keep them correlated.

Event definition

The Cloudevents specification describes an event as:

An "event" is a data record expressing an occurrence and its context.

Events are routed from an event producer(the source) to interested event consumers.

The routing can be performed based on information contained in the event, but an event will not identify a specific routing destination.*

Events will contain two types of information: the Event Data representing the Occurrence and Context metadata providing contextual information about the Occurrence.

The takeaways from the above description are :

• Events express occurrence and context.

• Events are routed so an event transits along many communication hopes before reaching the consumer.

• The routing is done based on event information, so the event must provide enough information to simplify routing and event transition.

• The Context includes some information about the occurrence, so the context represents even-related information.

Event Transition

As the event-driven architecture is a distributed communication pattern, Event transition is one important point to take into consideration. An event can be transited from any starting point and be consumed in one or more places. The event will be routed along many network or software hops, like a broker, enrichment process or aggregation process.

Event Occurrence

An occurrence is a definition of 'What is the change in a process', it is important to well define the occurrence and the principal events in a context but keeping separation between internal state changes and external ones helps to define better and guarantee the distributed communication quality.

Keeping the occurrence internal state helps reduce the distributed communication complexity, by abstracting the internal process from the external process.

Event Envelope

Looking at a simple envelope, there are two levels of information, a letter inside the envelope, representing the detailed intentional information and some information presented on the envelope like an Identifier, Confidentiality stamp, Sender, and Date.

The internal letter will be interesting when the envelope arrives at its destination and will be in hand. but the external information is useful for tracking, distributing, and routing.

Cloudevents Design Goal

Cloudevents specification represents some standards and principles around the event-driven architecture but the initiative like all other standards is to tackle some real problems.

The goal of the CloudEvents specification is to define the interoperability of event systems that allow services to produce or consume events.

CloudEvents are typically used in a distributed system to allow for services to be loosely coupled during development, deployed independently, and later can be connected to create new applications.

The above is from the cloudevents primer design goal documentation, the primer focuses on the following considerations:

• Protocol and Channel agnostic

• Extensibility

• On top of standards

Channel / Protocol Agnostic

Cloudevents standard relies on the principle of being protocol and channel-agnostic, the specification understands well the presence of different protocols and communication channels in real systems and provides some guidelines to approach them.

The Specification provides guidelines for the following protocols:

• AMQP

• MQTT

• NATS

• Websockets

• HTTP

Also, The specification represents some guidelines related to channels to address the adoption of a standard on tom of channel presence. Cloudevents introduces the concept of adapters and provides some practices to overcome channels as a mean of event distribution.

Extensibility

Being a specification related to a distributed event-driven architecture, Cloudevents introduces extensions to overcome communication and process problems by using extensions as a standard way of extensibility on top of Cloudevents.

An extension is a set of context-level attributes that help the adoption of standards or practices to tackle an event-driven design problem. extension must follow the primary data types provided by Cloudevents.

Cloudevents supported types are:

• Binary: Sequence of Bytes RFC4648.

• Integer: Signed 32bit numeric range between -2,147,483,648 to +2,147,483,647 RFC 7159, Section 6

• String: Sequence of allowable Unicode characters

• Boolean: True or False

• Timestamp: RFC 3339.

• URI: Absolute URI RFC 3986 Section 4.3.

• URI-Reference: Relative URI RFC 3986 Section 4.1.

Cloudevents available extensions are:

• Distributed Tracing: The extension goes on top of Open Tracing standard.

• Expirytime: solves the event validity problem

• Sequence: solves the event ordering problem

• Partitioning: Solves the scaling problem by adding the related partition key to events to help brokers and consumers better identify events.

• Dataref: Solves the problem of large event payload by producing the payload file/storage location as part of a smaller event.

• Authcontext: solves the problem of identifying the principal or actor initiating the occurrence.

Putting Cloudevents on AWS

AWS provides a wide range of infrastructure including communication channels that help to distribute the events between different software such as Event Bridge, SQS, SNS, and kinesis. choosing the right service per requirement is an important part of design.

Implementing examples

The provided example represents a distributed event-driven approach for an e-commerce software system.

The example source can be found in the following GitHub repository, Follow the Readme instructions to deploy and test the provided examples

The above design follows a simple basket item approval leading to an order validation and delivery process.

• The basket item approval command reaches the order system.

• The ordering system distributes an event of type order.placed

• The Shipment system starts preparing the packaging

• The product system validates the availability of the product

• The order system listens to product availability

  • If the product is available, distribute an order.confirmed event

  • If the product is not available distribute an order.cancelled event

• The Shipment system sends order.shipped event if received an order.confirmed event

• The Notification system listens to order.shipped events

Order system

The Order System has two modules, The ingestion is responsible for receiving the orders and the product listener is responsible for acting behind any product state change.

The ingestion receives the basket.item-approved event that is an event respecting cloudevent standard envelope, coming over HTTP protocol using ApiGateway.

The integration of ApiGateway and SQS using AWS CDK takes care of considering headers and body and adapting them as a standard event payload. The approach is what cloudevents adapter specification represents.

  private static readonly sqsRequestTemplate = `Action=SendMessage&MessageBody={
    "data" : $util.urlEncode($input.body),
    #foreach($param in $input.params().header.keySet())
    "$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end

    #end
  }`

The order DDB stream lambda sends multiple versions of order.placed to an SNS topic.

Product System

The product system is a subscriber of the order system, listening to order.placed events using a SQS queue. the SQS is configured with RawMessageDelivery at SNS subscription level. this helps to avoid the message being wrapped in an SNS envelope. ( AWS RawMessageDelivery Documentation )

The SNS/SQS subscription adaptation must be done in two steps IAC adapter using RawMessageDelivery and Software adapters to fetch the event out of wrapped event in SQS event model.

    ordersTopic.addSubscription(new SqsSubscription(productsQueue, {
      rawMessageDelivery: true,
      filterPolicyWithMessageBody: {
        source: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'ecommerce.orders.service'
          ],
        })),
        type: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'order.placed'
          ],
        })),
        dataversion: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'v1.0'
          ],
        })),
      }
    }))

The product system listens to v1.0 of order.placed event, so if any new version of order.placed will be introduced, this will not impact or introduce duplicated reception in product system.

The snippet shows the code adapter to retrieve the envelope from SQS event model.

import { EventBridgeEvent, SNSEvent, SQSEvent } from "aws-lambda";
import { EventModel } from "../models/cloud-event";

type EventType = SQSEvent | SNSEvent | EventBridgeEvent<string, any> | EventModel<any, any> | any;
const getEvent = <T,U>(
  event: EventType
  ): EventModel<T,U> | Record<string, any> | null => {
  ...
  if( event.Records[0].eventSource == "aws:sqs" )
    return JSON.parse(event.Records[0].body);
  ...
  return null;
}

export const DeSerialize = <T,U>(
  event: EventType
  ):EventModel<T,U> | Record<string, any> | null => {
  const evt = getEvent<T,U>(event);
  console.log({
    ...evt, 
    recipient: process.env.SOURCE,
  });
  return evt;
}

Shipment System

The shipment system listens to all order.placed, confirmed, and cancelled events, and listens to v2.0 of events.

ordersTopic.addSubscription(new SqsSubscription(productsQueue, {
      rawMessageDelivery: true,
      filterPolicyWithMessageBody: {
        source: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'ecommerce.orders.service'
          ],
        })),
        type: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'order.placed',
            'order.cancelled',
            'order.confirmed'
          ],
        })),
        dataversion: FilterOrPolicy.filter(SubscriptionFilter.stringFilter({
          allowlist: [
            'v2.0'
          ],
        })),
      }
    }))

Notification System

The shipment service sends the events to an event bridge bus, and the Notification system listens to the shipment service using an event bridge rule.

The rule removes extracts $.details representing the Cloudevent respected event from the EventBridge wrapped event payload.

new Rule(this, 'rule', {
        eventBus: shipmentEventBus,
        eventPattern: {
          detail: {
            type: ['order.shipped'],
            source: ['ecommerce.shipment.service'],
            dataversion: ['v1.0']
          }
        },
        targets: [
          new targets.SqsQueue(notificationQueue, {
            deadLetterQueue: dlq,
            message: RuleTargetInput.fromEventPath('$.detail'),
          }),
        ]
    });

Producing Events

The producers of events use a helper method to generate the events, the method will generate an Event Id, Idempotency key , correlation Id, sequence id, and accepts the event type , event payload, version, and causation id as parameters.

export const InitEvent = <TData, TEventType>(
    source: string,
    eventType: TEventType,
    eventData: TData,
    dataVersion: string,
    dataSchema?: string,
    causationId?: string,
    correlationid?: string
     ): EventModel<TData, TEventType> => {

    return {
        idempotencykey: uuidV5(JSON.stringify(eventData), "40781d63-9741-40a6-aa25-c5a35d47abd6"),
        id: nanoid(),
        time: new Date().toISOString(),
        data: eventData,
        type: eventType,
        source,
        dataversion: dataVersion,
        dataschema: dataSchema,
        causationid: causationId,
        correlationid: correlationid ?? nanoid(),
        specversion: "1.0.2",
        sequence: ulid(),
    }
  }

The Idempotency key helps to avoid unintended behavior on the consumer side in case of event duplication.

The Sequence helps to keep track of the ordering of events on the consumer side.

Observing Events

To observe the event production and consumption for simplicity we use cloudwatch service, the goal is to discover how the cloudevent context info is important to be observed.

As the events can reach the lambda service from different services using the adapter concept to extract the event payload is the proposed approach by cloudevents. All lambda handlers in this example use a custom Deserialize helper to extract the Cloudevent model from the infrastructure event.

export const DeSerialize = <T,U>(
  event: EventType
  ):EventModel<T,U> | Record<string, any> | null => {
  const evt = getEvent<T,U>(event);
  console.log({
    ...evt, 
    recipient: process.env.SOURCE,
  });
  return evt;
}

The helper function logs the cloudevent event payload to simplify the observability and data extraction.

Running the example

The source code provides a Postman collection in 'assets' folder under Cloudevents.postman_collection.json name, to run it simply import the collection in postman and change the request URL by the ApiGateway url returned at the end of orders system deployment. or use the following curl command to send an event.

curl --location 'https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/live/sqs' \
--header 'x-api-key: ec1a9e8f-b8fc-4a6d-9069-108775d67af8' \
--header 'causationid: 6e67e1a4-e323-492e-a7ff-a489a54ba63d' \
--header 'source: ecommerce.baskets.service' \
--header 'type: basket.item-approved' \
--header 'id: 872fab6b-4f22-4951-874d-021d68d39154' \
--header 'specversion: 1.0.2' \
--header 'time: 2024-04-06T22:40:33.413Z' \
--header 'dataversion: v1.0' \
--header 'correlationid: 3a02915a-ba3e-4e58-b7c3-642efaa31a1a' \
--header 'Content-Type: application/json' \
--data '{
    "orderDate": "2024-01-01T12:55:00.990Z",
    "price": 1000,
    "quantity": 2,
    "productId": "PRD_12345643",
    "userId": "a5449147-ab45-4bec-a0be-f00daf5f2871"
}'

The process behind this request will place an order but the order will be canceled later because the product availability will not be confirmed by the lack of product in the product system DynamoDB table.

As represented we can observe the event type and version consumption. and see the event transition process. To simulate the process for an available product we can add the following product in the table.

{
  "productId": "PRD_12345643",
  "price": 500,
  "stock": 1,
  "status": "IN_STOCK"
}

Sending a new request will result a full order process, including confirmation and shipment approval.

Also extracting some statistics helps to see the active consumption and for example find outdated event versions and active ones.

Event Discovery

As the example already uses the cloudwatch, to prepare a catalog of events use of lambda extensions can be a solution to achieve event discovery and schema extraction and documentation.

The following design demonstrates how lambda extensions can be used as a sidecar to feed the event discover process.

The extension receives the logs and moves them to a kinesis data stream that results in triggering a lambda function that puts events into an EventBridge custom bus with an enabled discovery option.

The extension subscribes to the lambda telemetry api by registering to the extension api to receive the Invoke and shutdown invocations.

const RUNTIME_EXTENSION_URL = `http://${process.env.AWS_LAMBDA_RUNTIME_API}/2020-01-01/extension`;

await fetch(`${RUNTIME_EXTENSION_URL}/register`, {
   method: 'post',
   body: JSON.stringify({
       'events': [
           'INVOKE',
            'SHUTDOWN'
        ],
   }),
   headers: {
        'Content-Type': 'application/json',
        'Lambda-Extension-Name': basename(__dirname),
   }
});

Also, the extension needs to subscribe to the telemetry api and provide a HTTP listener to allow the telemetry api send the logs to the extension.

Adding the extension to the lambda can be done as shown in the following CDK code.

const orderPlacedFunction = new NodejsFunction(this, 'OrderPlacedFunction', {
   entry: resolve(join(__dirname, '../../src/service/ingestion/order-receiver/index.ts')),
   handler: 'handler',
   ...LambdaConfiguration,
   role: orderPlacedFunctionRole,
   layers: [
     telemetryExtensionLayerVersion
   ],
   environment: {
     SOURCE: 'ecommerce.orders.service',
     TABLE_NAME: this.OrdersTable.tableName,
   }
});

The example has the extension attached to all lambda functions, this will send all logs to the kinesis data stream for all functions and let the schema-registerer send those logged events to the custom event bus.

The schema-registerer function has a simple logic as below

await Promise.all(event.Records.map(async (record) => {
    const eventData = JSON.parse(Buffer.from(record.kinesis.data, 'base64').toString('ascii'));
    await client.send(
      new PutEventsCommand({
        Entries: [
          {
            EventBusName: process.env.EVENT_BUS_ARN!,
            Detail: JSON.stringify(event),
            Source: event.source,
            DetailType: `${event.type}.${event.dataversion}`
          },
        ],
      }),
    );
}

In the above example, the DetailType in PutEvents call is a concatenation of type and version

After sending a request the event schemas will be available in the schema section of the EventBridge.

Cataloging

Automation and documentation are two principal points in governance where we capture how the system behaves, and completing everything in an automated manner will be one important part of that journey.

In the above solution, The catalog will be generated using GitHub trigger and aws Code Pipeline, The pipeline will generate, build, and deploy the EventCatalog as a static website.

The source of AsyncApi specs will be a s3 bucket with a domain-based scaffolding as below.

The pipeline build process will Synchronize the s3 bucket to the local folder and generate the Domain, service, and events followed by building the eventcatalog bundle.

 version: 0.2

env:
  parameter-store:
    SPEC_BUCKET_NAME: /catalog/bucket/specs/name

phases:
  install:
    commands:
      - echo Installing dependencies...
      - npm cache clean --force
      - cd catalog && npm install --froce && cd ..

  pre_build:
    commands:
      - echo "Pre build command"
      - mkdir specs
      - aws s3 sync s3://$SPEC_BUCKET_NAME/ specs
  build:
    commands:
      - cd catalog
      - npm run generate
      - npm run build

artifacts:
  files:
    - '**/*'
  base-directory: catalog/out

The catalog project has two stacks, the pipeline stack and the catalog stack, the catalog stack represents the following diagram.

The Catalog and Pipeline stacks can be found as part of the source code in './src/platform/cdk'

• AsyncApi Spec will be uploaded in Specs Bucket

• The Event bridge will trigger the code pipeline

• The Pipeline will Sync all Specs in s3 and regenerate the EventCatalog

• The Static S3 WebSite will get updated by new bundle

Catalog of Thousands of Services

The above cataloging section focused on a simplified way of automating the Catalog generation but there is a last question to answer, How we manage thousands of Specs under ownership of hundreds of teams?

The answer is, a service owns the software and all corresponding documentation being OpenApi, AsyncApi, Readme, etc., By defining the spec under ownership of a team and as part of service source code, we need a way to simplify the cataloging in a central and automated way by relying on each service spec in each service repository.

Github actions are a good candidate to replicate the AsyncApi spec and make a copy in specs bucket. The original workflow will be a bit verbose as below

Github offers reusable workflows that can simplify the process of adoption in service teams, for this article simplicity a simple workflow action is the choice

name: AsyncApi Spec Sync
on:
  push:
    branches: [ main ]
env:
  AWS_REGION: eu-west-1
jobs:
  sync_spec:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@master
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Sync spec to S3
      run: |
        aws s3 sync ./spec s3://$(aws ssm get-parameter --name "/catalog/bucket/specs/name" | jq -r '.Parameter.Value')/

The sync will trigger the regeneration process and update the website.

The EventBridge rule listens to default event bus and trigger the Catalog CodePipeline when event changes are pushed to the bus. The EB rule using AWS CDK will be as below.

new Rule(this, 'rule', {
  eventPattern: {
    source: ['aws.s3'],
    detailType: [
      'Object Created',
      'Object Deleted'
    ],
    resources: [ props.specsBucket.bucketArn ]
  },
  targets: [ new CodePipeline(pipeline) ]
});

To receive the S3 event notifications to the EventBridge default bus the parameter must be enabled on S3 bucket.

this.specsBucket = new Bucket(this, 'CatalogSpecsBucket', {
  objectOwnership: ObjectOwnership.BUCKET_OWNER_ENFORCED,
  removalPolicy: RemovalPolicy.DESTROY,
  autoDeleteObjects: true,
  eventBridgeEnabled: true
});

Event Catalog Config

The event catalog use a generators section as part of the config file to execute AsynAPI plugin. the config looks like the following snippet.

const path = require('path');

module.exports = {
  ... ,
  generators: [
    [
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        pathToSpec: [
          path.join(__dirname, '../specs/Order/1.0.0', 'asyncapi.yaml')
        ],
        versionEvents: false,
        renderNodeGraph: true,
        renderMermaidDiagram: true,
        domainName: 'Orders System'
      },
    ],
    [
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        pathToSpec: [
          path.join(__dirname, '../specs/Product/1.0.0', 'asyncapi.yaml')
        ],
        versionEvents: false,
        renderNodeGraph: true,
        renderMermaidDiagram: true,
        domainName: 'Product System'
      },
    ],
    [
      '@eventcatalog/plugin-doc-generator-asyncapi',
      {
        pathToSpec: [
          path.join(__dirname, '../specs/Shipment/1.0.0', 'asyncapi.yaml')
        ],
        versionEvents: false,
        renderNodeGraph: true,
        renderMermaidDiagram: true,
        domainName: 'Shipment System'
      },
    ],
  ]
}

For the moment, the article has no solution to sync the config generators section in an automated way, so here for any new service there is a bit of effort adding the service.

Run the solution

The only required step to trigger the catalog generation process is to push a change to the main branche. this will trigger the s3 sync from local folder.

The S3 sync action will lead to the event bridge rule event match and trigger the catalog pipeline

Using the CloudFront distribution url the event catalog will be online and available

Conclusion

The distributed systems are really hard and when it comes to EDA in some mesures the pain-points of distributed system stays with an additional level of complexity that will not be visible in the start of EDA adoption journey, but becomes a real obstacle when system evolves and lots of services communicate using events.

The EDA being a great candidate for agility can become a break of agility without having a minimum of standards. in this article we had a look at a simplified journey of putting standards , operating and observing them. for sure there are the missions parts that can varie per company size , culture and existing tools but the idea stays the same.

Enjoy reading

Adoption of micro services was a finer level of improvement in distributed systems but actually those fine grained services behind achieving localised scalability , reliability and performance couldn't solve the overall system experience. Cascading latencies , Cascading failures , cascading downtimes was yet present and hard to resolve. The micro-service design approach was again relied on Request / Response communication over unfair network with its potential drawbacks in terms of system experience.

The actual Distributed system problems was at communication boundaries and putting a huge amount of effort to make every thing service scalable at 10x higher rate seemed overkill. The resolution to those challenges was putting all puzzle pieces together and find from where the local problems are propagated all over the system and the response was the Request / response lines.

The simplest part of these distributed system complexities is achieving the localised improvements. and the hardest part was solving the communication problems.s

Event Driven Challenges

The event driven design is distributed design like traditional distributed and micro service design has some complexities to solve, solving the above mentioned distributed bottlenecks just guided the softwares to the localised optimised state but the synchronous communication bottlenecks became an obstacle. Adopting EDA was a solution to improve those communication bottlenecks but added some inter-component and localised complexities.

The EDA biggest deal, when comes to overall system state and inter-component communication, is the state consistency. delayed propagated state and duplication.

Event Streaming

Event streaming is an operational approach relying on a number of events representing small changes in different services. Event Streaming helps to resolve the Consistency problem as the most important system level challenge in EDA. The delayed and duplication challenges mentioned earlier will be kept localised at service level and must be handled by any context Owner.

Event Streaming on AWS can be achieved using kinesis data streams as a Pull based solution:

The pull based streaming can be achieved by using Kinesis Data Streams with its own pros and cons.

Here an example figure showing the pulling from Kinesis

• The producer write records in stream

• Lambda event source mapping pull the records

Some details :

• ESM pulls the kinesis data streams once per second

• The Data Streams Guarantee the ordering per partition key

• Kinesis has a 2 MB/S , 5 RPS shared by all consumers.

• Kinesis Share limits are per shard , adding new shards can be a solution

Let's add some new consumers and run a test to see what happens. imagine we add some consumers up to 15 ( 15 seems enough for this test )

Creating a kinesis Data stream and 15 consumes using CDK

    const functionRole = new Role(this, 'FunctionRole', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')
      ]
    });

    const dataStream = new Stream(this, 'DataStream', {
      shardCount: 1,
    });

    for(let i = 0; i < 15; i++) {
      const functionName = `function-${i}`;
      const logGroup = new LogGroup(this, `Function-${i}-LogGroup`, {
        logGroupName: `/aws/lambda/${functionName}`,
        ....
      });

      const consumerFunction = new NodejsFunction(this, `Function-${i}`, {
        .....
      });
      consumerFunction.addEventSource(new KinesisEventSource(dataStream, {
        startingPosition: StartingPosition.LATEST,
        batchSize: 10,
        bisectBatchOnError: true,
        retryAttempts: 3
      }));
    }

    dataStream.grantRead(functionRole);

  }

Now it can be interesting to see how this solution behaves at build and runtime

Build time:

• During the deploy randomly you will face consumer source mapping errors as you reach some small limits but at longterm this will not be a problem

• If you have a batch of related records all consumers are implementing the same logic to consume, evaluate last state and processing those events, again this can not be a problem at a given time but think of presenting a new Pending event state between Created and Validated , here all consumers need to reevaluate the logic of state evaluation.

Run time:

• How the consumers behave from a latency point of view that is absolutely possible with the known kinesis data streams limits , 5 RPS, 1 second Polling per consumer by ESM. when a throttle happens this leads to a delay in consumer polling.

• A single malformed record can act as a poison pill and stop the sequence of records being consumed.

• The DLQ does not have the record data, passing the record retention period the event will be completely lost.

• If two consumers communicate together after processing of initial reception of an event from kinesis the delay represented above about throttling and latency can initiate problems or add extra communication complexity.

  

Broadcasting

The event broadcasting is another way of distributing events in an event driven system, this can be achieved using amazon SNS or event bridge bus.

The following figure represents the event broadcasting using SNS

• The producer publish events to the topic

• The sns will send a copy of event to all subscribers

• The sns can distribute the events to different type of subscribers as Lambda, SQS, Http/2, SMS, Mobile Platform and email.

Some details :

• The SNS Standard guarantees at least once delivery and does not guarantee ordering

• The SNS FIFO guarantees the ordering and partitioning of group of related events.

• SNS Quota indicates 100 TPS for subscribe action per account

• SNS has a 256KB event size limit

• SNS FIFO supports deduplication

Creating a above design using CDK

    const functionRole = new Role(this, 'FunctionRole', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')
      ]
    });
    const topic = new Topic(this, 'Topic');

    for(let i = 0; i < 15; i++) {
      const functionName = `function-${i}`;
      const logGroup = new LogGroup(this, `Function-${i}-LogGroup`, {
        logGroupName: `/aws/lambda/${functionName}`,
        ...
      });

      const consumerFunction = new NodejsFunction(this, `Function-${i}`, {
        ...
      });
      consumerFunction.addEventSource(new SnsEventSource(topic, {}));
    }

Build time:

• During the deploy randomly you will face subscription errors if number of consumers reach limits but at longterm this will not be a problem

• Having the same problem of KDS earlier the consumers can have a duplicated logic to evaluate events, but using SNS this will be harder at consumer side as the SNS distribute events one by one and has no supported batching. a new Pending event state between Created and Validated , here all consumers need to reevaluate the logic of state evaluation but to be able to execute that logic they need to persist the events first and fetch them per batch to evaluate.

• The SQS can subscribe to the SNS topic so this can be a serverless and managed way of having Offloading, Batching, Temporary Storage.

Run time:

• The SNS event distribution is fast and near realtime.

• The events are distributed based on filter policy if configured

• The failures are handled in a exponential way and up to 23 days if the subscriber is not reachable.

• Each subscription can have a DLQ

• The latency and performance of distribution is near realtime even a significant number of subscribers.

EventBridge is a powerfull service that helps to achieve the event broadcasting, it is a highly scalable and managed service.

• The producer put an event into the bus

• The event bus will send a copy of event to each subscriber via rules

• The event bridge supports a wide variety of services as a rule target like Api gateway, Api destination, SQS, Kinesis Data Streams, Lambda ( ASYNC ) , AppSync and etc ..

Some details :

• The EventBridge provides a at-least-once delivery and does not guarantee ordering

• EventBridge Quota indicates 18,750 TPS for rule targeting beyond that the throttling cause delays in event distribution.

• EventBridge has 100 rule per bus limit

• EventBridge has a 10,000 TPS for put event into the bus

Creating a above design using CDK

    const functionRole = new Role(this, 'FunctionRole', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [
        ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')
      ]
    });
    const eventbus = new EventBus(this, 'EventBus');

    for(let i = 0; i < 15; i++) {
      const functionName = `function-${i}`;
      const logGroup = new LogGroup(this, `Function-${i}-LogGroup`, {
        logGroupName: `/aws/lambda/${functionName}`,
        ...
      });

      const consumerFunction = new NodejsFunction(this, `Function-${i}`, {
        ...
      });

      new Rule(this, `EventBridgeLambdaInvokeRule-${i}`, {
        eventBus: eventbus,
        eventPattern: {
          detailType: ['my.custom.detailtype'],
        },
        targets: [ new LambdaFunction(consumerFunction) ]
      }); 
    }

Build time:

• The duplicated logic problem again persists fro event evaluation, The complexity is same as using SNS per distribution of events one by one and has no supported batching.

• The SQS can a target of rule so this can be a serverless and managed way of having Offloading, Batching, Temporary Storage.

Run time:

• The events are distributed fast

• The events are distributed based on filter policy if configured

• The failures are handled in a exponential way and up to 24 hours and up to 185 times if the target is not reachable.

• Each rule target can have a DLQ, and each rume can have up to 5 targets.

• The latency and performance of distribution is near realtime.

Responsibilities in Design

When designing Event driven architecture the requirements are different based on requirements and the rate of changes by entity that application manages. When designing EDA the relevant questions to ask and brainstorm are somehow as below:

• The Rate of changes on a single entity

• The variety of event types

• The relation or ordered priority between event types

• Who are the consumers?

• What does care a consumer about?

• Is the consumer in the same cell or business domain?

Defining each software responsibilities and how it manages the value to other participants in a distributed system is one of the important pillars of a design, actually this is important to define how we operate.

Ecommerce Example

The example provides a simple scenario to demonstrate a distributed system and how responsibilities around events are shared between different applications.

The operational process will be as illustrated, from the order arrival until the product delivery. it s clear that in operational perspective this design works.

Zooming and focusing on Product and order service shows more details about the details.

In this level the e-commerce website process will be:

• The website shows the products listing

• The order sent from website for selected products

• Order service notifies product service by reception of order

• The product service sends a replay to confirm the availability of product

Looking at Order and Product communication.

The Order service disseminates the order events and listen to product availability response.

• The Product service by reception of order.recieved event verifies the availability of product and update the product stock availability

• The product service send the product.available response message to notify the order service by availability of products in order

• The order service updates the order status to IN_PROGRESS and

The product lifecycle will be:

• The sellers add products via backoffice

• The product service sends events to listing service

• The listing service updates its local datasource to respond to website listing and search options.

Edge cases

• If the user modifies the order by removing an item, the order system need to notify the the product service to avoid the stock being exhausted this is achievable by sending a new order.modified*event letting the product service to reattribute the canceled product count to the product stock. but a better approach will be sending a meaningful event behind a real fact, so order service will be responsible of verifying the change in the modified order and send the corresponding event to the product service like order.product_canceled giving the product id and the original count.*

• If the user send in really short duration of time the creation and modification demands this can be useless to distribute both events to the product service, lets imagine the modification arrives before creation to the product service. it is clear this situation brings a lot of inconsistency to the product service.

Responsibilities

• Order service must guarantee the health of order reception, handling the change tracking and notify product service at a highest rate of trust. this means the order service must be able to react behind a stream of commands and take the best decision internally before distributing the events.

• The product service must be able to delay the distribution, merge or abandon an event due to its actual internal state.

Using right services

To accomplish some sort of tradeoffs discussed in Edge cases and avoid giving the responsibility to event consumers take decision out of their domain context, choosing the write services and design is important and in cases can be a vital decision.

The following diagram shows how the Order service can adopt a design to cover those tradeoffs and become a self serving microservice.

In this design the order service treat the stream of orders , this let the order service internally use the techniques like batching to verify a batch of related events and verify them and validate if any event state must be calculated or refined before broadcasting the notifications or integration events to other systems.

Source Code

Summary

In this article i tried to show some characteristics of the Evenbridge, SNS, Kinesis that are highly used when it comes to event distribution and represent the detailed limits to help better you see the tradeoffs when deciding the best service or a combination.

Walking through the examples, some real representative scenarios are explored to help better understand the core problem behind the technical consideration and help align the Bounded contexts, softwares and responsibilities related to them at enterprise level.

Thinking about responsibilities any service has in a distributed design can bring a lot of advantages in long-term and protect the software to be more contextualised and well mastered on a single business problem, avoiding producing a high level of technical debt related to a miss of well definition of context and responsabilities.

You are looking for some products on DealOfDay and find one that you are interested in buying. You reach out to the seller either by asking for more details or expressing your interest in the product. DealOfDay provides a messaging feature to facilitate communication between buyers and sellers. However, some customers have reported receiving suspicious messages from sellers, which could potentially be fraudulent. To prevent such problems, the DayOfDeal product team has decided to implement real-time fraud detection to better protect the system and its users.

Design

The starting point is an Apigateway that connects directly with the conversation Dynamodb table. If you want to learn more about direct integrations and how API gateway interacts with other AWS services like DynamoDb, I previously wrote about those integrations and their behavior. You can find more information about it ( Here ).

The source code here

Triggering State Machine

Using the event bridge Pipes to integrate the DynamoDb streams and step function is as easy as the following snippet

new CfnPipe(this, 'ConversationTablePipe', {
      roleArn: pipeRole.roleArn,
      source: this.conversationTable.Table.tableStreamArn!,
      sourceParameters: {
        dynamoDbStreamParameters: {
          startingPosition: 'LATEST',
        },
        filterCriteria: {
          filters: [{
            pattern: `{ 
              "eventName": [ "INSERT" ] 
            }`}]
        }
      },
      target: this.stateMachineStack.stateMachine.stateMachineArn,
      targetParameters: {
        stepFunctionStateMachineParameters: {
          invocationType: 'FIRE_AND_FORGET',
        },
        inputTemplate: `
          {
            "Id": <$.dynamodb.NewImage.Id.S>, 
            "message": <$.dynamodb.NewImage.message.S>,
            "timestamp": <$.dynamodb.NewImage.timestamp.S>
          }`
      }

The pipe reads from DDB streams and invokes the state machine asynchronously.

Conversation History

The solution receives the messages one by one, having messages individually is interesting but as a conversation needs to keep track of a contextual exchange we need to register all conversations to give a context back to Bedrock, this helps to prepare a better Prompt and leading the LLM provide a better result.

The DDB database keeps track of all messages in a conversation by keeping a Conversation Id as PK alongside the message and the message submission timestamp.

The workflow

The workflow consists of the following steps

The Workflow can be described as below

• Query DDB Table per ConversationId

• Fetch Prompt Markdown from S3 bucket

• Transform The prompt using a Pass State

• Call Bedrock Model to get results

The snippet represents the workflow implementation using AWS CDK

const defintion = new CustomState(this, 'Query Conversatuion', {
      stateJson: {
        Type: 'Task',
        Resource: "arn:aws:states:::aws-sdk:dynamodb:query",
        Parameters: {
          TableName: props.Table.tableName,
          ScanIndexForward: true,
          KeyConditionExpression: `Id = :id`,
          ExpressionAttributeValues: {
            ":id": {
              "S.$": JsonPath.stringAt('$[0].Id')
            }
          }
        },
        ResultSelector: {
          'messages.$': '$.Items'
        },
        ResultPath: '$'
      }
    }).next(new CustomState(this, 'Recap Conversation', {
      ....
      }
    })).next(new CustomState(this, 'Prompt Preparation', {
      stateJson: {
        Type: 'Task',
        Resource: "arn:aws:states:::aws-sdk:s3:getObject",
        Parameters: {
          Bucket: props.Bucket.bucketName,
          Key: "prompt.txt"
        },
        ResultSelector: {
          'body.$': '$.Body'
        },
        ResultPath: '$.prompt'
      }
    })).next(new Pass(this, 'Format Prompt', {
      parameters: {
        "output.$": "States.Format($.prompt.body, $.messages)"
      }
    })).next(new BedrockInvokeModel(this, 'Invoke Model With Prompt', {
      contentType: "application/json",
      model: {
        modelArn: props.modelArn,
      },
      body: TaskInput.fromObject(
        {
          inputText: JsonPath.stringAt('$.output'),
        },
      ),
    }));

Run the Solution

After deploying the solution we can call the API gateway giving a message, a conversation id and a message date time.

{
    "conversationid": "Qt-K5wvjjF4O4m-qSYWtW",
    "timestamp": "2024-02-22T19:40:03+00:00",
    "message": "Buyer: Hello i am interested in your house, can we fix a visit?"
}

Sending the request will trigger the step function workflow

These are the most whole workflow steps from start to the end resulting in a Suspicious rate being 0 or 1.

The Prompt

This is a really simple part of this example, defining a prompt for this use case was not too hard, The prompt is a predefined paragraph that includes a variable placeholder.

You are a conversation moderator on the DeadOfDay site and you are monitoring the following conversation between these two people: ' {} ', Rate me the suspicious nature of this conversation based on the messages exchanged. Only answer me with 1 if the conversation seems suspicious to you and 0 otherwise.

You. can find the ' {} ' placeholder in the middle of the prompt, this is the way to let the ASL replace it with the corresponding conversation.

States.Format($.prompt.body, $$.Execution.Input[0].message)

Bedrock results

Sending the prompt to Bedrock is simple using the step-functions service integration, the state machine definition will be as below

{
  "End": true,
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Parameters": {
    "ModelId": "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-text-lite-v1",
    "ContentType": "application/json",
    "Body": {
      "inputText.$": "$.output"
    }
  }
}

Calling bedrock and based on our prompts the results will be 0 or 1 to indicate 'Is It Suspicious?', 1 means Yes, and 0 No.

Here is the Bedrock response for the test message shown above.

{
  "Body": {
    "inputTextTokenCount": 77,
    "results": [
      {
        "tokenCount": 2,
        "outputText": " 0",
        "completionReason": "FINISH"
      }
    ]
  },
  "ContentType": "application/json"
}

The outputText represents the LLM answer.

Now, Let's continue the conversation

{
    "conversationid": "Qt-K5wvjjF4O4m-qSYWtW",
    "timestamp": "2024-02-22T19:54:44+00:00",
    "message": "Seller: Yes sure, just you need to send me a 5000 dollar of deposit before the visit"
}

The above request will result in a Suspicious message detection if we apply a context, being the previous conversation message

- Buyer: Hello, I am interested in your house, can we fix a visit? 
- Seller: Yes sure, just you need to send me a 5000 dollar of deposit before the visit

The importance of context

On 7 April 2020, I became a dad, this was a big step but at the same time during frustration I learned a lot, Never say a single Yes or No, This is not helpful when you talk with a brain, a brain needs a context either the result will not be effectively corresponding to the wished one, but it can be helpful in a second phase.

Giving the second request ( the seller suspicious message) without providing the conversation history results in the following response from Amazon Titan Text V1 Lite, With an Output text as a '0' meaning that the message is not suspicious.

{
  "Body": {
    "inputTextTokenCount": 83,
    "results": [
      {
        "tokenCount": 2,
        "outputText": "0",
        "completionReason": "FINISH"
      }
    ]
  },
  "ContentType": "application/json"
}

To better prove the need for a context I tried to invert the conversation message order just by setting the 'ScanIndexForward: false' this will revert the query order.

Having the previous conversation as

• Buyer......

• Seller......

The inverted conversation will be as

• Seller.....

• Buyer ....

Having a suspicious conversation because of seller message previously, this time the LLM will indicate the conversation as an unsuspicious conversation even having the seller message inside.

Enjoy Reading

Lambda Runtime

Lambda functions run inside a container image for managed and custom runtimes. In both cases, the runtime prepares an execution environment, and the container provides a runtime interface to interact with it.

AWS lambda offers the following supported managed runtimes

• NodeJs

• Python

• .Net

• Java

• Ruby

AWS Lambda provides OS-Only runtimes with a runtime interface integrated to offer an operational runtime. The OS-Only runtime is an excellent choice for ahead-of-time (AOT) compiled languages. On the other hand, custom images are perfect for runtimes not managed by AWS and those that are not AOT.

IL & JIT

Compilers serve the purpose of translating high-level languages into low-level languages for processors to comprehend. This process optimizes the code and enhances performance during runtime. In certain programming languages like C#, the compiler generates the Intermediate Language (IL) to optimize the runtime process and Just in Time (JIT) as a means of communication between the application and the processor during runtime. Although this method has proved to be effective, it requires a language-specific runtime to be available on the container at runtime. To see deeply this process , sharplab.io will help to visualize the different levels of interpretation of IL and JIT from a high-level language syntax (example here).

AOT

AOT, which stands for Ahead of Time, is a type of compilation process that helps to minimize the interpretation process at runtime. During the build phase, the AOT interprets the software code into a processor comprehensive language and creates a self-contained package. This package allows the software to run in any container without requiring a language-specific runtime to be available.

One of the great advantages of AOT is that it provides a way to interpret to a low-level language or for static languages to interpret to machine code. During the compilation, AOT applies the necessary optimizations that were traditionally done during JITing. However, as a disadvantage, the optimization cannot be done at 100% (but mostly done) because some optimizations must be evaluated at runtime. In this kind of situation, runtime JITing shows its proper advantages.

Experience of IL, JIT and AOT

In my experience, when working with CRUD applications, using AOT (Ahead-of-Time) compilation was the best option because there were no complexities that required JIT (Just-in-Time) compilation at run-time. However, in other scenarios where there were long-running, CPU-intensive calculations for large batches of data and effective parallelization was needed, using Dotnet6 runtime performed better. For transactional processing with small batches of data (20 items), AOT compilation was found to be better performing. I believe it would be worth revisiting and re-preparing the examples I worked on years ago and writing a dedicated article on this topic.

Rust & Go functions

I have only written a single function in Go throughout my entire experience, and I've never written a line of code in Rust either. However, I find it fascinating to follow the community and hear about these languages. Recently, I learned from Benjamen Pyle that Go and Rust are AOT and provide self-contained packages. This news made my day, and I tried to discuss it with my friends. Everyone was excited about it, and we decided to give it a try.

As a part of an experiment, I followed a blog post by Benjamin Pyle.

For my first lines of code in rust, I was volunteered to errors, I started to change a simple handler in rust adding an array and using some manipulation on that array, surprisingly I found that the compiler stopped me from producing the runtime race conditions or bugs in production, this is a great DX, I felt this behavior like a good linter but in reality it’s not just a linter and it s a real thing that I can not give it a name. I am discovering rust and see what is under the hood so I let you discover this DX and share your observation.

Node Js

Since I moved to AWS, I used mostly typescript ( Nodejs ) and I'm already happy about this jump from C# ( This was due to lack of performance in .netcore1 runtime in 2018 ). The Nodejs performs well in most scenarios and could tackle Python, the new versions are more performant, and in some benchmarks, I found nodejs close to Go ( Go performance is great and enough for me )

Nodejs Runtime is managed by AWS and the migrations from 10 to 12, 14, 16, 18, and now 20 are always done smoothly for me. The nodejs in most cases have a two-digit milliseconds execution duration and the cold start is acceptable in any business in which I was involved (the last time I observed the cold start rate it was ~0.06 % in the production account).

Back to 2018

Back in 2018 in our company, we moved from C# to nodejs for serverless solutions, there were points why we preferred to continue using C#.

• 500 C# developers and the move was not fast

• Existing ecosystem, there was a wide range of internal c# libraries that helped the teams to apply easily the best practices.

• Reuse of existing code, For a while, we pushed the hexagonal design in most of our complex software the reuse of business logic was easily achievable.

• The risk of degradation while rewriting code was high and as we were adopting a fast delivery approach and a rapid migration it was preferable to use C#.

Living with NodeJs

There were no particular complexities and problems apart from stopping using the same logic in a new language. With nodejs we had a good-performing event-driven platform with optimised performance and less headache than before. The first benchmarking and results were surprising, for nodejs 300–500 ms cold start and 100–200 ms execution time vs c# based system of 200ms average in the existing container-based system. The gain was not evident but after going into production our observations was

• The cold start happened occasionally

• The Managed runtime and support for nodejs was faster than dotnet by aws

• Typescript is great as we came from an OOP and static language world.

• Project layout was simpler to evaluate and change in the JS ecosystem, we just all-time used tree-shaking, and everything went well

• It was simpler to hire AWS serverless Nodejs profiles than C#

Keep going with NodeJs while suffering

We continued and will continue NodeJs for all the above-mentioned reasons. but there are suffering moments while using NodeJs.

• Changing thousands of functions is not simple with every deprecation

• Putting the Build & Run mindset was simple but the mindset for upgrading to the latest runtime is hard to settle for all staff

• Every upgrade needs a whole regression test phase to be sure as the code failures are at runtime while everything goes well locally.

• While the NodeJs V8 engine does some sort of compilation there is no way of having a 100% assured package vs dotnet as a compiled language.

Decouple the software from container details

This is the exact thing that happens when using AOT and this is possible for dotnet Go and Rust. This may be achieved for Nodejs with some over-engineering but to be honest I tried and never had success.

LLRT

Recently AWS provided the LLRT runtime that brings real optimisation and is very close to well-performing runtimes running in AWS lambda functions. The LLRT comes with some sort of limitations or restricted capabilities by design as mentioned here in the documentation.

The runtime provides the very basic functionalities as indicated in the documentation but for me, it covers most of my function needs.

More needed

As nodejs is around modules and packages If a function needs more capabilities for a use case we can add that package and put it as part of the function package. this is actually how it works for all packages I use.

Example

To start my experimentation I found a great repository letting me start my tests on CDK and include the llrt runtime in the bundling process.

The construct use a afterbundling hook to add the llrt as part of packages

         afterBundling: (i, o) => [
            // Download llrt binary from GitHub release and cache it
            `if [ ! -e ${i}/.tmp/${arch}/bootstrap ]; then
              mkdir -p ${i}/.tmp/${arch}
              cd ${i}/.tmp/${arch}
              curl -L -o llrt_temp.zip ${binaryUrl}
              unzip llrt_temp.zip
              rm -rf llrt_temp.zip
             fi`,
            `cp ${i}/.tmp/${arch}/bootstrap ${o}/`,
          ],

I created a sample repository for both NodeJs20 and LLRT lambda functions using CDK.

To deploy the functions just run following command

cd cdk && npm i && npm run cdk:app deploy

Looking at cdk path the .tmp folder represents the llrt bootstrap on ARM64

The example use Function url to simplify the testing

Invoking the LLRT function Url and looking at cloudwatch logs gives interesting insights

Init: 54.52 ms

Duration: 1.63 ms

i was curious to invoke the warm container again and this was weird to me

Try again and again this was changing all over the time and in max duration i had a report as below

This is 7 times more than previous invocations but it is a great performance yet.

The execution environment while responding to requests permanently has a single-digit duration of around 1.4 to 1.9 ms, but if I leave the execution environment for a while and use the same execution environment again the first duration was 2-digit milliseconds but the average duration is always really interesting

I would like to compare this with the managed Nodejs20 by invoking nodejs function function url

Interesting results, this is a classic nodejs function that I observe in production so no surprise, but trying to send more invocations was when I tried to focus more and more on details, something was interesting happening that I never experienced while observing Nodejs18.

The Nodejs 20 durations are great for the same code , i had same results as LLRT for managed runtime .

The other point got my attention was memory consumption.

LLRT: 20MB to 25MB

Nodejs: 89MB - 93MB

And a last point was the Billed Duration and this is normal as LLRT is not a managed runtime and you are billed for init phase + duration

Estimation

Finally i was curious to estimate the cost of running LLRT and NodeJS

As i had no more credits in my accounts i decided to estimate per pricing page by introducing a scenario but finally i decided to run a solid test using artillery

config:
  target: https://XXXXXXXXXXXXXXX.lambda-url.eu-west-1.on.aws
  phases:
    - duration: 1200
      arrivalRate: 100
      name: "Test Nodejs 20"
  environments:
    production:
      target: https://YYYYYYYYYYYYYYYYY.lambda-url.eu-west-1.on.aws
      phases:
        - duration: 1200
          arrivalRate: 100
          name: "Test LLRT"
scenarios:
  - flow:
      - get:
          url: "/"

Memory Configured : 128Mb

Architecture: ARM64

Request count: ~6 000 000

After using artillery to do a spike test and verifying the results with following cloudwatch log insight query

filter ispresent(@duration) |
stats sum(@billedDuration), sum(@maxMemoryUsed/1024/1024), count(*) as requestcount

This shows off well that the pricing is close and in kind of typical business with these sorts of loads, it is just equivalent. To be honest, the LLRT is in the experimental phase and I will never adopt it for production but what LLRT offers we can achieve using Managed Runtimes, the only GAP is a self-contained package that can be handy in lots of use cases ideally for software code that I run.

A final word

Often, in companies, we define a tech radar that considers all historical, strategic, and technical aspects. The last consideration is the plan. There are a lot of details to decide whether to adopt a programming language, tooling, or vendor services. The cost of moving to a new programming language is often higher than improving the practices and optimization (at least in my experience), but the fact that tech grows is evident and if we do not question ourselves internally, or externally, we will never grow.

Going to Rust is a nice move. Using LLRT is fascinating. Go is great and C# has and has for me the greatest and cleanest syntax I have found in some languages I have worked with, like Node.js, Python, F#, Delphi, VB and C++, and Java, a lot close to Java, but I moved to Node.js ecosystem after 20 years of .NET programming just because of tradeoffs and the tradeoffs are everywhere. But approving tradeoffs as a fact is also important. Do we stop c# or Serverless ? For sure, c# is the loser in this game. Do we adopt LLRT, or do we improve our current software? For sure, improving is faster than moving to LLRT and doing lots of time-consuming tests.

The adoption of new things in a business is not for fun and must have an impact on the system.

But StepFunction is not today just a Serverless orchestration service and let to achieve more significant results in different distributed scenarios. By the release of Distributed Maps, it became a good fit for distributed data manipulation like large files ( JSON, CSV ) or files in an S3 bucket.

Distributed Map

In general words , the way distributed map works is by dividing a large asset into chunks and let them be treated in isolation and simultaneously.

In technical words, Distributed map parallelize the treatment of batch of items in isolation logical contexts called child workflow executions. This isolation also helps to control in isolation the errors that any batch of items can encounter during treatment.

Distributed map helps control the collaboration between different services via refined configurations like MaxConcurrency.

Configuration

Some useful configuration to understand are explained here but to have a better understanding i recommend to refer to the AWS Documentation

MaxConcurrency: maxConcurrency( default 1000 ) is a mean to specify the number of concurrent child workflow executions that can be executed simultaneously.

ItemReader: This config indicates the source of dataset that the distributed map must use to fetch the data. this can be a S3 bucket, A CSV or a JSON File.

ItemBatcher: You can define the Max number of items or Max Bytes of Input Distributed map passes to each child workflow execution.

ItemProcessor: The configuration that represent the workflow definition and states of workflow to treat the batch of items as well the ProcessorConfig to indicate the type of StateMachine being STANDARD or EXPRESS and The mode.

How distribution works

• Reader Reads the DataSet per ItemReader config

• Batching Batch data to array of items per ItemBatcher config

• Item Processor Distribute the batch sets of items and execute the child workflow executions giving a batch of items as input.

You can configure Item Selector and Result Writer , to lean more about refer to Documentation

Map Run resource

Map Run resource behaves as a coordinator for Child Workflow executions by coordinating following details.

• Concurrency

• Batching

• Keeping Track of Child execution States

in Practice when you define MaxConcurrency or MaxItemsPerBatch as in our example, the Map Run associate items by assuring that the Batch Size does not bypass the 256 KB and based on batching possibilities allocates concurrent child executions.

Try it on your own

This article source code is publicly accessible here, If you would like to try the examples and different patterns follow the instructions in README.md file in this repository

Using Simple Distributed Maps

In this example we read a large number of files for a s3 bucket ( 65K Json objects of 200KB ) and will look in details how the workflow behaves.

The parent DISTRIBUTED map configuration has following details

The INLINE map configuration ,inside the parent Distributed map, has following details

The Example will be as illustrated in the following figure

Running this example longs around 10 minutes and will result a failed status after that time due to history limit of 25000 ( hard quota ) .

Changing the batching MaxItemsPerBatch configuration is one of the most straight forward solutions to this limitation. After setting the MaxItemsPerBatch to 1000 items the process same s3 objects will result a success status with a duration of 05:40.531 ( 5.5 Minutes ).

This result can reasonably cover a variety of scenarios but for scenrios with large amount of data can take a long time and does not seem the best fit in term of operation, performance and requirements.

Using Nested Distributed Map

In this example, the configuration is partially same as before , we read a large number of files for a s3 bucket ( 65K Json objects of 200KB ), the only difference is that for Chile Workflow execution is encapsulated in a second ( Nested ) Distributed Map to add a second level of parallelism at batch level.

The parent DISTRIBUTED map configuration has following details

The Nested DISTRIBUTED map configuration has following details

The nested INLINE map configuration has following details

The Example will be as illustrated in the following figure

Running an execution create a first Map Run resource ( More info here )

Looking at Map Run resource, there are a listing of executions, each with a Batch of Items as below.

In this example looking at any single execution with around 1800 items, the execution is a nested distributed map by its own. by this example we create a Top Down Distributed Map hierarchy at 2 level.

Here an example of what looks a child execution with a workflow State containing an INLINE Map State.

This Nested Distributed Map also has a listing of second level child workflow executions but this time with limited Items per Batch around 50 for each execution.

This seems a reasonable situation to be treated by INLINE Map ( Limited concurrency and History quota )

Looking at this execution list all executions, the start and end time of executions are approximately close showing the parallelism and resulting a close execution duration ( in our example around 2 minutes ) and looking at the Parent Distributed Map durations we notice the same.

The Execution ran for a duration of 01:20:959 (1.5 Minutes) with a success result.

Conclusion

Distributed map is a great feature and shows off well the power of AWS Step Functions. The Distributed Map can be used in a variety of situations when the need of performance , simplicity and process isolation can be a concern.

In this article we could achieve a better understanding while processing amount of data in a scalable , reliable and performant manner.