![\"A](\"/2025/09/spec-workflow/workflow.png\"\n)
Overview of the spec-driven workflow – five stages plus a decision log.
\nGenerative AI has changed the way we develop applications. Even if you haven’t embraced the hype yet, you can’t avoid reading mentions of it, and this post is no different. The biggest challenge for AI-generated code is to get consistent, good-quality output with the least amount of effort. After all, what’s the point of generating code if you then spend more time debugging it? This post explains how I try to achieve this goal.
\nEver since I was introduced to spec-driven development in Kiro, I’ve seen it as an excellent candidate for managing the quality of generated code. The idea of having documentation and a detailed design as the guiding light of your development isn’t new, and when done well, it tends to deliver good results. There are no perfect solutions (yet?), so all we can do is try to improve how we work day by day.
\nI believe I’ve put my own spin on the general way spec-driven development works, so it’s a good time to share my current workflow. Even if it just serves as a historical record for myself.1 I’ve mostly switched to using Claude Code, so some of the details mentioned will be specific to that, but most parts of this workflow can be used by any tool2.
\nSpec-driven development is a well-known pattern, but please allow me to give a brief recap of what it means, so that I can then go into the details of what we actually want to achieve with it.
\nFirst, there is the idea. The thing we want to do, this will start out as a rough idea and maybe we’ll do some iterations around what exactly we want it to look like, what it means, etc.\nBut then we start collecting our requirements. This is an exhaustive list of all the things we want to achieve. This should include our technical requirements as well as any functional, and possibly business requirements.
\nAfter we’ve got our requirements, it’s time to design what we want to build. This is where we’ll go into the nitty-gritty of how we want to achieve things. What should the UI look like (if there is one), how should the code work, where does this fit inside the existing project, etc.\nAnd then when the design is finished, we can work on creating the tasks that need to be completed.
\nLast, of course, is the actual execution of these tasks.
\nDuring the course of all of this, you will need to make decisions. These can be big decisions or small ones, but keeping track of these is what a decision log is for.
\nOverview of the spec-driven workflow – five stages plus a decision log.
\nThe interesting bit here is how this has changed with the use of agentic programming tools. It used to be that the execution phase was the longest, but that is no longer necessarily the case. At least not in terms of input from me. The best-case scenario is that I spend between 60 and 80 per cent of the time in the requirements, design, and task phases. The bulk of the remaining time is spent reviewing the generated code.
\nSo, how do I use this workflow? Below, I’ll go through the steps one by one, but it’s also important to note that in order to keep my context small and prevent it from accumulating conflicting information, I clear the context between every command. Everything should only work based on the input it gets from the previous step, and I commit between every step3 so that I can easily roll back. I’ve noticed that people are often still hesitant about undoing changes from an agent; don’t be. You don’t put effort into the code generation itself, so you can easily undo and try again. Or even have multiple branches working on the same spec and picking the one you like most.
\nA quick note about tools: if you’re not familiar with the Model Context Protocol (or MCP), it’s a protocol used by a lot of tools to make it easy for your agent to interact with. I don’t use a lot of them day to day, but try some of them out regularly. You don’t want to have too many of these running at the same time, because while they’re useful, they can also take up a lot of context. The other tool I use a lot in Claude Code is subagents. Subagents allow you to run specialised tasks in a separate context window and can run in parallel. If you haven’t tried them before, I highly recommend doing so, and I’ll show how I use some of the ones I’ve built.
\nThe idea is straightforward. I have an idea of what I want to achieve, and I will write that down. If it’s something small, I can write it down directly in the prompt for the requirements (more on that below), or I may write it in a file. The idea has likely4 had some thought put into it. That thinking can be done by myself, or by discussing it with an actual person, or even an AI chatbot like ChatGPT or Claude (both have their strengths, and I tend to use either or both depending on the context).
\nRequirements gathering time! Obviously, this is where we nail down what we want. I’ve created a custom slash command in Claude Code that is named requirements. That link points to the latest version I’ve pushed to GitHub, so you can read it there, but let’s go over some of the main things in there.
\nAs you’ve read above, there are usually two ways in which I start this in a session:
\n/requirements I want to do this thing\n/requirements We'll be working on feature awesome-feature-name. Read the idea file and carry out your instructions.\nYou can see that I provide the feature name in the second example, while in the first one, Claude will propose one. Either way, the name of the spec is the name of the folder it will live in: specs/awesome-feature-name. To match this, I create a branch with the same name5 and the following steps will all be able to determine the spec name based on the branch.
Anyway, at this point the fun starts. While having AI write the requirements is a good start, the quality of a first run isn’t necessarily the best6 and no plan survives contact with the enemy peer reviewer. So, after generating the requirements document, my command will do three things:
The peer reviewer subagent is the most fun bit, as it does what might be considered slightly overkill. Using Sam McLeod’s devtools MCP, it will ask both the Gemini CLI and the Amazon Q Developer CLI for their opinions. Yes, it calls two different editors/tools that have their own agent running. Gemini CLI obviously uses Gemini 2.5 Pro under the hood, and while Q Developer’s Claude 4 Sonnet is not too different from the Claude 4 Opus that I have running in Claude Code, the settings and different system prompt still provide a unique perspective.
\nAll of this input is then collated and turned into updates to the requirements document. The fun thing here is that I’ve set things up to be critical enough that it will sometimes suggest not to implement a feature, or change the way I envisioned it. After this, there might be more questions for me or I have additional feedback, and we iterate until I have the exact requirements I want.
\n# Front Matter References Feature Requirements\n\n## Introduction\n\nThis feature adds the ability to add front matter content through CLI commands in the rune tool. Currently, users cannot add front matter using the command-line interface, requiring manual markdown file editing to establish metadata, references, and other YAML properties.\n\n## Requirements\n\n### 1. Add Front Matter via Create Command\n\n**User Story:** As a user, I want to add front matter content when creating new task files, so that I can establish metadata and references from the start without additional steps.\n\n**Acceptance Criteria:**\n1.1. The system SHALL extend the create command to accept --reference flags (repeatable) for reference paths\n1.2. The system SHALL extend the create command to accept --meta flags in "key:value" format (repeatable) for metadata\n1.3. The system SHALL support adding multiple references and metadata entries in a single create command\n1.4. The system SHALL create the front matter section at the beginning of the new file\n1.5. The system SHALL generate valid YAML front matter format\n1.6. The system SHALL add references as YAML array entries under the "references" key\nAs you can see in the above example (full version), the end result includes a lot of details about what we want to achieve, but doesn’t include how. In addition to this, every single time a decision is made, by me or the model, it will update the decision log so I can see why we went a certain route. This decision log remains important throughout the entire process, including the execution phase, as everything should take it into account.
\n# Front Matter References Decision Log\n\n## Decision 1: Feature Scope Simplification\n**Date:** 2025-09-01\n**Decision:** Limit feature to only adding front matter content, not editing or removing\n**Rationale:** User feedback indicated the initial scope was overly complicated. Only addition capability is needed since existing commands already display front matter.\n**Impact:** Simplified requirements focus on two commands: extending create and adding add-frontmatter\n![\"A](\"/2025/09/spec-workflow/requirements.png\")
Requirements phase flowchart – from idea to approved requirements (click to expand).
\nThe design phase is very similar in nature. As mentioned before, by this point, I’ll be working from a branch that matches the name of the spec. It will automatically find the correct spec and attempt to read the requirements. And all I have to do is run the workflow without any arguments.
\n/design\nI can of course provide it in the unlikely case I’m working on a differently named branch, I can include the specs/ prefix or leave it off. Usually though, I just make sure I’ve got my branch in order as all the following steps have the same logic.
/design transformation-pipeline\nThe command then does some validation, and if everything is fine, it will start doing research using the available tools, as well as once again asking the same subagents for their input. This process again often generates questions, and based on exactly what I need I may have a lot of comments and discussion going on here.
\nIn fairness, my input on these items is very much influenced by what I find important. One thing I will often do in this phase is call other subagents manually to review the design, especially my code-simplifier, to limit the amount of overcomplicating that can happen7.
\nBe aware that the design document is often quite long, and there can be many things in there that may not be relevant to the design you’re working on. For example, when I was building the tasks management tool I’ll describe in the next section, it tried to add lots of benchmarking, speed optimisations, and security validations, while I just wanted something that could easily parse a Markdown file that contains maybe 50 tasks.
\n# Front Matter References Feature Design\n\n## Overview\n\nThis feature extends the rune CLI with the ability to add YAML front matter content through command-line operations. The design focuses on two primary command enhancements: extending the `create` command to accept front matter during file creation, and adding a new `add-frontmatter` command for modifying existing files.\n\nThe implementation leverages and extends the existing front matter infrastructure in the codebase, which already handles parsing and rendering YAML front matter through the `FrontMatter` struct and the `ParseFrontMatter`/`SerializeWithFrontMatter` functions in the `internal/task` package.\n\n## Architecture\n\n### System Context\nThe design starts with a high-level overview, but quickly goes into the exact details on how something needs to be implemented. This includes showing parts of the code that can then later be used during the execution phase.
\n![\"A](\"/2025/09/spec-workflow/design.png\")
Design phase flowchart – from requirements to approved design (click to expand).
\nWith the design complete, we simply need to break down the actual work into tasks. One thing I’ve found is that when you give an LLM a file full of tasks, you REALLY have to force it to not go beyond what you want. Because of this, I’ve written a small tool called rune that creates the tasks file exactly the way I want it, and lets me retrieve tasks one at a time in the execution phase without the agent seeing the whole file8. For this phase, the first part of this is the important bit.
\n/tasks\nYes, it works like the design one in that it automatically does all the validations, finds the correct info based on the branch, etc. After that, it generates the tasks and this is probably the biggest part that is ripped off from Kiro. As I said, the tasks get put into a tasks.md file and I add a bit of metadata to that. Specifically, the command adds links to the requirements, design, and decision log files of the spec as references. If there is any other information relevant to all tasks, I will add it manually as well. An example of that might be an API reference for a library.
\nSome items I demand of the tasks:
\n---\nreferences:\n - specs/batch-operations-simplification/requirements.md\n - specs/batch-operations-simplification/design.md\n - specs/batch-operations-simplification/decision_log.md\n---\n# Front Matter References Implementation Tasks\n\n- [x] 1. Implement core front matter utilities and merge logic\n - Build foundational functions for parsing and merging front matter\n - [x] 1.1. Write unit tests for ParseMetadataFlags function\n - Test single key:value parsing\n - Test multiple values for same key creating arrays\n - Test invalid format detection\n - Test empty key/value handling\n - References: Requirements 1.2, 2.3\n - [x] 1.2. Implement ParseMetadataFlags function\n - Create function to convert 'key:value' strings to map[string]any\n - Handle multiple values for same key by creating arrays\n - Keep values as strings by default (no type inference)\n - Support colon in values by using first colon as separator\n - References: Requirements 1.2, 2.3\nThat said, this is the least interactive part of the whole workflow and mostly just works without any input. The example here only had 5 main tasks, but I added another 5 afterwards when I disagreed with the result. Showing that you should still verify the tasks will do exactly what you want them to do.
\n![\"A](\"/2025/09/spec-workflow/tasks.png\")
Task creation phase flowchart – from design to approved task list (click to expand).
\nFor the execution, I mostly let it do its thing. I will often be doing other things while coming back to review code and type /next-task again.
![\"A](\"/2025/09/spec-workflow/next-task-cli.png\"\n)
Command-line output when running /next-task.
\nIntroducing the rune tool has made things a lot easier in this step. I no longer need to have a bunch of demanding rules about only implementing a specific task and then seeing that ignored. That doesn’t mean it can’t still go overboard and do more work than needed, but because all interactions with the tasks goes through rune, the model literally doesn’t read the tasks.md file so that risk is lowered10.
\nAfter the command is done with the work, I then review it. If the tasks have been created well, there isn’t too much code to review. As everything is unit tested, I tend to focus more on the overall structure of how it was implemented, if I see any weird things, or just disagree with something. I will then ask why something was done, and either tell it to make changes, or I will roll back all changes and make adjustments to the spec before trying again. Unless the code is needlessly complex, I’m usually happy to let it through without any major changes.
\n![\"A](\"/2025/09/spec-workflow/next-task.png\")
Execution phase flowchart – shows how tasks are processed step by step (click to expand).
\nI mentioned earlier that the workflow isn’t perfect. In the end, we’re still working with flawed beings on both sides of the keyboard11, so all we can do is continually try to improve. I’m not going to go over every mistake that’s been made, but some things I’ve run into are:
\nIn a way, all of these points come down to the writing of the spec in its various stages. If I’m writing something small, I should include that it’s not meant for highly secure use cases. Because in the research phase it will look for the “best” possible way to implement it and the design phase explicitly calls out security.\nMy code simplifier subagent is also a direct response to this, I just can’t let that one be a direct part of the solution as it can be a bit trigger happy on features it doesn’t like.
\nThis workflow is constantly evolving, and I’m always trying to tweak the rules for the various commands. And of course, sometimes I make bigger improvements, like the introduction of rune or the decision log. I have many plans, and I’m sure I’ll be writing about them as they get ready.
\nOne thing I want to do is tweak the roles of Gemini and Amazon Q Developer. Especially with Q, I can make it use specific agent modes that I define to get better results, and I haven’t started playing with that yet. It’s actually an interesting way of achieving results and should give me a range of reviewers I can use for various roles.
\nThat said, all of my current workflow can be found in my agentic-coding repo and feel free to make use of that in whatever way works for you. I’d love to hear your input about things that work well or not.
\nAnd maybe in the future I can point at this post and laugh how wrong I was. ↩︎
\nI also use it with GitHub Copilot for example. ↩︎
\nI have a workflow for the commits as well, which I’m not going through in the post. You can read it here. ↩︎
\nI like to think this is always the case, but it is possible that at times I might be tempted to go “that sounds like fun, let’s build it”. Not often obviously, I doubt it happens more than 2 or 3 times a day. ↩︎
\nWhen using a similar system for clients, I will of course adjust to the requirements there. ↩︎
\nYes, I have shelled out for the cheapest Max plan so all the design work uses Opus, but that doesn’t make it perfect. ↩︎
\nA 2400-line bash script is a lesson well learned. Thanks Kiro. ↩︎
\nI’ll have a post in the near future about building rune itself and how it fits in this workflow. ↩︎
\nDon’t worry, like most developers, the model usually finds a way to do the implementation first anyway. ↩︎
\nIt still reads the design document, so it knows what needs to be built, but so far it hasn’t actively tried to build anything that it didn’t need for its task. ↩︎
\nI like to think I’m the one who is slightly less flawed, and please don’t disabuse me of that notion. ↩︎
\nIn early June, I received an email inviting me to an early look at Kiro. The only thing I knew at that time was that it was a new agentic IDE and created by AWS. Obviously, I was very interested in this, and today Kiro is available to everyone in public preview so that I can talk about it.
\nLike many others I’ve become more and more interested in agentic coding over the past months, and I’m usually using several different tools at the same time. Whether it’s a VSCode plugin like Cline or Copilot, a terminal app like Claude Code, a web-based tool like OpenAI’s Codex, or a standalone VSCode fork. Kiro falls in that last category, which has its pros and cons. The introduction blog post and official documentation covers the basics like installing, on-boarding, etc. So I’m not going to go into that. Instead I’ll focus on what I like about it, what can be improved, and what I learned about how to make efficient use of it.
\nIn many ways Kiro is an extension of VSCode, with the aim to make it a single experience focused on agentic coding. This, of course, means that it has the now familiar sidebar with a chat agent, but the strength of Kiro actually lies in the toolbar on the other side. This is where you find the MCP config (as expected), but also 3 other sections:
\n![\"The](\"/2025/07/kiro/kiro-sidebar.png\"\n)
\nThese 3 are the core of what makes Kiro, well, Kiro. The ideas aren’t unique, but they are well implemented in a nice package. Let’s start with the steering files.
\nIf you’ve done any agentic coding, you’re likely familiar with steering files. They have different names in different tools, but in general they are the files that guide/steer your agent into making the best use of your environment. This means things like understanding the structure of your code, what it does, and what it uses to achieve this result. When you first open a project in Kiro, it will show you a button to generate these files, but then it creates for you a product, structure, and tech Markdown file. These are fairly succint1, but mostly show how it all works. The structure of these files has changed a bit over the releases, but I can’t say right now if that’s because of tweaks or that my original ones were generated by Claude Sonnet 3.7, while the later ones were with Claude Sonnet 4.
\n![\"A](\"/2025/07/kiro/steering-files.png\"\n)
\nAside from these generated ones however, you can still add your own steering files. This means you can leave the generated ones alone (and potentially regenerate) and create new ones specifically for your purposes. Fun fact, when dealing with an early version of a tool like this, you can end up using these a lot to work around issues, some examples I’ve had to use:
\ncd . when running terminal commandsSo, these steering files are really nice. They allow you to easily have a good start for your agent to work with, and add custom rules you want to include. My biggest complaint with them, though, is that there is currently no option for a global steering file (or multiple files). And that is something I really miss, as there are plenty of things I’d like to have consistent across projects without needing to copy them.
\nSo, what’s the best way to use these? Well, you generate them, create your own to put in your own additions, and whenever you’ve done a major update, you should ask for them to be regenerated. That way the steering files will remain up to date. And of course, there are all the usual things you should do for an agent file, like make it easy for the agent to find tools (tell it to use your make commands or other scripts if you have them), include your own preferences around tool use and even things like spelling or comment structure. Do note that when you ask Kiro to regenerate your steering files, it will also do that for the ones you manually made.
\nThe hooks are a useful integration. It’s basically a set of pre-defined prompts that you can have automatically trigger, or manually. Again, Kiro is not the first to implement something like this (although the automatic part isn’t very common yet), but they did a good job here with the creation bit. Basically the flow is that you will provide a prompt of what you want to achieve, and then based on the steering files, and its knowledge of the project it will provide you with an efficient code. For example, I gave it the prompt
\n\n\nEvery time a task is finished, I want to check the code for efficiency. If there is a better way to get the same result, this should be noted in a file called IMPROVEMENTS.md.
\n
Kiro went to work and it created a hook with the description as shown below.
\n![\"A](\"/2025/07/kiro/kiro-hook.png\"\n)
\nInterestingly is that as it can’t trigger on the agent chat finishing a task, it figured a way around it by monitoring the task files (I’ll come back to these in a moment), but it’s nice to see it find a solution to my request. Now, these files will be edited for other reasons as well, and that’s presumably where the first line comes in. If I were to rewrite it, I would probably want to say something like “Verify that a task has been created or updated by running a git diff, if another change was made to the file, you will stop here.”
\nBecause that brings me to one thing I’ve found about these automated hooks, they can block you from continuing. Kiro can only have a single agent running in a window (using it simultaneously in multiple projects seems to be fine), so if you have a hook that will run after the update of a file it can block the execution of your main thread. Now, it will usually be nice and wait until the main thread is finished, but the moment that has a break (whether because it asks for permission or because it believes it’s finished), the hook will jump in and run.
\nI’ve had times where I couldn’t figure out why it wouldn’t continue when I prompt it, and usually this would be because Kiro was busy running its hooks. And maybe it asked for permission to run a command, or it was trying to fix some unit tests. You can’t know until you go there, which you can do via the Task list. Alternatively, you can also see which hook is running in the Kiro sidepanel by way of the spinning icon, but you’ll still need to go to the task list to interact with it.
\n![\"3](\"/2025/07/kiro/kiro-hooks-running.png\"\n)
\nAll in all, the hooks are a good addition, and I miss tools that don’t have them. Now, what are some things I’ve found work well? My usual “Update changelog and create commit” task is one that works well, because it’s a manual trigger. This particular one is one that I’ve got configured in pretty much all of my agentic coding tools, and my only gripe here is that I have to redefine it in each project instead of having it as a global command.
\nOther than that, because the hooks are blocking, you want the automated ones to be fast. That means something like “Go format and test” which runs all of the tests and linting, takes too long. Instead, I’ve since updated this to just run “go format”. Testing and linting are instead included in the agents steering file as part of all tasks. Another option to speed things up is to ask it to write its findings to a file, which you can then include in your work later on.
\nAnd now we come to the specs. Specs are basically a sort of information gathering phase. It works by providing your requirements, after which it will turn this into an EARS (Easy Approach to Requirements Syntax) style requirements document. An example of this is the below that was generated as part of one of my project improvements (specifically, an improvement to the automated AWS profile generation for Identity Center logins that I added recently).
\n### Requirement 1\n\n**User Story:** As an AWS CLI user, I want to specify how existing profiles should be handled during profile generation, so that I can control whether they are replaced or preserved.\n\n#### Acceptance Criteria\n\n 1. WHEN a user runs the profile generator THEN the system SHALL provide a flag to control existing profile behavior\n 2. IF the user specifies `--replace-existing` flag THEN the system SHALL replace existing profiles with new names based on the pattern\n 3. IF the user specifies `--skip-existing` flag THEN the system SHALL skip generating profiles for roles that already have profiles\n 4. WHEN no existing profile handling flag is provided THEN the system SHALL default to prompting the user for each conflict\n 5. IF both flags are provided THEN the system SHALL reject the command with a validation error\nOne downside I’ve found is that this is not an interactive process in itself. Kiro will generate the requirements document, and then ask you if it’s ok. It doesn’t ask questions in between about things that are unclear, but will make assumptions. Often this is fine, and you can make it adjust it (or even do it manually), but this is the part where input is good. That said, you can keep iterating on it during this phase.
\nIt’s the same thing with the design document that will be generated based on the requirements document. While this goes through your code, and looks at your requirements, it doesn’t actually seem to use the tools it has at its disposal unless you explicitly tell it to. That said, I’ve mostly had it work on tooling around AWS, so it’s possible it had that knowledge already available, but it might be worth telling it to use your MCPs etc. for getting up-to-date information. Interestingly, a lot of the decisions that seem to be made by Kiro in the requirements phase, actually show up with their rationale in the design document, along with any other decisions it makes for you. ALWAYS pay careful attention to this section, as it’s the most likely to bite you.
\nThe task list is the next thing to be created. This is a simple list of tasks, usually grouped and they will always have subtasks. The bottom item on the subtasks is actually a reference to the specific requirement from the requirements.md file, which is a bit confusing when you first see it as you may think it reflects dependencies between the tasks.
\n![\"An](\"/2025/07/kiro/kiro-tasks.png\"\n)
\nA very nice thing that you can see in the image above, is that you can trigger the tasks (and look at their run) directly from the tasks file. This is quite handy if you want to do a phased implementation and trigger each task manually and then review the work. Of course, you can also tell it to execute all tasks in the chat.
\nOne thing I would really like with the specs is a way to archive them, or to somehow mark them as done. Yes, I can move them out of the .kiro/specs directory manually, but I’d love to have a way within the kiro interface to see a difference between implemented specs and once that are still open. The list of my specs in some projects is starting to get large, and I fear that will only get bigger.
\nPersonally, I think the idea of these specs is great and have, ehm, been inspired by them when using other tools. While the plan mode in Cline or Claude Code is similar in idea, I really like how structured this works. Yes, I would probably like more back and forth with questions, but that can be replicated after the first time a document is generated. The one thing I haven’t really managed to find a good way to do is make additions to the requirements of a spec that I’ve already partially implemented and have that turned into good updates. To clarify, I have done so, but it makes it look quite messy, which is a shame as it makes them less nice for documentation purposes.
\nAs for what works best with these, it’s simply to follow the flow, but pay attention to the output and interact. Make sure that you agree with any assumptions it made, and otherwise change them. Tell it to tweak or even retry the generation of the design document or requirements. These are literally the most important parts of your work, because once the requirements and design are done well, you can easily tell the agent to implement the tasks without needing to look over its shoulder.
\nAnd speaking of the agent…
\n![\"Your](\"/2025/07/kiro/kiro-agent.png\"\n)
\nUnlike most agentic tools, Kiro’s agent comes with a simple question to start with: Do you want to go through the whole process of building a Spec, or just Vibe. With the Spec option you’ll be going through the process I described above, and with Vibe you’ll bypass that and have it act like a standard agent. To be clear, that seems to be the only difference between the two.\nAnd once you’ve chosen, there are a couple of options left: run in Autopilot or Supervised mode, and which model do you want to use: Claude Sonnet 4, or Claude Sonnet 3.7. Personally, I’m not entirely sure why 3.7 is still being offered as an option (to be fair, Sonnet 4 is still quite new), but if you prefer it over 4 then you can do so.
\n![\"Lots](\"/2025/07/kiro/kiro-context.png\"\n)
\nThere is also the option to add images, or a whole slew of context options, to your query. This works as expected and can be useful to provide the extra information the agent would want. Actually, I should probably call the agent by its name, Kiro, as that how it shows up in the chat interface.
\nAs for how well Kiro performs, it depends. Obviously, the better input you give it, the more you’ll get out of it, which is where the specs come in. Using the specs makes it a lot easier to get higher quality input, as you’ve already spent time working out the requirements and design. That said, I’ve had some great results, and I’ve had it generate a 2300 line bash script to serve as a GitHub Action. Below I’ve got a comparison though that will let you have a look yourself.
\nAn agent doesn’t work alone; it needs to interact with its tools, and that’s where Kiro isn’t quite there yet. On my main MacBook, I’ve been having some terminal integration issues. I don’t have this issue on my Mac Studio, so I fully expect that it’s something specific to this machine, but it’s frustrating as I don’t have the same problem in VSCode with either Cline or Copilot. This has improved, though, as most of the time it works fine, but I feel that having it be inconsistent is worse.
\nThe bigger issue is the MCP integration. Again, it’s improved a lot since my first look at Kiro, but it still doesn’t seem to support the newer features, such as remote or streamable MCP servers. This is a bit disappointing. Yes, as referenced in the linked issue there are workarounds, but we shouldn’t need hacky ways to do a standard action. I’m pretty confident that this will be resolved by the time the public preview is over, but it might be best if you (the person/AI reading this) upvotes that bugreport feature request. Some other things I’d like to see in the MCP integration is the ability to disable not just the whole MCP server, but specific commands in there.
Some other things to note: trusting commands is turning into a nice experience. As per the below picture, it asks you which of three levels of trust you’re happy to give it; the full command, the partial command (usually the base command and first argument), or the base command itself. This is a nice interface, but I’d like to see the ability to disallow specific commands. As an example, I might like to allow git * to run automatically, but not git commit. Right now, I would need to allow each subcommand before getting that result. Still, it’s actually a good interface for this.
![\"Trust](\"/2025/07/kiro/kiro-trust.png\"\n)
\nBut, the actual result of trusting commands is a bit unclear. Sometimes it works perfectly, other times it will keep asking you to trust the command. This is especially true when piping output to a file or when chaining commands. Even something like timeout 30 go test ./... can ignore the fact that I’ve whitelisted all go commands (and go test specifically) and ask for confirmation. As it happens, I’m not a fan of whitelisting a command like timeout * when I don’t know how that’s going to be checked. That said, it is possible to modify the whitelist manually so you can make that more suitable to your needs.
One thing I did yesterday (early today? it’s 1pm right now, so who knows), after Kiro was updated to the public release version, was ask both Kiro and Claude Code to implement a feature so I can compare the results. Would it have been a better comparison if I used something like Copilot or Cline? Probably, but with Copilot, that meant I’d have to approve commands every couple of minutes, and Cline would’ve used my actual money.
\nAnyway, using the Claude chat interface, I asked Claude Opus 4 to create a requirements and design document for implementing proper error handling and validation in my go-output library. It dutifully did so, and then I had both Kiro and Claude Code work on it in separate branches. For Kiro I went through the spec generation process (“Read the requirements and design documents found here and generate the requirements for the spec based on that” and follow this through). For Claude Code I simply asked it to generate the task list. And then I asked both to execute the generated tasks. To be clear, for this test I used minimal direction. I didn’t interact with the requirements part of the spec at all, and for the design document, the only change I requested was to include all the code snippets from the original design document.
\nI’ll let you be the judge regarding the respective PRs (found here for Claude and here the incomplete one for Kiro), but the experience is quite different. Claude Code was a lot more hands-off. I gave it the task, had to approve a couple of times for things being done in the session, and off it went. It was a lot faster too3. Kiro on the other hand, was having some terminal integration issues where I had to tell it to put the output of tests into a file. Kiro also spent a lot more time going around in circles trying to fix issues with the tests (and got stuck in a loop once), meaning that it was still working on the code when Claude Code had already refilled its credits and finished (and the public preview went up).
\nThere are many things to like about Kiro. The specs and hooks, especially, are very useful in having a more organised way of working with your agent. As I said before, I’ve happily incorporated the specs into my agentic flows, so obviously I like the way that works. Does that mean you should use Kiro? I’d say that’s up to you, but I highly recommend at least checking it out. Yes, it will still be rough around the edges, especially for MCP use, but that’s going to improve. And during the public preview it’s also free, which is of course the best price.
\nOr I just haven’t used it on a complex enough project ↩︎
\nYou have no idea how happy I was when this got resolved ↩︎
\nUntil I ran out of credit, obviously. The downside of being on a measly Pro subscription is that you frequently look askance at those upgrade prices and wonder if maybe it’s worth it. As I write this, I’ve got 10 minutes left until it resets. Not that I’m counting. ↩︎
\nWow, I just discovered this post never got published, so about 2 years late, but still potentially relevant I’m posting now about the talk I gave at the Golang Melbourne meetup 2 years ago. This talk is about my go-output module, which is the module I wrote to ensure the output formatting and methods between my various tools are the same. More background is in the talk, and the recording has now gone live. Also, as a bonus, I briefly explain Presso-Driven Development.
\nMy talk is the second one on the video but the embedded video uses a timestamp to get you straight there. However, if you’re interested in Go1 and particularly in mocking your tests, I recommend watching the preceding talk by my DigIO colleague Mirjam Uher.
\n\nIf you’re not interested, my talk might not be of great interest to you either. ↩︎
\nFor a fresh perspective, it’s often helpful to look at a problem or task from a different lens. Sometimes, this will be better; sometimes, it’s just a good way to refresh and renew your interest in something you’ve been doing for a while. In this post, we’ll be looking at how similar designing for the cloud is to writing a story. What are the similarities, what are some key differences, and how does that actually help in any way?
\n\n\nA long, long, time ago, in a galaxy far far away, a little hobbit lived in a hole in the ground.
\n
When you read the above sentence, you get an idea in your head about the story that is being told. Depending on the kind of stories you’ve experienced before, that idea will consist of spaceships and robots, or maybe dwarves, rings, and dragons1. Either way, that single sentence shows how much humans tend to think in stories. Stories are a powerful vehicle to transmit information, so let’s have a look at how we can apply that knowledge.
\nFirst, we should define a story. At an almost instinctual level, all of us can recognise a story, but what are the key ingredients? What kind of things can we find in both the latest Marvel blockbuster and an ancient story like The Iliad? In my mind, the main ones are the setting, protagonist, antagonist, any other characters, and challenges.
\n![\"Simple](\"/2025/05/cloud-design-as-a-story/overview.png\"\n)
\nThe setting is where things take place, the protagonist is who we’ll be rooting for, the antagonist actually is the baddie of the story, there will be side-characters of various importance, and the challenges are what need to be overcome to come to a happy conclusion. Of course, these are not the only things that can make up a story, and nor does every story contain each of these ingredients. But they are the things that make many stories interesting and ensure that we keep coming back for more.
\nSo, how does that translate to cloud design?
\n![\"A](\"/2025/05/cloud-design-as-a-story/hero.jpg\"\n)
\nLet’s start with our hero! The main character of any cloud design should, of course, be the application itself. Without an application, what would be the point of anything we design? That said, what is an application? Back in the olden days of web development2, an application was a piece of software that you built and then threw over the fence to the ops team to deploy on their precious production server. These days, however, there is a lot more to it, especially if you want to get the most out of running your application in the cloud. Whether that is for cost-efficiency, security, or any of the other reasons that are nicely encapsulated in the Well-Architected Framework.
\nWhen it comes to the protagonist in a story, the story’s author at least needs to know a lot about them. This includes the superficial things like their looks or the hero’s background, but also the more intrinsic elements that fuel their motivations. And the same goes for the story of your cloud design. What language is the application written in? Is it even a single language? What kind of architecture does it have, is it a micro-service or monolith? Will it run serverless, on containers, on VMs, or a combination of it all? Why were these choices made?
\n![\"Hand-drawn](\"/2025/05/cloud-design-as-a-story/protagonist.png\"\n)
\nAsking these questions is a big part of how you design your architecture, as every answer can lead you to a different path with different trade-offs and follow-up questions. For example, opting for a serverless micro-services architecture where the different parts of the application can be written in different languages will have a very different deployment method from a single monolithic application that needs to run on an x2iedn.32xlarge instance.
\nAnd of course, keep asking the questions:
\nAsking these questions is what you need to flesh out the hero of your story, but of course, there is more to a hero than just their intrinsic items. Where we live has a big impact on ourselves, and the same goes for our hero.
\n![\"A](\"/2025/05/cloud-design-as-a-story/scene.jpg\"\n)
\nSo, where does our hero live? In a fantasy story this can be an inspiring fantasy world with lush forests and giant mountain peaks surrounded by dragons, but for an application? I hate to break it, but unless you’re an amazing artist it probably won’t look as nice as that.\nAt the lowest level this is likely a VPC. While some serverless solutions can live outside of the VPC, in the end a VPC is still the basis that everything else builds on. But what does this look like in practice? VPCs have been around for so long there are plenty of best practices for it, and depending on your organisation you may have limited control over it. Exactly like in the real world.\nAs an example, let’s take a standard 3-tier subnet structure spread out over 3 Availability Zones.
\n![\"Sketch-style](\"/2025/05/cloud-design-as-a-story/vpc.png\"\n)
\nIn this environment your application and database live in a private and data or isolated subnet respectively. For your application this VPC is its home, but of course that doesn’t give much of a setting. So let’s look at the broader landscape. First, there needs to be a way to interact with the world outside of the home. A load balancer can serve as our front door, where we receive visitors, and we can have a NAT Gateway or Transit Gateway to facilitate our hero going out themselves. But this is still limited.
\nAgain, you end up asking a lot of questions to figure out the details:
\nThese questions will help you envision the home of your hero better, and thereby how it fits into it all. Of course, not every design will need to care about all of this, just like not every story involves a trip to the other side of the world to throw a ring in a mountain. But, what is a world without anyone in it? Let’s look at what fills this world.
\n![\"A](\"/2025/05/cloud-design-as-a-story/cast.jpg\"\n)
\nA solitary journey can be a lonely one, and generally speaking stories and design are about the interactions. Who our hero meets is a major part of the story, and that doesn’t change when our hero is an application. There are other applications in your environment that it will interact with, third-party services, and of course those who will be using the application.
\nSo the main question you have to ask here is “Who does our hero interact with?”. But in addition to that, you also need to figure out how that interaction will take place. In a story that has a medieval setting, you don’t usually have instant communication over long distances3 and similarly you have to take what’s available into account. Do they communicate over VPC Peering? A Transit Gateway? Or do you use PrivateLink connections to various internal and external endpoints? What kind of protocols are used, both for interactions with the end-users and other applications/APIs, and what’s the impact for that on your security groups and NACLs? Who do you need to send alarms to when something goes wrong?
\nSpeaking of things going wrong…
\n![\"Close-up](\"/2025/05/cloud-design-as-a-story/villain.jpg\"\n)
\nMany stories have a villain, and the story of cloud design is no stranger to this. Unfortunately, in the case of a villain, cloud design is more likely to fall in the mystery genre as we don’t really know who or what our protagonist will have to face. There are a great many cyber security threats out there, all of whom have different motives and strategies. So instead of trying to figure out who we need to protect against, you should instead focus on what you’re protecting.
\nAre you protecting your data? Or is there critical infrastructure that can’t go down? There are many different things that can be targeted by an attack, and you need to determine what it is that’s most important to protect. And once you know what’s most important, you can ensure that your cloud environment prioritises protecting that. Of course, don’t ignore everything else, and always keep in mind that a layered defense, or defense in depth, will get the best results. Villains aren’t the only challenge you may need to confront however, so let’s see what else is out there.
\n![\"A](\"/2025/05/cloud-design-as-a-story/challenges.jpg\"\n)
\nNo story would be complete without a challenge to overcome. As it happens, the challenges we need to overcome don’t involve throwing an evil ring into a volcano or blowing up a moon-sized space station. Instead when dealing with the cloud we have to face other types of challenges. Which can be challenging themselves4!
\nThese challenges can take many forms and can be either internal or external. Internal challenges can include things like having to deal with an old application that doesn’t support modern architectures. Or you are faced with a release and approval process that requires many meetings and literal months before a change can be deployed to production. Maybe there are business or security requirements that you need to follow.
\nAnd then there are the external challenges. The biggest challenge here is often regulation, especially if you’re working in industries that have a lot of oversight, and the only way to face that is by doing the work to comply. Similarly, whether by regulation or otherwise, you may not have a choice which AWS region you can use, and whether that region has that one feature or service you really need.
\nBut if you think of these as part of a story, maybe, just maybe, you will find a way to deal with them all and have your happy ending.
\n![\"A](\"/2025/05/cloud-design-as-a-story/happy.jpg\"\n)
\nAnd then our story is complete! We’ve released our cloud infrastructure; our hero is happy and talking with all their friends in our beautiful world. The antagonist is kept away, and we’ve overcome all of our challenges. And so our story ends on a happy note and we can watch the credits roll.
\nExcept… this is the real world and the point where the metaphor breaks down a bit5. Because, unlike a story with a defined beginning and end, the world of the cloud is always changing and evolving. Because of that you have to keep up to date to stay relevant; this can include updating your protection against new plots the antagonist may be hatching6, but the architecture you use can also become deprecated and replaced by something better.
\nOne other key difference between cloud design and crafting a story is that you’re not alone. Where a story is created by a single person, or small group of people, for cloud design you have the whole world as your ally. Obviously this includes the colleagues you work with, but there is your local community7 you can discuss things with, there is also a lot of documentation out there, and people who have faced the same questions. And nobody minds if you use the same ideas and solutions for your design.
\nAnd with regard to using other people’s solutions, there is a quote attributed to William Faulkner that is as true for a story as it is for your cloud design:
\n\n\nIn writing, you must kill all your darlings
\n
In practical terms, what this means is that you shouldn’t hold too tightly onto anything you’ve built. Yes, you may have sweat blood, tears, and lots of AI tokens over building a working solution, but if AWS releases a service that does almost exactly what you need, you should take a very good look at it. Maintaining your custom solution is harder than using a service, and will take more of your time - time that can be better utilised elsewhere.
\nAnd of course, there is the other thing we’ve all experienced when it comes to cloud design8. Once you’re done building something, you get to do it all over again for the next project. But maybe, for your next project consider using a framework like this to envision your design as part of a story. Think of your hero, their place in the world, the friends they make along the way, who might be opposing them, and what other challenges will be faced along the way.
\n![\"A](\"/2025/05/cloud-design-as-a-story/sequel.jpg\"\n)
\nI’d love to hear in the comments, or elsewhere, if you have thought of design like this and if this take is useful. Or just fun and you would like me to expand on it. And, if you’re interested, I’ve given a couple of talks about this topic. The first one embedded below was at the Melbourne AWS User Group in March last year9, and the second was a shorter version at ADAConf in Melbourne.
\n\n\nOr you could be an overzealous copyright lawyer and start to wonder if there’s money to be made here. Spoiler: there isn’t. ↩︎
\nLet’s not go back all the way to when we bought our applications on floppies or CDs in a store. That just makes me feel way too old. ↩︎
\nUnless magic is involved. Pro tip: don’t rely on magic for your cloud architecture. ↩︎
\nJust, you know, without having to risk your life. ↩︎
\nOr does it? ↩︎
\nOr a new antagonist is introduced. ↩︎
\nAs a meetup organiser, I can objectively say that they’re a great way to learn and meet people. ↩︎
\nEspecially if you’re a consultant like me. ↩︎
\nGuess who took forever to write this up? ↩︎
\nEarlier this morning, AWS released PartyRock, an Amazon Bedrock Playground. In short, this is a fun little playground that lets you create miniature apps using both text and image generation. Through the Community Builders program, I got access a couple of days ago and have been playing around with it. As a spoiler, it’s the most fun I’ve had playing with AWS tools for a very long time. So, grab your air guitar and have a look at what this most resplendent app actually does.
\nFirst, let’s be very clear here that this is a playground. It’s clearly meant to highlight fun things you can do with generative AI, and not necessarily meant to be something you use for what you might call “proper work”. However, it is a lot of fun to make these little apps, and you can get stuck on doing your best to improve them very quickly. Or just seeing what it comes up with. That said, I have seen some more practical examples of what it can do including code generation, trip planning, and similar things. Is a PartyRock app the best way to do these things? I don’t know and I don’t care as I’m sure that the people who built it had fun.
\nAnyway, to explain how it works, let’s go through a quick example. To stay in the theme of PartyRock, we’ll want Bill and Ted (from the movie series1) to explain things to us. Having logged into PartyRock we simply click on the “Generate App” button. This brings up the App Builder, where you can provide a description of your app. Based on that description, PartyRock will then literally generate an app for you.
\n![\"An](\"/2023/11/party-on-dude/partyrock-bill-ted-generate.png\"\n)
\nGenerating the app takes about a minute or so and a fun little detail is that the generated app won’t be the same every time. If you provide the same prompt multiple times, you will get different apps. And of course, as with all generative AI interactions, slightly changing the wording you use can give a very different result. As an example, for the above prompt of “Explain or describe the concept asked about by the user as a conversation between Bill and Ted from the Bill and Ted movie series”, I’ve received a chat interface, separate output widgets for Bill and Ted, and a single output widget where a conversation takes place. Obviously though, afterwards you can change the generated app however you like. (At least, within the constraints of PartyRock).
\nI’m not going to take you through the editing UI; it’s easy to understand how everything works, and just trying it out for a minute or two will explain it better than I probably can. Also, Jeff Bar literally went through every step in the announcement post. Suffice it to say, I went with the single output widget for the conversation and decided to add an image generation widget that uses the prompt “A portrait of Bill and Ted from the Bill and Ted movies looking confused about @What would you like to know about”. The @-sign there refers to that being a field it will get the input from.
\n![\"An](\"/2023/11/party-on-dude/partyrock-billted-swf.png\"\n)
\nAs you can see, the image isn’t great, but the provided answer is both fun2 and informative. And making this only took a couple of minutes (not counting the 30 minutes experimenting with prompts to see what kind of answers came out of it). Yes, you can choose which model is used for the text generation, but other than that you can only change the prompts and the sizing and placement of the widgets.
\nWhich brings us to the most obvious thing about PartyRock: the look and feel. Even if you haven’t looked at the announcement post, it should be pretty obvious that this doesn’t run in the AWS Console. The look and feel are certainly different from what we’re used to with AWS, and may not be to everyone’s taste. But then, what is? It does make you feel like you’re working in a sandbox environment, and the grid for the widgets works pretty well, so there are no complaints from me about that.
\nNow, the service isn’t perfect obviously. You can get the occasional error, especially with image generation, but there is very active development going on3, and I don’t want to dive into the issues I’ve encountered because they could all be gone already by the time this is posted. Also, let’s keep it fun.
\nSo, instead, let me highlight some of the fun things I’ve built with this by showing examples. Which reminds me, did I mention there is a snapshot feature that generates a URL you can share that contains your inputs and the generated outputs? I really love this feature as it’s a great way to show the results you can get.
\nSo, let’s have a look:
\nFrom my Bill and Ted explainer app, Bill and Ted explaining the heat death of the universe, and then suddenly deciding they should do something about that. I feel like this shows so well how things can take some very interesting twists.
\nFrom my Fantastical Services app that generates a fantasy background for AWS Services:
\n![\"An](\"/2023/11/party-on-dude/partyrock-simpledb.png\"\n)
\nFrom my Monkey Mayhem Generator, which I created to create short fun stories for my daughter:
\n\nAnd last, but certainly not least, my Meetup Intro generator tells you exactly why you should come to our meetup next week.
\nSo, while I’m sure there are more useful playgrounds out there than mine, I had a lot of fun creating them and highly recommend everyone to do the same. Luckily for all of us, it’s a free service right now that doesn’t even require an AWS account.
\nSo, give it a shot by building something at PartyRock. And of course, be excellent to each other!
\nIf you haven’t watched any of the Bill and Ted movies, feel free to watch them and be fully consumed by the insane idea of a couple of time-travelling, failed musicians whose music will supposedly save the world in the future. ↩︎
\nAssuming you think the way the characters talk is fun. ↩︎
\nOne bug I reported was a misspelled error message, a couple minutes later I got the same error but this time it had both the correct and original misspelled message. After that I only got the fixed message. So, very quick turnaround. ↩︎
\nIn January last year there was an announcement for the X2iezn instance type going GA. When I saw that, two thoughts went through my head: “How do I pronounce this on a podcast?” and “What does that even mean”?
\nIn this case, the announcement post actually explained what everything meant. Clearly, someone at AWS understood that this block of letters might be a bit confusing. But I figured, let’s get this cleared up in the future by doing a bit of research and getting it all in a single place that I can then look up to see what it means in the future. And while I could keep this somewhere only for me1, it’s just as easy to keep it public. Hopefully, this will help some of you as well. I’ll try to keep this updated as new types and attributes come in.
\nI have used two main sources of documentation for this, the “marketing” page about EC2 instance details and the documentation page about the same. These have slightly conflicting data in places, and are clearly not kept in sync, but as the marketing page seems to contain more detailed information about the instances I have used that as the leading source, with the documentation being used to fill in any gaps.
\nIf you’re not interested in the whole document, but are looking for something that gives you all the useful information, you can download the below cheatsheet (pdf) I made that explains the various instance families, attributes, and sizes.
\n![\"The](\"/2023/05/ec2-instance-naming-explained/instance-cheatsheet.png\"\n)
\nLet’s break down how an instance is put together and then go into the details for each of these sections. I’ll use the X2iezn as the source for this explanation.
\nAn instance type consists of four parts:
\nSome of these are a lot easier to understand than others, but let’s go through them one by one.
\nOn the EC2 types website, AWS has grouped these families into 6 groups. To make things easier, I’ll use these groups as well even though two of them only contain a single family. Instance families started out being represented by a single letter, which at times became a bit awkward and unwieldy, and lately AWS has started using longer family names.
\nOne note before we look at the families concerning capitalization of the families. AWS is not consistent about this and often shows a family and generation as starting with an uppercase letter (e.g. T3 or Mac1), but will show a specific size using lowercase (e.g. t3.medium or mac1.metal). There are exceptions to this as well, and instead of trying to make sense of it all, I’m following the pattern I name above where I capitalize the families unless it’s for a specific size.
\nThese are the usual workhorses and the standard instance types you’re most likely to use unless you’ve done some investigating and work on figuring out what’s best for your environment. It consists of the following types:
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| Mac | \nPowered by Apple Mac mini computers | \nThe first one on the list, but already doesn’t follow the standard pattern of a single letter. Mac instances (in AWS) have only existed since late 2020, and it’s unlikely Apple would allow a different name. | \n
| T | \nBurstable general-purpose instance types | \nBasically the same as m instances, but with only a portion of the CPU always usable and the rest through burst credits. Interestingly, in the documentation it is said the T stands for Turbo, but this term isn’t used anywhere else | \n
| M | \nGeneral-purpose instance type | \nThe standard general-purpose instance family | \n
| A | \nThe first EC2 instances powered by AWS Graviton Processors | \nBasically Graviton 1, which started as its own family until it was turned into a type detail. Obviously an outdated family | \n
Compute optimized instances generally have half the memory of an equivalent general purpose instance. Which delivers a cost-saving for applications that don’t need as much memory. This
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| C | \nAn ideal fit for compute-intensive workloads | \nThe standard compute-optimized family | \n
Memory optimized instances have a multiple amount of the memory of an equivalent general purpose instance. Which makes it a good fit for anything that needs lots of memory, like Java applications.
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| R | \nOptimized for memory-intensive applications | \nThe standard memory-optimized instance family | \n
| X | \nOptimized for enterprise-class databases and in-memory applications | \nDepending on the generation and type details, these have multiple times the memory of the equivalent R instance | \n
| U / High Memory | \nPurpose built to run large in-memory databases | \nRidiculous amounts of memory, and the weirdest naming scheme of any instance type | \n
| Z | \nOffers both high compute capacity and a high memory footprint | \nBasically an R instance, but with a high-frequency processor. If this came out now it probably would be a variant of R instances | \n
As you can see in the table, there are two interesting families here. The Z instances, which as we’ll see later is now just a attribute but like the A family got its own type when it came out. It should be fairly clear these are on their way out and will soon enough become a variant of the R6 or R7 generations.
\nThe other weird one is the High Memory instances. These have a different naming scheme, where the name actually shows how much memory they have; u-6tb to u-24tb. These are still part of the family name, but more a sub-family and not quite shown as such as some others we’ll be discussing later.
These are instance types optimized for specific use cases, such as machine learning or video transcoding. Each of them is special in its own way, and to really know what suits you best you’re better off looking into the details.
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| P | \nGPU-based instances | \nThese use a P instead of G because that was already in use | \n
| DL | \nPowered by Gaudi accelerators from Habana Labs | \nOur first 2 letter instance type, but not a subfamily of the D instances which are specialized in HDD storage | \n
| Trn | \nPowered by AWS Trainium | \nAn instance type for training your ML models, developed by AWS and one of the view that has a name longer than a single letter | \n
| Inf | \nBuilt from the ground up to support machine learning inference applications | \nUses Inferentia chips in addition to the usual Intel ones. Probably the first one to use a longer family name | \n
| G | \nDesigned to accelerate graphics-intensive applications and machine learning inference | \nHas been longer around than the P instances, and stole the G name | \n
| F | \nCustomizable hardware acceleration with field programmable gate arrays | \nFPGAs are cool, and obviously this is a good name | \n
| VT | \nDesigned to deliver low cost real-time video transcoding | \nAnother 2 letter name. At least this one makes sense, but why not use V? | \n
While the group as a whole is called accelerated computing, it’s clear that most of the instance families are focused on one part of the AI/ML stack or another. With the increasing focus on AI, I expect that these instance types receive a lot of attention and will have new types get introduced soon enough. I also wouldn’t be surprised if at that point AWS breaks the AI/ML focused families out into their own category like they did for the HPC ones which originally fell under Compute Optimized.
\nThese instance families are all about the local storage. Either NVMe SSDs or classic HDD.
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| I | \nProvides Non-Volatile Memory Express (NVMe) SSD-backed instance storage | \nIf you want lots of fast local storage, these are what you need. The I stands for I/O, although they could have chosen S | \n
| D | \nFeature up to 48 TB of HDD-based local storage | \nIf you really need a lot of (slow) local storage. D presumably stands for disk. | \n
| H | \nFeature up to 16 TB of HDD-based local storage, deliver high disk throughput, and a balance of compute and memory | \nI’m guessing H stands for High throughput, but I suspect this family is on its way out as it hasn’t received any updates and the processor it uses is now 7 years old | \n
The I family of instances is currently the only one that has a modifier in front of the generation number, specifically the Im4gn and Is4gen instances. We’ve seen a couple of instance types with longer names (Mac, Trn), but this is the only case of an actual sub-family defined in this way. I hope this was done on purpose and not that someone forgot how the naming scheme works.
\n| Sub-family | \nAWS Description | \n
|---|---|
| Im | \nProvide the best price performance for storage-intensive workloads | \n
| Is | \nOffers the lowest cost per TB of SSD storage and the highest density of SSD storage per vCPU | \n
That said, I have no idea what the m and s stand for. It’s not medium and small, as the Is instances have more memory than Im. It’s a bit of a stretch, but the only thing I can think of is based on the announcement where the use cases are described and the Im4gn is described as being good for databases like MySQL and the Is4gen are good for streaming. Someone, please tell me the actual meaning behind these letters so I can get that thought out of my head.
\nThese are instances focused on delivering the best result for High-Performance Compute workloads. Not much else to say, as it’s all in the name. Weirdly enough, depending on which part of the official documentation you look at these can be categorised as HPC Optimised or as part of the Compute Optimised section. As stated above, I’ve chosen to use the more complete appearing marketing page as the main source, which is why they are in this section.
\n| Family | \nAWS Description | \nAdditional notes | \n
|---|---|---|
| Hpc | \nOptimized for tightly coupled, compute-intensive, high performance computing workloads | \nA new instance type from January 2022, that’s so powerful it couldn’t be reduced to a single letter | \n
The generation of a family. Higher is newer (M7 is newer than M6), but newer families will obviously have lower generation numbers (e.g. Trn1 is newer than M6), so these generation numbers can’t be compared between families. AWS does use this to promote their Graviton processors by always making them part of a new generation of processors. So Graviton2 came out first in the M6g when Intel was still only on M5, and Graviton3 can be found on the M/C/R7g while no equivalent 7th generation Intel or AMD instances exist.
\nThe processor type letter didn’t exist until Graviton2 came out, and it didn’t even include options for Intel or AMD until the M6i and M6a respectively became available. Unfortunately, this is also not consistently applied even for new instances except when they run Graviton which always gets a g.
Typically everything without an attribute is Intel, as for a very long time that was the only option available, but that’s not always the case. An example of this is the new G5 and G5g families. G5g is clearly Graviton, but unless you are familiar with the G series of instances you might be mistaken in your believe that G5 is an Intel type2.
\nThe chosen letters in this case are clear, and already mentioned above but let me repeat them anyway:
\ni: Intela: AMDg: GravitonTo be clear, this doesn’t indicate anything other than the type of chip. So Graviton2 and Graviton3 are both represented by a g, and when in the future i and a have different chipsets they will still keep it to just i and a.
One thing to note here, Mac instances come in both Intel and Apple Silicon (aka ARM) processor types but AWS doesn’t make a distinction there. It’s simply that the mac1.metal instances use Intel while mac2.metal instances use Apple Silicon. As it’s unlikely Apple will ship any more Mac Minis with Intel processors that probably won’t be a big concern going forward.
\nThere is one attribute that means different things based on the instance family group. This is the attribute e.
e stands for extra3, specifically for memory or storage. The only instance families with this attribute are in the memory optimized and storage optimized groups and they exacerbate the values of the group. So, for memory optimized instances the amount of memory is doubled compared to the same type without an e, and for storage optimized you get more storage (there is no fixed pattern for that one that I could figure out).
| attribute | \nmeaning | \n
|---|---|
| i | \nThis is an Intel instance | \n
| a | \nThis is an AMD instance | \n
| g | \nThis is a Graviton instance | \n
| b | \nThis instance has higher EBS bandwidth | \n
| n | \nThis instance has more network bandwidth | \n
| t | \nThis instance has a higher CPU clock speed | \n
| d | \nThis instance has local NVMe storage | \n
| e | \nThis instance has its defining property improved | \n
At first glance, sizing is very straightforward. They basically work on the concept of t-shirt sizes4 and the size is always based on the number of vCPUs available. Medium size instances have a single vCPU, large has 2 vCPUs, xlarge has 4, and from there you multiply by the number in front of it so 2xlarge has 8, etc. The one exception to this is the T-type instances. These don’t go lower than 2 vCPUs even for the nano sizing. However, there the size is mostly based on vCPU credits that determine how long you can use the full capacity of the instance.
\nAn instance family type then usually has a multiplier for the amount of memory; for the M types this is 4, meaning you get 4GiB of memory for each vCPU core, with the C types it is 2, and with R it is 8.
\nThe one complication that comes with sizing is all the special modifiers that can have differences based on the size of the instance. For example, for C6gn, the network optimized part scales up with the instance size. So a c6gn.8xlarge has 50Gbps network bandwidth, but a c6gn.16xlarge has 100Gbps.
\nAnd then there is the “metal” size. This isn’t really a size as much as a different type of instance as it doesn’t offer you a virtualised instance but what AWS terms “bare metal”. This means you get access to the full underlying hardware and don’t share it with anyone. This also means you don’t get a choice about the sizing of a metal instance as it’s always the maximum possible size for the instance family. In addition, however, you don’t get any overhead caused by virtualisation.
\nAnd have, considering I wrote most of this a year ago before I forgot about it… ↩︎
\nI hope you realise that means it’s AMD seeing there are only 3 types. ↩︎
\nProving that it’s not just me who comes up with weird names ↩︎
\nOk, not micro or nano and I admit not seeing many 112xlarge t-shirts either. ↩︎
\nLast Thursday I gave a presentation at the Melbourne Golang meetup. In this presentation, I had a slide that contained a warning about “Presso-Driven Development”. Presso-Driven Development, or PDD, is a term I use to explain how some of my code comes into being. To make my life easier, and see how it fares in the wider world, I figured it’s time to put a proper definition out there.
\n![\"A](\"/2023/05/presso-driven-development/pddslide.png\"\n)
\nDisclaimer; this would likely be called Presentation-Driven Development if I lived anywhere other than Australia, which would make it sound incredibly boring.
\nA presso1 represents any form of showing people something you’ve built. This can be in person or online, live or recorded, formal or casual, interactive or one-directional. There needs to be a time constraint attached to it, however; a set date when you need to deliver this by.
\nPresso-Driven Development takes place when giving a presso is the driving force behind building a feature. The idea for the feature already exists before you plan to give the presso, but giving the presso depends on the feature being implemented. This feature represents something you want to implement, whether that is a small functionality in an application or a complete application, but it is something that can be used to achieve an actual task and will be functional beyond the presso itself. And of course, once you have completed building what you need for the presso, you don’t touch it again until after the presso to prevent bugs2.
\nNot required, but a recommended best-practice for PDD, is to make sure your feature includes at least one visually fun property that doesn’t add value other than looking cool.
\nNow, you might ask, why would anyone build features after they’ve decided to give a talk on it? Isn’t that a bit backwards?
\nYes, yes it is. I understand that a more logical sequence of events is “Think of something cool/useful” -> “Write an implementation of this” -> “Give a talk about it”. And that makes perfect sense when you live in an ideal world and have lots of spare time.
\nIn a world filled with other activities3 however, this is more likely to be: “Think of something cool/useful” -> “Don’t have time to implement it” -> “See an opportunity to give a talk about that cool thing” -> “Propose that talk” -> “Talk is accepted” -> “Realise you should probably make sure it actually works before the talk” -> “Finally implement it” -> “Give the talk”. The timeframe for this can also be very long as sometimes an idea can stay in that second step for months, if not years, and is completely dependent on everything else going on in your life. The below diagram attempts to show the different paths for an ideal world where everyone has all the time, and the one we happen to live in.
\n![\"A](\"/2023/05/presso-driven-development/pdd-diagram.png\"\n)
\nIn short, presso-driven development means that you can actually implement these features/tools that you want to play around with.
\nMost “Something”-Driven Development methods are focused on getting high-quality code, documentation, tests, etc. That… is not a driving factor for Presso-Driven Development. As should be clear from the above background section, PDD generally happens when you don’t have a lot of time to get your desired feature working. In the unlikely event that you’re finished building the feature but have spare time before your presso, there is no reason not to add documentation, tests, etc. As long as you don’t touch the code itself anymore! After all, to reiterate, once it works, don’t touch it until after your presso. Most importantly though, don’t forget that you could spend that time working on the cool but useless visual property!
\nIt is my belief that Presso-Driven Development is a very common development strategy, and I have probably missed some parts of this. I would love to keep this post updated with details on how PDD is used in the world, so if this strikes a chord with you please let me know either here in the comments, on Mastodon as @arjen@ig.nore.me, on LinkedIn, or in person. Similarly, if you think of a way to improve the definition let me know.
\nOr presentation, if you really must use such a long term. ↩︎
\nUnless something changes in the service you’re depending on, in which case you also shouldn’t forget to add a slide complaining about that. ↩︎
\nWhether that is work, social activities, or the never-ending task of being a parent. ↩︎
\nFor this episode, Arjen is joined by Jason and Matt to discuss the news from the first quarter of 2023. As that is a bit much, they decide to focus on the Melbourne region first and then pick some of their favourites round-robin style.
\nThis episode was actually recorded on 23 March, so it’s been a while…
\nYou can find the full shownotes and links at the episode page.
\n", "summary": "For this episode, Arjen is joined by Jason and Matt to discuss the news from the first quarter of 2023. As that is a bit much, they decide to focus on the Melbourne region first and then pick some of their favourites round-robin style.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "image": "https://ig.nore.me/img/awsug.png", "date_published": "2023-04-30T22:48:32+10:00", "date_modified": "2023-04-30T22:48:32+10:00" }, { "id": "https://ig.nore.me/presentations/2023/04/automating-a-cloudformation-deployment-visualisation-dashboard/", "title": "Automating a CloudFormation deployment visualisation dashboard", "url": "https://ig.nore.me/presentations/2023/04/automating-a-cloudformation-deployment-visualisation-dashboard/", "content_html": "As part of the Community Builders program that I joined recently, the APJ (Asia-Pacific + Japan) region recently started a monthly Open Mic night. These were already happening semi-regularly but have now become a monthly thing. So I figured I’d throw my hat in the ring to speak at the first one and my talk was selected. In this talk I’m giving a demo of fog’s automated deployment report functionality. It’s not the best talk I’ve given, but it was still fun and hopefully it’s interesting. The intro for the talk starts about 90 seconds in if you don’t want to sit through the explanation of the Open Mic. Also, there’s a couple of interesting talks following it so stay no need to stop after my talk has finished.
\n\n", "summary": "As part of the Community Builders program that I joined recently, the APJ (Asia-Pacific + Japan) region recently started a monthly Open Mic night. These were already happening semi-regularly but have now become a monthly thing. So I figured I’d throw my hat in the ring to speak at the first one and my talk was selected. In this talk I’m giving a demo of fog’s automated deployment report functionality. It’s not the best talk I’ve given, but it was still fun and hopefully it’s interesting.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "image": "https://ig.nore.me/presentations/2023/04/automating-a-cloudformation-deployment-visualisation-dashboard/screenshot.png", "date_published": "2023-04-27T22:50:25+10:00", "date_modified": "2023-04-27T22:50:25+10:00" }, { "id": "https://ig.nore.me/2023/02/aws-community-builder/", "title": "AWS Community Builder", "url": "https://ig.nore.me/2023/02/aws-community-builder/", "content_html": "I’m happy to announce that I’m now part of the AWS Community Builder program, in the Dev Tools category. In case you’re not familiar with the Community Builders program, it is a program that offers resources, education, and networking opportunities for those who are active in the AWS community. It is also a program you need to apply for yourself, during one of the periods it’s open for applications. In the meantime, there is the option to put yourself on the waitlist.
\n![\"\"](\"cb-social.png\")
In a way though, that makes this the third such program I’m a member of. First, I’ve been in the AWS Ambassador (formerly APN Ambassador1) program for a long time, and the Cloud Warrior program that fed into it before that. Then in 2019 I became an AWS User Group Leader, or Organiser, which while not entirely defined as a program similar to the others is sort of run like it. So, aside from just mentioning that I’m now a Community Builder this post will go over some of my thinking about joining the program. And at the end I’ve included some updates regarding social media.
\nBecause it sounded like fun. I know a fair number of people both in Australia and elsewhere who have been part of the Community Builders program for a while now and they are enjoying it. In addition, it’s something new to try out and see how the program works. And joining it was relatively straightforward, it’s basically a form asking you why you want to join the program, which two categories you want to be in (Dev Tools was my first choice) and you need to show that you’ve actually contributed to the community. Despite my lack of writing on this site, I have done quite a bit of that ,and obviously anything that counts as a community contribution for the Ambassador program counts for the Community Builders program as well, including all the things I do as a User Group Leader2.
\nThat’s a good question. If it’s relatively easy for me to enter this program why would I wait until now? The answer is two-fold. On the one hand there is the excuse of having a young child and not wanting to pick up more things like this, but the truth is more likely that I was too lazy and didn’t want to join yet another program. But as I said, I kept hearing good things about the program and figured it was time to see for myself what’s in there.
\nI can show another badge on this website and get some more swag? No, in all seriousness I’m actually looking forward to learning more about what the program actually means in practical terms. While hearing things is nice and the website has a nice spiel, it’s always finding out what it actually does that matters. To be honest though, mostly I’m hoping to meet new friends who I can talk to about a lot of this fun stuff and maybe collaborate on some things on.
\nI’m so glad you asked! I haven’t been very good about posting things here and while I intend to catch up on that, I’ll put a summary of some of these things here:
\nObviously I’m still running (with help of course) the Melbourne AWS User Group every month, but in addition I’m one of the organisers for DevOpsDays Melbourne. A conference we started organising in 2019 with the plan to run it in 2020. And then the world fell apart, but it’s now only 2.5 weeks out (16-17 March). There are still tickets available, so come along!
\nThe open source code I’ve worked on the most over the past couple of years is likely fog. I’ve spoken about this at various venues, but in essence it’s a tool for managing your CloudFormation. It allows you to CloudFormation deployments in a better way, but it also lets you get an overview of deployments that have happened. This is a tool that originally started its life as a bash script many years ago that I finally ended up re-writing in Go to make it more useful and nicer to work with. And I’m still working on improving it, currently I’m adding a functionality that provides better drift detection than the standard one from CloudFormation3.
\nIn addition, I’ve been making improvements to my still terribly named awstools and ended up splitting the formatting/output part of it into its module named go-output4 that is used by both awstools and fog.
\nWhile the Melbourne AWS User Group podcast had a bit of a hiatus, we started season 2 back up after re:Invent. The cast of the podcast has changed a bit, but we have a good group of people who like to make fun of things where needed, but also know what they’re talking about. So, exactly the kind of thing you want in a podcast. The first episode of season two is embedded below, and you can follow through for the second one (a third should be recorded not too long in the future).
\n\nI’ve given a couple talks, but my favourite that I haven’t put up here yet was at the User Group in Perth. Not only did it give me the chance to visit Western Australia but it was great to interact with the community there.
\n\nApologies for tacking this on at the bottom here, but this way I don’t need to do a separate post about it. Basically, many of you will likely have heard that Twitter is currently in a weird state. In fact, between no longer being able to use my preferred third-party clients and other questionable decisions, it’s in a state where I personally don’t really want to use it anymore.
\nIf you follow updates to this site through the bot account on Twitter, that should still work as I write this (and otherwise you might never read this), but I honestly don’t know if it will continue to work long-term. I do know that I won’t be paying to use the API. So, instead I set up a personal Mastodon instance that I will be using for my personal account (@arjen@ig.nore.me) as well as any bot accounts, like one for this site (@feed@ig.nore.me), that I want to have. In addition, I will be more active on LinkedIn and have created a “company” page for the website as well where everything will automatically be posted.
\nThis is a name change that I 100% support by the way. Being an APN (AWS Partner Network) Ambassador always required explanations of what that actually is as nobody outside of the partner network actually knows that acronym, and AWS APN was obviously redundant enough that I didn’t want to use it. ↩︎
\nYes, it almost seems like cheating. ↩︎
\nThis is very much in a “it works on my machine” state right now, but I’m quite happy with the way it handles tag changes and shows drift for NACL entries. ↩︎
\nI suspect that between these two my naming skills are clearly of a level that I would fit right in with the people naming services at AWS. ↩︎
\nAt the Serverless meetup in February I gave an overview of what was announced at re:Invent (and in the pre:Invent period). I tried to keep it entertaining instead of a simple rundown, so have a look and hopefully you’ll see something interesting.
\n\n", "summary": "At the Serverless meetup in February I gave an overview of what was announced at re:Invent (and in the pre:Invent period). I tried to keep it entertaining instead of a simple rundown, so have a look and hopefully you’ll see something interesting.\n ", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "image": "https://ig.nore.me/presentations/2023/02/all-the-cool-stuff-from-re-invent-aka-a-serverless-re-cap/serverless-or-not.png", "date_published": "2023-02-21T22:24:10+11:00", "date_modified": "2023-02-21T22:24:10+11:00" }, { "id": "https://ig.nore.me/2022/12/2022-aws-re-invent-important-highlights-between-the-keynotes/", "title": "2022 AWS Re:Invent; Important Highlights Between the Keynotes", "url": "https://ig.nore.me/2022/12/2022-aws-re-invent-important-highlights-between-the-keynotes/", "content_html": "This was originally posted on the CMD Solutions blog, as all of Mantel Group’s brands (of which CMD was one) combined, some older blog posts got lost in the shuffle or links changed, so I’m replacing my linkblogs with the original content to ensure they stick around.
\nIf you’ve attended a conference before, you know that the most exciting things don’t necessarily happen in the sessions themselves. Unfortunately, I’m not in Las Vegas, so I can’t enjoy the hallway track. Still, AWS is kind enough to offer an alternative by releasing some of its announcements outside the keynotes. Best of all, you can see these announcements happening throughout the day without needing to get up at 3 am! Below are some of the announcements that stood out to us.
\nZero-trust networking has become more popular in recent times as it is a secure way to manage access to otherwise inaccessible environments without the use of a VPN (Virtual Private Network). So it shouldn’t be too surprising that AWS has been working on their own version of it to make life easier for those of us who run our applications there. Which is where Verified Access comes in.
\nBefore looking into the service itself, let’s go over what makes a zero-trust solution potentially better than a VPN. In my opinion, there are several advantages:
\nSo with that in mind, how does Verified Access solve these issues? First, it’s a managed service so you don’t need to worry about maintaining it. As for granular access, you create an endpoint for every application and can then use the properties of your IAM Identity Center (or an OpenID Connect provider) to define access. This also means you can automate this and make it part of your code. Unfortunately, it doesn’t look like there is CloudFormation support just yet, but I’m hopeful that will arrive soon. As for the last point, all you need to access the applications is a browser (limited to Chrome and Firefox right now) to access the service and it will work.
\nAnd the best part? Even though it’s still in preview (and not recommended for production workloads) it’s already available right here in our Sydney region.
\nVPC Lattice is a new service network. What this allows you to do is easily connect different kinds of applications, whether they’re serverless, container-based, running on EC2 instances, or even VMs in your own data centre.
\nYou do this by setting up the service network and then adding services to it. These services are virtual constructs that in turn point to your applications. In addition, you can configure routing rules and listeners to connect to. All of this allows you to then have a single endpoint that can be called. Oh, and did I mention you can use the Resource Access Manager to share your network with other accounts so they can hook into the same service network?
\nSecurity also plays a big part in VPC Lattice, as it allows you to set granular access controls for your services and it comes with access logging as well.
\nAll that said, it’s still early days for VPC Lattice. It looks like it will be a good solution for allowing easier interconnections between related applications, but right now you need to sign up for the preview to get access and even then it’s only available in the Oregon region.
\nThere have been some interesting security updates as well, and potentially the one with the biggest impact is proactive rules for AWS Config. This means that you can have AWS Config do checks not only on resources that have already been deployed, but also for resources before they get deployed. This does require a bit of extra work, as you need to use your CI/CD platform to use this. In effect, you have to use an API call (which can be an SDK) or CloudFormation Guard to trigger the proactive verification. You then need to check the results and can stop your pipeline if it’s not compliant. The advantage here is that you can have all of your compliance checks in a single place and use the same checks for existing resources (or ones spun up outside of your CI/CD pipeline) and new ones. Which should make it easier to remain compliant. This feature is currently rolling out to all regions and hopefully is available for use in Sydney by the time you read this.
\nIf you’re familiar with Inspector, you will already know how useful this is for scanning your EC2 instances and container images for vulnerabilities. This has now been extended to Lambda. Unlike for EC2, Inspector doesn’t scan the Lambda functions on a regular schedule but instead it will do so when the function is updated or a CVE is announced. This seems to me to be a good way of doing this as it means your stable and long-lived functions don’t get constantly scanned. One downside is that it currently only supports functions written in Java, NodeJS, and Python but on the other hand these are the most popular languages so it should cover most use cases. This is available in Sydney as well, but you do need to enable it by following these steps. As a last note though, if you haven’t looked at Inspector for a while, this is a good time to do so. It has improved significantly over the past few years.
\nKMS External Key Store, or XKS, is the last one I’ll mention here. While it may be a bit of a niche case, the ability to run your own HSM (Hardware Security Module) and have that integrate with KMS can be a game changer. What this basically means is that if you already have an HSM in place in your own data centre, you can now integrate this with KMS to use that to generate the keys for your AWS environment. The most likely reason you would do this is for regulatory reasons, and that should make your life a lot easier. And yes, this is available in Sydney.
\nA quick summary of some of the other announcements of note.
\nThese are only the main items that were announced outside of the keynotes so far, but there are a large number of other announcements as well. Unfortunately we can’t cover all of them in these blogposts, but if there’s anything we discover it will likely show up in a future blogpost.
\n", "summary": "If you’ve attended a conference before, you know that the most exciting things don’t necessarily happen in the sessions themselves. Unfortunately, I’m not in Las Vegas, so I can’t enjoy the hallway track. Still, AWS is kind enough to offer an alternative by releasing some of its announcements outside the keynotes. Best of all, you can see these announcements happening throughout the day without needing to get up at 3 am! Below are some of the announcements that stood out to us.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2022-12-01T16:26:34+11:00", "date_modified": "2022-12-01T16:26:34+11:00" }, { "id": "https://ig.nore.me/2022/11/top-highlights-from-aws-pre-invent-2022-were-most-excited-about/", "title": "Top highlights from AWS pre:Invent 2022 we're most excited about", "url": "https://ig.nore.me/2022/11/top-highlights-from-aws-pre-invent-2022-were-most-excited-about/", "content_html": "This was originally posted on the CMD Solutions blog, as all of Mantel Group’s brands (of which CMD was one) combined, some older blog posts got lost in the shuffle or links changed, so I’m replacing my linkblogs with the original content to ensure they stick around.
\nEvery year as we get closer to re:Invent, AWS starts releasing the big items that didn’t quite make the cut for the major announcements. That doesn’t mean these releases are less interesting than what will be released at re:Invent; just that they don’t necessarily fit the narrative for the conference. In fact, some of my favourite announcements in previous years happened during this time. So, with re:Invent now only a couple of days away, let’s look at some of the gifts AWS brought us during pre:Invent.
\nIn the past few weeks, AWS opened not one, not two, but three new Regions! These are in Switzerland, Spain, and a second region for India in Hyderabad. Unfortunately, this is not the Melbourne region we’ve been waiting for but is still a big deal for workloads that can make use of these. It’s important to note that you will have to enable these regions for your accounts before you can use them.
\nAdditionally AWS pre-announced an upcoming region in Bangkok, Thailand. Surprisingly this was not announced on the AWS blog, but instead on the Amazon blog where it’s part of an announcement about investments AWS is making in the region. This also means we don’t have any details on when we can expect this region. Maybe more news will be coming about this at a later date.
\nAs the Melbourne region was estimated to be opened during the 2nd half of 2022, is it likely to be announced during re:Invent? Unfortunately, it’s not very likely as AWS has never announced the opening of a region during re:Invent. However, there is a good record of regions being opened between re:Invent and Christmas, including last year’s Indonesia region, so we might still get to play with our new region this year. And you never know about re:Invent either.
\nHave you ever run into a situation where you can’t find something you built, but you know it must be around somewhere in this account? Maybe you just want to be able to easily see a list of all your resources, or in a specific region? Well, that’s where the new AWS Resource Explorer comes into play.
\n![\"\"](\"/2022/11/top-highlights-from-aws-pre-invent-2022-were-most-excited-about/76a70b123caac542655ffa37fc26f4fbf607e1af.png\")
Once you enable this, you will be able to have a look through your account, filter it by region or type, and get a good overview of what you have. There are a number of ways you can filter this and you can set up specific views as well. And best of all, it finally allows something that I’ve been hoping for since the new search bar was introduced: searching for resources.
\n![\"\"](\"/2022/11/top-highlights-from-aws-pre-invent-2022-were-most-excited-about/f4ac93531d85e280ae7d29a1cf1a3e198c90101c.png\")
Of course, it’s not all perfect and aside from it still being a little limited in the resources it can detect, the main improvement I’d love to see is integration with Organizations so it will work cross-account. Purely based on the first screenshot above, I feel like this is something that will come but will likely take time. All in all though, this is a great new service that will make understanding your AWS resources a little easier.
\nLambda is the flagbearer for Serverless architectures, and it won’t come as a surprise that it comes with several interesting announcements. And while support for Node.js 18 is nice, it’s not the kind of thing that makes that much of a difference to how we use it.
\nThe first thing that came out was a new way to retrieve parameters and secrets from your Lambda function. You can now do this by way of calling a URL. This means your application code doesn’t need to load the appropriate AWS SDK which makes for a smaller and faster application.
\nThen there is the Telemetry API. This is a new way of retrieving logs for Lambda through an extension and replaces using the Logs API. It contains more data and is easier to link to a centralised log collection.
\nThe last one I’ll mention here is a pure development tool. The SAM CLI is a good way to test your Lambda functions locally, but one limitation to it was that you had to use CloudFormation (or specifically SAM) to define your infrastructure. For fans of Terraform, of which there are plenty here at CMD, there is now good news as the SAM CLI now supports Terraform as well.
\nOne announcement that got a number of us excited is CloudFront’s support for continuous deployment. While the name may not sound very exciting, what this allows you to do is set up a staging CloudFront distribution that you can point at your next version or use it to test changes to your Cloudfront configuration. You can then access it either by providing a special header in your calls to do (automated) testing, or send a portion of your traffic to it. This let’s you do either a blue/green or canary style deployment. Once you’re satisfied you can then make the staging distribution the main distribution and the deployment is complete.
\nAnother interesting change here is the ability to transfer Elastic IP addresses between your accounts. Over the years best practices have moved from having a single or small number of accounts to a more complex structure of multiple accounts. Moving your applications to other accounts isn’t always easy, and one potential issue is when you have a lot of third-party integrations that have whitelisted IP addresses in your old account. These are usually the Elastic IPs tied to your NAT Gateways.
\nWith the ability to move your Elastic IPs to another account that means this is no longer an issue as those IPs can come to your new account. And in cases where you share those IPs across multiple applications, you can have a setup where you have a dedicated outbound VPC that you route all your outbound traffic through. This can all be tied together using Transit Gateway for a transparent traffic flow.
\n![\"\"](\"/2022/11/top-highlights-from-aws-pre-invent-2022-were-most-excited-about/0ef6c783ebdf31cd4f5c7e6b9c8286de0edbf193.png\")
And speaking of restructuring your accounts, while AWS has a solution for managing your accounts using Control Tower, they have now also released CloudFormation support for managing your Organization structure, including the creation of accounts, management of OUs, and even your various policies like SCPs.
\nSecurity is an important part of any cloud journey, so we’ll look at some of the things we can use from that. First, while it is recommended to use short-lived sessions using IAM roles with some kind of federated access like AWS IAM Identity Center (formerly AWS SSO), if you are using IAM users you can now assign multiple MFA devices to them as well as to the account’s root user. This for example allows you to use either a FIDO security key as well as a token generator like Google Authenticator or Authy to access your account.
\nIf you are using IAM Identity Center, you can now set the maximum duration of CLI and SDK sessions. You could already do this for Console and application access, so this builds on top of that. The maximum duration can be configured to be any amount of time between 15 minutes and 7 days, but it should be noted that if the session for your IdP (for example Okta or Microsoft Exchange) times out, this will also cancel the session.
\nSome other notable releases in the security space include the ability to set a delegated administration account for CloudTrail, which allows you to have all your centrally managed CloudTrail trails to collate in an account you specify instead of required to go to the Organization’s management account. And another change is that you can now export CloudWatch Logs to SSE-KMS encrypted S3 buckets. Until now you couldn’t do this and instead had to rely on custom solutions to export logs to a bucket protected with a custom key.
\nThe above is only a fraction of the 400+ announcements AWS made in the past month, but they do reflect some of the things that I believe may have a positive impact on your usage of AWS. Of course, if you wish to learn how we can help you use these new features you’re welcome to reach out to us. In the meantime I’m looking forward to seeing what re:Invent will bring us next week.
\n", "summary": "Every year as we get closer to re:Invent, AWS starts releasing the big items that didn’t quite make the cut for the major announcements. That doesn’t mean these releases are less interesting than what will be released at re:Invent; just that they don’t necessarily fit the narrative for the conference. In fact, some of my favourite announcements in previous years happened during this time. So, with re:Invent now only a couple of days away, let’s look at some of the gifts AWS brought us during pre:Invent.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2022-11-24T16:40:15+11:00", "date_modified": "2022-11-24T16:40:15+11:00" }, { "id": "https://ig.nore.me/2022/07/why-the-aws-melbourne-region-is-a-good-thing-for-apra-regulated-businesses/", "title": "Why the AWS Melbourne Region is a good thing for APRA-regulated businesses", "url": "https://ig.nore.me/2022/07/why-the-aws-melbourne-region-is-a-good-thing-for-apra-regulated-businesses/", "content_html": "This was originally posted on the CMD Solutions blog, as all of Mantel Group’s brands (of which CMD was one) combined, some older blog posts got lost in the shuffle or links changed, so I’m replacing my linkblogs with the original content to ensure they stick around.
\nWe’re now in the second half of this year, which means that the new Melbourne Region for AWS can open up any day now, and that makes this is a good time to have a look at what this can mean for an APRA regulated business. This post builds upon the excellent posts written by my colleagues about the new AWS Melbourne Region and moving to AWS as an APRA regulated business1.
\nThe most important part here is to figure out why, as an APRA regulated business, you would care about the existence of the Melbourne region. After all, we already have the Sydney Region and that runs everything perfectly fine. The answer to that question comes from the Information paper - outsourcing involving cloud computing services provided by APRA itself. There are two quotes in there that are relevant here:
\n\n\nAs with any outsourcing arrangement, it is prudent for an APRA-regulated entity to only enter into cloud computing arrangements where the risks are adequately understood and managed. This includes demonstration of:
\n\n
\n- ability to continue operations and meet obligations following a loss of service and a range of other disruption scenarios;
\n
There are more items in that list, but the above is the most relevant with regards to the Melbourne Region. While a well-designed application will be able to deal with Availability Zone outages, there is still always the risk of a Region failing completely (not very likely), or a single service you rely on in a Region failing (a more common event). In both of these cases you want to be able to rely on a secondary Region for either Active-Active or Active-Passive failover.\nAWS has a lot of Regions, including ones that are relatively close to Australia, but again from that same information paper:
\n\n\nAn APRA-regulated entity should consider the benefits of the following factors as ways of reducing inherent risk as part of the solution selection process:
\n\n
\n- Australian-hosted options, if available, in the absence of any compelling business rationale to do otherwise. Australian hosting eliminates a number of additional risks which can: impede a regulated entity’s ability to meet its obligations; or impede APRA from fulfilling responsibilities considered necessary in its role as prudential regulator;
\n
While it is not strictly necessary to be able to fall back on another Australian Region, it will make things a lot easier as you don’t need to consider and manage the risks of storing data or using compute overseas. And speaking of risks, from a compliance standpoint AWS services are the same in every Region they run in.
\nBased on the above, the answer would be yes. The end goal of a multi-Region setup would be something like the Active-Active or Active-Passive setups that my colleague Samer described in the aforementioned post. A setup like that will take some time to do well though, so you’ll likely have to make sure that your applications are ready for this.
\nIn addition, due to the sheer number of services that AWS offers not everything will be available on day 1. Over the years AWS has improved their launch offerings and does its best to fill any gaps as quickly as it can, but there is always a chance that a service or feature you require isn’t there yet. When that happens there are two main things you can do.
\nFirst, if you have a Technical Account Manager or other AWS representative for your company, you can leverage them to influence the order in which services are added so make sure to raise your requirements with them. And secondly, there may well be workarounds for your workflow. For example other services that let you achieve the same result, perhaps even in a better way that is more suited to a multi-region setup. Of course (obligatory plug), CMD would be more than happy to help you explore those options.
\nSo, how should we go about using the new Region? As I mentioned, there is no need to wait until there is full feature-parity for everything you use. Especially as full feature-parity is unlikely to happen anytime soon (which, living in Melbourne, is not something that makes me happy to write and I would love to be proven wrong).
\nMulti-region setups are not uncommon. While they are not used as much in Australia as they are in for example the United States (which has had multiple Regions for a long time), there is a lot of documentation about the various benefits and pitfalls. AWS also has given lots of talk about multi-region best practices and these can be found on YouTube.
\nSo, we know what we want to achieve and have looked a bit deeper into how this will work. The next step would then be to pick a workload to truly make multi-region. While it would be great to have your system of record be safely and securely multi-region as soon as possible, I’m pretty sure I don’t need to explain why it would not be a good idea to use that as your test system. Instead pick a workload that is not critical and preferably doesn’t get a lot of traffic to verify that your research is correct. After that you can then continue with more critical workloads.
\nAnother way to test the new Region is to run new workloads there. Especially if you have services where it would be beneficial if they run closer to Melbourne this may be useful.
\nFinally, even before running your applications there, you can start using the Melbourne Region to ensure your critical data is safe and secure in multiple geographic locations. If disaster strikes and you need to rebuild your infrastructure, infrastructure as code practices will help with that. However, if you lose your data (and possibly even your backups) that is a far bigger problem.
\nThe main tool of choice for this is AWS Backup. AWS Backup is a managed service that you can configure to make backups of many of your data sources and configure it to store those both in the Region they’re created as well as in a different Region. Using this you’ll have a backup of your databases and filesystems in a separate Region.
\nIf you don’t use Backup yet, there are alternatives for most services to copy backups and snapshots cross-region. For example RDS allows you to copy snapshots to a different Region. One way to set this up is by having EventBridge monitor for the creation of new snapshots and in turn have that trigger a Lambda function that copies the snapshot to the Melbourne Region. And this isn’t limited to RDS either as you can do similar things for other services that store your data. But as always, keep in mind that an untested backup is not a real backup, so make sure you frequently verify your backups (both the process and the actual backup itself) work.
\nBefore the opening of the Melbourne Region it may have flown under the radar, but it is possible these days to use the same KMS key in multiple Regions. Which means that you can safely encrypt your data in Sydney, and send it to Melbourne without having to decrypt it in between but you’ll still be able to access it.
\nSimilar is the ability for Secrets Manager to replicate your secrets to another Region. Setting this up means that you only need to manage your secrets in one place but will be able to use the same values in multiple Regions. One use case for this would be for storing the secrets of your database users or the API keys for accessing third-party services. That way if you need to fail over, or just want a backup of your secrets, you will have those available in a secure manner.
\nSo, at this stage we know that in theory the Melbourne Region can help our APRA regulated business be more robust and able to handle disasters of various kinds while remaining compliant.
\nAt this stage, before the actual launch, it makes sense to start building a plan around how you want to use the new Region short-term and longer term. Then once the launch has happened, you can execute this plan and reap the benefits as soon as possible.
\nSome questions you can ask around this plan are: Which applications and data sources do I want to make multi-region? What does this mean for your security posture and training? What services and features need to be available before your applications can run fully active-active or active-passive multi-region? If you initially want to use this for backups, how are you going to set this up and verify these backups?
\nAnd remember, you may not get everything perfect the first time around, but that is why you don’t start with your most important applications or data. And of course, try to influence the roadmap for AWS. The more requests they get for a service to be available in Melbourne, the more likely it is that it will show up soon.
\nIf you would like to discuss this more with an AWS Partner who has worked with the Financial Services Industry since its inception, and happens to be AWS Services Partner of the Year, please reach out to us.
\nThese particular blogposts are no longer online, so I removed their links. ↩︎
\nThis was originally posted on the CMD Solutions blog, as all of Mantel Group’s brands (of which CMD was one) combined, some older blog posts got lost in the shuffle or links changed, so I’m replacing my linkblogs with the original content to ensure they stick around.
\nOnce in a while AWS releases a feature that people have been demanding for a long time, and they did so again when they added the ability to close accounts from your Organization management account. Let’s have a quick look at why it’s so good to have this ability, how it actually works, and what this will enable us to do.
\nClosing an AWS account is not something new. It’s obviously been possible for a long time, but you could only do this using the root user of the account in question. When you only have a standalone account, that isn’t really an issue. You set the details when you created the account and assuming you still have access to them you can just log in to delete your account.\nThat situation changes however when you’re using an Organization. Not only will you more likely be dealing with a larger number of accounts, you often won’t have the root password as accounts created from the Organization are given an automatically generated password. This in turn means that you need to go through the forgot password section of the Console login to get the password sent to your email. And if you don’t have access to that email, well… that’s a bit of an issue.\nSo, you can imagine this puts up a barrier around closing accounts and many Organizations are left with unused accounts. In fact, I can tell you that within a day of this coming out I cleaned up the accounts in my personal Organization. A task I’d been putting off because it was too much of a hassle.
\nWith the new close-account API, closing accounts has become a lot easier, so let’s see how it actually works. As is often the case with AWS, there are multiple ways to do this. You can use the Console, the AWS CLI, or one of the SDKs to include it in your management tools.\nThe CLI command is very straightforward as you can just run
\naws organizations close-account --account-id 123456789012\nUsing the Console isn’t much more complex. Within your Organisation view, open the account you wish to delete.
\n![\"\"](\"/2022/06/why-the-aws-close-account-api-is-a-big-deal/close-account-account-list.jpg\")
Then you press the Close button.
\n![\"\"](\"/2022/06/why-the-aws-close-account-api-is-a-big-deal/close-account-close-button.jpg\")
And you get to see the below popup, where you have to acknowledge all the things and type in your account ID.
\n![\"\"](\"/2022/06/why-the-aws-close-account-api-is-a-big-deal/are-you-sure.png\")
Afterwards your accounts overview will have a small (suspended) message added to your account.
\n![\"\"](\"/2022/06/why-the-aws-close-account-api-is-a-big-deal/suspended-account.png\")
This is an example of me cleaning up my personal accounts, and as you can see in the popup above it shows a list of things you should be aware of and that you need to acknowledge. Clearly using the CLI is a lot faster.\nThat said, there is a limit to the number of accounts you can close in a short period of time. By default the quota is set to being able to close 10 accounts, so if you have a large number of accounts that you wish to clean up you should start by requesting a quota increase.
\nThere are some other caveats related to closing accounts. These are not specific to using the close-account API, but are important to be aware of. First of all, you won’t be able to reuse the email address that is attached to the root user to create a new account. Every account needs a unique email address attached to it and that includes closed accounts.
\nSecondly, once an account is closed, it will first go into a 90-day cooldown period where you cannot use the account. Make sure to terminate everything in your account before closing it as some resources might keep incurring costs. A good tool to clean up your accounts is aws-nuke. In addition, it’s possible to reactivate your account within those 90 days at which point you will be charged again for any resources that are still in there. The purpose of this is obviously to ensure you are able to recover an important account if it was deleted prematurely. Please note that as this is an Organization, billing is handled through your Organization’s management account so any charges you had before closing the account will still show up there.
\nAs with all API calls, access to the close-account API is managed through IAM, and in addition it can only be used in the management account. Assuming you limit who has write access to the management account (if not, please do so) this in itself will already help with ensuring most people can’t go around deleting accounts.
\nIn addition however, it is possible to attach IAM policies to the roles or users that have the ability to close accounts. Right now it doesn’t seem to work with SCPs, but using these IAM policies prevents certain accounts from being closed by accident. You can do this by hardcoding the account IDs or by the use of tags that you apply to the accounts in your Organization structure. This means that you can set it up so that non-essential accounts can potentially be deleted, but prevents this for your important accounts.
\n{\n "Version": "2012-10-17",\n "Statement": [\n {\n "Sid": "PreventCloseAccountForTaggedAccts",\n "Effect": "Deny",\n "Action": "organizations:CloseAccount",\n "Resource": "*",\n "Condition": {\n "StringEquals": {"aws:ResourceTag/AccountType": "Critical"}\n }\n },\n\t\t{\n "Sid": "PreventCloseAccount",\n "Effect": "Deny",\n "Action": "organizations:CloseAccount",\n "Resource": [\n "arn:aws:organizations::555555555555:account/o-12345abcdef/123456789012",\n "arn:aws:organizations::555555555555:account/o-12345abcdef/123456789014"\n ]\n }\n ]\n}\nAbove I already pointed out that this allows easier clean-up of accounts that are no longer required, but what indirect benefits does this bring? Obviously, it depends on you but I can give you a couple of examples.
\nThe big thing is around temporary accounts. There are two main use cases I can see for that. First, you can create temporary accounts for demos. When you give a public talk it helps when you don’t have to worry about potentially showing other things that you run in your account. This ability means that you can easily create a new account specifically for your presentation and then afterwards you just close the account. And you don’t have to worry about any details that matter forever being captured on a video recording.
\nSecondly, you can create temporary accounts for training purposes. Imagine that you want to learn something new, but don’t want to risk messing up your personal account? Create a new account and when you’re done you can quickly delete it again. Similarly, if you run a training you can create a bunch of accounts and then delete them after the training is done. An example of this is the LearnCMD training we just ran for which we created about 60 accounts for students and teachers.
\nAnother option that is more geared towards the business world is the ability to have sandbox accounts per developer. This is a dedicated account where members of your team can play around, and when they leave the company you can make the destruction of the the account part of the automated off-boarding process.
\nNo doubt you can come up with some ideas yourself for the ability to quickly destroy accounts, and personally I’m quite excited about some of these use cases.
\n", "summary": "Once in a while AWS releases a feature that people have been demanding for a long time, and they did so again when they added the ability to close accounts from your Organization management account. Let’s have a quick look at why it’s so good to have this ability, how it actually works, and what this will enable us to do.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2022-06-28T00:00:00Z", "date_modified": "2022-06-28T00:00:00Z" } ] }