Revisiting an Approach to Listing S3 Buckets
A while back I contributed some feedback on a pull request that involved code that would call an AWS API to obtain a listing of S3 buckets on an AWS account.
The original implementation involved making a single call, extracting the relevant properties and returning that as a List.
My feedback – that made sense at the time
I’ve worked on a few services that accumulated a broad range of features, so I wanted to ensure that this particular code wouldn’t lead to missing the full list of buckets, or involve memory bloating when a high number of buckets was involved.
I was still very new to Kotlin, and had recently been reading about Sequence as a mechanism for lazy loading. So, I suggested that the API would be a bit more stable if we tweaked the code to iterate through all available results and yield them via Sequence instead of one big List.
There was a bit of back and forth, with the code eventually being updated to lazily load the way that I had suggested.
Change merged, on to the next thing…
Now curiosity has caught up with me
I’m a bit bored at the moment, so I have been reflecting on how the lazy loading mechanism could have changed the nature of handling errors.
One of the earlier versions of the code would load all of the data and then expose it to the caller, but what would happen in the lazy loaded implementation where if multiple calls are made to the AWS API they could fail partway through the listing?
Let’s imagine that we had 5,000 buckets involved, and that each individual load would bring back 1,000 results at a time meaning that multiple calls would be involved.
Failing on the first call seems like something that could be detected and handled, but if any of the subsequent calls failed we would want to detect that and respond accordingly. This should be something that the interface should highlight to anyone wanting to call it.
Trying out some code
I’ve set up a project from scratch, involving the latest available version of Kotlin and AWS’s Kotlin SDK client – which is not the same AWS client as the project I was working on, so you’re note getting any deep insight into the original codebase.
Looking at the s3 client API gave me further things to consider. There’s a Paginated approach on offer, it doesn’t use a List or a Sequence, instead it involves a Flow…
Sequence vs Flow
- Sequences are synchronous and blocking
- Flows are asynchronous and non-blocking
- (Like Project Reactor in Java)
Given that the API call to AWS involves a remote call with some network IO, it makes sense to go with the Flow approach and the coroutines that are involved.
Testing
I wanted to set up localstack to run some integration tests for this toy project, but I now realise that localstack has changed a bit since I last looked into it. They now have a range of products and associated licensing, meaning I would need to register to make use of the up to date implementations.
So for now, I will just start off with some unit tests with mocking instead.
This is working out as a nice follow-on to an approach that I was trying out last year, where I theorised that for testing Kotlin we should move away from the libraries used for testing Java. So, I’m trying out mockk for mocking in Kotlin, as I have seen it in use in enough projects to convince me that it will be more suitable and / or idiomatic to Kotlin than Mockito or similar libraries that I have used with plain Java.
The code
I have pushed my toy project code to a repository in GitHub:
https://github.com/Sounie/making-a-bucket-list
It’s not intended to demonstrate anything production ready, but serves as an excude to publish this post with it’s punny title.
Conclusion
What have I learnt?
- There’s a trade-off of involving lazy loading, as the caller needs to be aware of the possibility of failing part-way through receiving the results.
- Flow is available as an alternative to Sequence, and is more suitable where IO is involved.
- Localstack licensing has changed
- Mockk is definitely a better option than Mockito for testing Kotlin, as it offers feature with awareness of Coroutines and probably a bunch of other stuff.
Leave a Reply