Testing a size-limited storage with docker

Did you ever have problems regarding the size of your storage? If you use docker in your daily life, either for deployment or simply for testing, let’s see how you can reproduce these problems with the storage.

You consider storages to have an unlimited size. You can easily have storages with 1TB and you usually can expand the storage somehow. Normally you don’t have to worry about lacking disk space, but depending on the application you’re developing you might fill the storage at some point.

For example, if you have to manage information about a maximum of 50 users, as long as you have enough space, the size it takes on disk will remain stable. You’re sure the application won’t use more than 5GB of disk space.

However, if you work with big files, you won’t be able to keep the disk usage under normal constraints. A big file could use 50GB or even more, and you’ll likely need to store a bunch of them. Your 1TB storage might not be enough. In addition, other apps will take disk space, so your remaining size could be reduced to less than a half.

Moreover, cloud-based deployments might impose quotas, limiting your effective storage to 10GB unless you pay for more.

Why do we need to test with a size-limited storage?

As said, you can’t really trust that your application will have an unlimited storage size. You might get an error about insufficient space on disk at any time. Even if you control the available disk space, you might need to do something when you’re under 10% of disk space.

This isn’t just a matter for your server application, but to any client that your application could have. For example, a mobile app can upload a file to your server, but the server might not have enough space. The server needs to cleanup everything after the failed operation (metadata that could remain assuming the operation succeeded), and it also needs to ensure the client receives an appropriate error. Obviously, the mobile app needs to handle that error.

You cannot solve a problem if you cannot reproduce it

Reproducing these kind of problems isn’t easy:

• The storage is usually large, and we need to fill the storage somehow. We’d need to:
  • Upload GB-size files or bigger.
  • Generate a big random file in the storage
  • Generate a large amount of files in the storage
  • A combination of all of the above
• Filling the storage can be problematic if it’s in your own host. You could have problems trying to test the scenario due to the lack of space in your computer.
• Even if the server is in its own machine, other unrelated services could have problems to run.

We’ll have to assume the problem will happen in our application, and the rest of the system will behave properly. For example, while uploading a big file through apache or node.js, we’ll have to assume that the file is uploaded properly and it reaches our application; apache might not get the whole file due to space problems, and apache could fail the upload, but not us.

tmpfs to the rescue

The main trick we’re going to use is to mount a tmpfs FS in a specific folder. We expect the application to write everything inside that folder. Depending on how the application writes into the storage, you might need different mounts: one for the logs, one for uploaded assets, one for uploaded certificates, etc. This is mainly to ensure that each piece writing in the disk can handle the error properly

The key point is that tmpfs allows us to control the size of the FS. We can easily limit the FS to have 50MB. This means that the target folder will have contents up to reaching those 50MB.

This approach has several advantages:

• You don’t bother other applications and services. The limitation is only applied to the target folder, and it’s only known to the application. Other applications shouldn’t need to know about that folder. As long as you have enough space in the host, other applications won’t notice
• You can control the size just before mounting tmpfs. You can setup the mount to use only 50MB, so filling it is very easy.

Note that the tmpfs stores all its information in the RAM and maybe in the swap. Oversizing tmpfs could cause a deadlock in the machine, so you should keep the size small regarding the memory size. Probably less than 10% of the available memory should be safe. You need to consider other applications running, the memory the OS is using, and other things that could be already using the RAM.

A practical example using docker

Let’s see a practical example using docker and docker-compose.

We have reports about errors not being properly handled. We have some suspicions that the error could be caused by the s3 storage running out of space. Luckily, we have a docker image we can use, so we’ll just mount a tmpfs volume in the right place.

The relevant contents of the “compose.yml” file are below.

version: '3'

services:
  s3server:
    image: scality/s3server:latest
    restart: always
    ports:
        - 8000:8000
    environment:
      - ENDPOINT=s3server
    volumes:
      - s3tmpdata:/usr/src/app/localData

volumes:
  s3tmpdata:
    driver_opts:
      type: tmpfs
      o: size=104857600
      device: tmpfs

What is important is:

• “s3tmpdata” volume is created with the tmpfs, with a limit of 100MB
• the volume is mounted in the “/usr/src/app/localData”, where the data will be uploaded accordingly to the image’s documentation.

Note: the docker image also exposes the “/usr/src/app/localMetadata”, which will hold metadata. This isn’t relevant for this particular test though.

What we have achieved running docker-compose with that configuration file? Uploading a file to that s3 storage will place some data in the “/usr/src/app/locaData”. Once the s3 server fills the 100MB, the server will have an error about not having enough storage. Then we’ll know what the s3 server answers, and then we can handle that error on our side.

In case we were developing the s3 server, we could also reproduce the error the same way.

It’s important to note that we assume that the error will happen only when the s3 server writes in the “/usr/src/app/localData” directory. We don’t expect problems while writing in a different folder. In case we want to check what happen if we can’t write in the metadata folder, we could change the target of the mount point to “/usr/src/app/localMetadata”

Conclusion

Lacking disk space is something that could happen if we don’t limit (or can’t limit) the amount of information we write on disk. Mounting and using a small FS helps to reproduce the problem and handle it properly.