The messages’ PartitionKey was the ID of the IoT device and the RowKey contained the time when the messages were created. And there, the problems arose.Īs the messages were already there, we first thought about a solution with the given setup and table design. Therefore, after these 180 days, we wanted to delete the old messages. However, despite the relatively low costs of Azure Table storage, the costs add up quickly.ĥ00 billion entities (1 KB each) would cost around 32,000 € (data center in West Europe) simply storing the data, without performing any operations. No big deal for Azure, the table storage can easily handle that. With a retention period of 180 days, there can be over 500 billion entities in the storage table. Even though each of the messages had less than 1 kB, this is not a negligible amount.
We calculated with around three billion new messages each day at full load. And by using basic mathematics, we know that this adds up quickly and we have billions of messages in the table. These messages were all stored in a single Azure storage table. It was an IoT project with thousands of devices sending tens of thousands of messages each day. An issue we had with deleting literally billions of entities gave us the idea to write this article. See Table storage concepts.ĭeleting billions of entities sounds like an exaggeration. A property is basically a key-value pair. Further, each entity is made up of up to 255 properties. Similar to a row in a relational database, we have entities in the Table storage, which is limited to 1 MB in size. If you are familiar with SQL databases, this concept sounds familiar. Under a storage account, you can have multiple tables (see image below). NET, Java, Node.js, Python, PHP, etc.Īzure Table storage (as well as the other standard storage types) are organized under a so-called storage account. There is a REST API that makes it simple to interact with Azure Storage, and there are SDKs available in a lot of different programming languages, including.
The other three types are Blob storage, File storage and Queue storage, which I will not introduce any further here. Table storage is one of the four standard storage types that is very simple to set up in the Azure portal. In this article, I will focus on Azure Table storage, which is a relatively low-priced, scalable NoSQL storage suitable for storing large amounts of data while keeping costs low. On the Azure platform, there are many different storage solutions, such as various kinds of database solutions, Data Lake Storage (mainly targeting data science applications), queue storage, managed disks, blob storage, and a lot more. One of the big cloud platforms is Azure, which is created by Microsoft. (FYI: I am talking about the second thing here.) Reaching the limits of Azure Table storage…or trying toĬloud - for most people just an accumulation of water droplets in our atmosphere, for others an accumulation of various computing services in order to build a powerful IT infrastructure.