I have a solution which consists of 3 services, there are 2 queues to pass on messages from one service to the other, and it’s using the EasyNetQ C# library to connect to RabbitMQ.
This is the outline of the architecture:
- Receive data from network TCP (do some processing) push to
- Read from Queue A (do some processing) push to Queue B
- Read from Queue B, send processed data to a 3rd party API
Services 1 and 2 are old services while 3 has been introduced recently.
The thing is that 1 and 2 handle one log at a time, so the publish/subscribe pattern makes sense i.e. service 1 receives one message and pushes this one message to Queue A for further processing, but the Service 3 should send as many messages as possible to the 3rd party API, as the API can accept upto 1000 messages int a single HTTP POST request.
I thought of introducing a timer for example Every X mins the timer will trigger an event to read upto 1000 messages from the Queue and send them via HTTP to the 3rd party API, but I can’t seem to find a way to manually read N number of messages from the Queue.
At the moment I publish the message from Service 2 to Queue B like this:
Then in the 3rd service I have a subscription
but this subscribe will always read one message at a time from the Queue.
So, what would be the best way to handle this?