Working on a global project requires onboarding of many new data sources while complying to legal data residency requirements. While this is being worked out for our company we will be starting to produce a high amount of logging data very quickly.
To give us a head start we will be using Cribl LogStream and a simple data collection pipeline.
Cribl LogSteam is mainly an intelligent log information routing and preprocessor tool.
Our initial goal of this pipeline is that logging data produced in a given region worldwide will be sent to our regional AWS based Cribl LogStream servers . We will just create a simple pipeline to write everything into a regional S3 AWS data bucket (destination) so we can keep the raw log files within the regions we are building out. This also allows us to save the full fidelity log files for compliance reasons on cheap AWS S3 storage.
Once we get the final approval to be allowed to consolidate some of the original data we will create a second pipeline thru Cribl where the source is our AWS S3 raw log data and the destination Splunk HEC, while we just select the required fields and the final location we want.
This solution will give some required flexibility initially while saving Splunk ingest license cost in the long run.
Happy cribling and splunking.