How to Ingest Data into Falcon LogScale Using Python

This post describes how data is ingested CrowdStrike Falcon® LogScale from your macOS platform using Python. This guide is great for setting up a one-node proof-of-concept (POC) so you can take advantage of LogScale’s free trial.

Before you can write your ingest client, you need to lay a good foundation. This means that you prepare your macOS instance with the following steps:

• Download Homebrew
• Update your default macOS Python
• Install the Python package manager
• Download the LogScale libraries

Ready? Let’s begin.

Prepare your macOS instance

One of the methods of ingestion is use LogScale software libraries which are available in different languages. Today we work with Python and MacOS.

Step 1: To install Self-made, a package manager for macOS. Installing new packages with Homebrew is a simple command line in Terminal, similar to installing new packages in Linux. Follow the instructions on the homebrew website.

step 2: Use Homebrew and update your default macOS Python. As you may know, MacOS 10.15 (Catalina) currently runs on Python 2.7, although newer versions are available. For MacOS it is important that the default Python stays at 2.7. We need to update your Python to the latest version while keeping version 2.7 for essential macOS features.

Follow these instructions from Matthew Broberg: The right and wrong way to set Python 3 as the default on your Mac.

Update for VenturaNote: Python 3.9.x is available for the latest version of MacOS 13.2.1 (Ventura), but it is not installed by default. You need to install XCode to install Python via terminal with the following command:

xcode-select –install

More information can be found here: Python3 now included in Ventura

step 3: Once we have the appropriate version of Python running on your macOS, we need to install the Python Package Manager, pipso that we can install LogScale’s client library. Normally, pip comes with Python and there is no extra step to install.

To see if pip is installed, run the following command in your terminal:

python -m pip --version

If pip is installed, you will see the following output:

Alternatively, you can install pip manually by opening the terminal and running the following command:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

Then run python -m pip --version again to verify pip is installed and you have the latest version.

step 4: Install LogScale Python software library. The humiolib library is a wrapper for LogScale’s web API that supports easy interaction with LogScale directly from Python. For more information see our humiolib github.

You can start the installation by running the following pip command in your terminal:

pip install humiolib

This command gives you a series of outputs that tell you what files will be installed. After completing the installation, you have completed the preparation work for your macOS instance. Now we can move on to the fun stuff.

Create your ingest client

It’s time to start writing to an ingest client. Check out this sample program:

Let’s break down some parts of the code.

You must at least add humiolib to run the codes required to send logs to LogScale.

from humiolib.HumioClient import HumioIngestClient

You also need to create an ingest client with attributes that tell the client where to send this log.

client = HumioIngestClient(
  base_url= "The url where LogScale resides",
  ingest_token="An API token from LogScale"
)

The API token can be obtained from your LogScale instance.

Structured log messages

There are two types of messages you can send to LogScale: structured and unstructured.

In most of our use cases, LogScale receives structured data as a JSON object. There is no strict format for how the JSON object is structured, but you must ensure that the JSON object is valid. You can inspect the structure of a JSON object using a tool like JSON Lint.

Additionally, with structured data, you can send valid timestamps as part of the log entry, and LogScale will use the provided timestamp instead of inserting its own. Therefore, make sure that the timestamp for the log entry is less than 24 hours after it was sent. Otherwise, LogScale assumes that the data is older and deletes the log entry without an error message.

Below is an example of structured data:

structured_data = [
 {
 		"tags": {
 			"host": "str(ip)",
 			"host_name": "str(host)",
            "filename": "str(caller.filename)",
 			"line": "str(caller.lineno)",
 			"error_level": "INFO"
 		},
 		"events": [
 			{
 				"timestamp": str(datetime.now(timezone("EST")).isoformat()), #.strftime("%Y-%m-%d %H:%M:%S %Z"),
 				"attributes": {
 					"message": "Structured message",
 				}
 			}
 		]
 	}
 ]

Once the structured data is validated, you can send it to LogScale using the following function, where the variable structured_data is the object you created above to store your JSON:

client.ingest_json_data(structured_data)

Support for unstructured data

Alternatively, you can send unstructured data to LogScale. Unstructured data is timestamped upon ingestion because it is a long, comma-delimited string of characters. Therefore, the timestamp you may or may not include in the log entry does not affect the recording timestamp. Below is an example of unstructured data:

unstructured_data = ["Unstructured message","Hello Python World",str(datetime.now(timezone("EST")).isoformat())]

You can send it to LogScale with the following function where unstructured_data is the object that contains your message. Please note the differences in syntax between the inclusion of structured and unstructured data.

client.ingest_messages(unstructured_data)

If you’ve followed all of the above steps, you should see messages in your LogScale instance. Happy logging!