Curtis Collicutt Curtis Collicutt TIDAL SERIES Newsletter TIDAL SERIES Newsletter LinkedIn LinkedIn GitHub GitHub

falcoctl

tl;dr: falcoctl is an attempt to make it easier to distribute and upgrade Falco artifacts, such as rules and plugins.

What is Falco?

Falco is a cloud native security tool that provides runtime security across hosts, containers, Kubernetes, and cloud environments. It leverages custom rules on Linux kernel events and other data sources through plugins, enriching event data with contextual metadata to deliver real-time alerts. Falco enables the detection of abnormal behavior, potential security threats, and compliance violations. - Falco

Falco is like a security camera for your servers, your Kubernetes nodes, and your cloud environments. It’s a way to detect and respond to security threats in real time.

The underlying technology relies on rules, which are really a set of configuration files. The point of these security rules is that the world around us is changing, and so we need new and updated rules to help us detect and respond to new and emerging threats. If you simply deploy the standard open source Falco rules and never change them, or do not add your own custom rules based on your own unique needs, you’re missing out on much of the power of Falco.

Once you have the base Falco technology, which is of course completely invaluable, the real work is in creating and managing the threat detection rules and then responding to alerts.

Distributed Systems

It’s important to note that Falco deployments are like a single security camera. They are not, by default, a distributed collection of security cameras that all send their information to a central location, nor do they by default get their configuration from a central location.

This presents a challenge in that organizations often run many hundreds, thousands, or even tens of thousands of nodes, each with their own Falco installation. Additionally, Falco can also have plugins installed on each node, which further complicates the distribution of configuration.

  • Configuration files - As mentioned, Falco is a rules based engine–so it needs the rules to work, and thus we have to manage those rules, and distribute them to potentially tens of thousands of nodes.

How do we distribute these rules?

  • Plugins - Over the last couple of years Falco has added a plugin framework which allows you in many ways to vastly increase the capacity of what Falco can monitor, i.e. historically Falco has looked at system calls, events from the Linux kernel, but with plugins you can add in other data sources, such as audit logs.

So we have rules and plugins to manage, but how?

That is where falcoctl comes in.

What is falcoctl?

Here’s a good blog post from the Falco team that explains it well:

Since the launch of the plugin framework in January 2022, our adopters have requested an out-of-the-box solution to manage the lifecycle of rules (installation, updates). We heard your request and also created a guide to help you smoothly install the plugins. The Falco maintainers proposed the following solution to help with these issues: falcoctl. Falcoctl is a CLI tool that performs several useful tasks for Falco.

falcoctl effectively manages artifacts for Falco, including rules, plugins, and configuration.

  • Install the falcoctl binary CLI application
LATEST=$(curl -sI https://github.com/falcosecurity/falcoctl/releases/latest | awk '/location: /{gsub("\r","",$2);split($2,v,"/");print substr(v[8],2)}')
curl --fail -LS "https://github.com/falcosecurity/falcoctl/releases/download/v${LATEST}/falcoctl_${LATEST}_linux_amd64.tar.gz" | tar -xz
sudo install -o root -g root -m 0755 falcoctl /usr/local/bin/falcoctl
  • Run it
root@falco:~# which falcoctl
/usr/local/bin/falcoctl
root@falco:~# falcoctl

     __       _                _   _ 
    / _| __ _| | ___ ___   ___| |_| |
   | |_ / _  | |/ __/ _ \ / __| __| |
   |  _| (_| | | (_| (_) | (__| |_| |
   |_|  \__,_|_|\___\___/ \___|\__|_|
									 
	
The official CLI tool for working with Falco and its ecosystem components

Usage:
  falcoctl [command]

Available Commands:
  artifact    Interact with Falco artifacts
  completion  Generate the autocompletion script for the specified shell
  driver      Interact with falcosecurity driver
  help        Help about any command
  index       Interact with index
  registry    Interact with OCI registries
  tls         Generate and install TLS material for Falco
  version     Print the falcoctl version information

Flags:
      --config string       config file to be used for falcoctl (default "/etc/falcoctl/falcoctl.yaml")
  -h, --help                help for falcoctl
      --log-format string   Set formatting for logs (color, text, json) (default "color")
      --log-level string    Set level for logs (info, warn, debug, trace) (default "info")

Use "falcoctl [command] --help" for more information about a command.
root@falco:~#

Manage Falco Artifacts with falcoctl

What we want to do is use falcoctl to manage our Falco artifacts. There are several components and features of falcoctl that we can use to do this.

  • Add an “index”
root@falco:~# sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml
2024-11-01 19:30:11 INFO  Adding index
                      ├ name: falcosecurity
                      └ path: https://falcosecurity.github.io/falcoctl/index.yaml
2024-11-01 19:30:11 INFO  Index successfully added
  • Review the config
root@falco:~# cat /etc/falcoctl/falcoctl.yaml 
artifact:
    follow:
        every: 6h0m0s
        falcoversions: http://localhost:8765/versions
        refs:
            - falco-rules:3
driver:
    hostroot: /
    name: falco
    repos:
        - https://download.falco.org/driver
    type:
        - modern_ebpf
    version: 7.3.0+driver
indexes:
    - name: falcosecurity
      url: https://falcosecurity.github.io/falcoctl/index.yaml
      backend: ""
  • Search for Falco artifacts
root@falco:~# falcoctl artifact search falco
INDEX        	ARTIFACT              	TYPE     	REGISTRY	REPOSITORY
falcosecurity	falco-incubating-rules	rulesfile	ghcr.io 	falcosecurity/rules/falco-incubating-rules
falcosecurity	falco-rules           	rulesfile	ghcr.io 	falcosecurity/rules/falco-rules
falcosecurity	falco-sandbox-rules   	rulesfile	ghcr.io 	falcosecurity/rules/falco-sandbox-rules
  • Search for Kubernetes artifacts
root@falco:~# falcoctl artifact search kubernetes
INDEX        	ARTIFACT          	TYPE     	REGISTRY	REPOSITORY
falcosecurity	k8saudit-eks      	plugin   	ghcr.io 	falcosecurity/plugins/plugin/k8saudit-eks
falcosecurity	k8saudit-gke      	plugin   	ghcr.io 	falcosecurity/plugins/plugin/k8saudit-gke
falcosecurity	k8saudit-gke-rules	rulesfile	ghcr.io 	falcosecurity/plugins/ruleset/k8saudit-gke
falcosecurity	k8saudit-rules    	rulesfile	ghcr.io 	falcosecurity/plugins/ruleset/k8saudit
falcosecurity	k8smeta           	plugin   	ghcr.io 	falcosecurity/plugins/plugin/k8smeta
falcosecurity	k8saudit          	plugin   	ghcr.io 	falcosecurity/plugins/plugin/k8saudit
  • Install the falco-rules artifact
root@falco:~# falcoctl artifact install falco-rules
2024-11-01 19:33:05 INFO  Resolving dependencies ... 
2024-11-01 19:33:05 INFO  Installing artifacts
                      └ refs: [ghcr.io/falcosecurity/rules/falco-rules:latest]
2024-11-01 19:33:05 INFO  Preparing to pull artifact
                      └ ref: ghcr.io/falcosecurity/rules/falco-rules:latest
2024-11-01 19:33:06 INFO  Pulling layer 8da145602705 
2024-11-01 19:33:06 INFO  Pulling layer b3990bf0209c                                            
2024-11-01 19:33:06 INFO  Pulling layer de2cd036fd7f                                            
2024-11-01 19:33:06 INFO  Verifying signature for artifact                                      
                      └ digest: ghcr.io/falcosecurity/rules/falco-rules@sha256:de2cd036fd7f9bb87de5d62b36d0f35ff4fa8afbeb9a41aa9624e5f6f9a004e1
2024-11-01 19:33:07 INFO  Signature successfully verified! 
2024-11-01 19:33:07 INFO  Extracting and installing artifact
                      ├ type: rulesfile
                      └ file: falco_rules.yaml.tar.gz
2024-11-01 19:33:07 INFO  Artifact successfully installed                                       
                      ├ name: ghcr.io/falcosecurity/rules/falco-rules:latest
                      ├ type: rulesfile
                      ├ digest: sha256:de2cd036fd7f9bb87de5d62b36d0f35ff4fa8afbeb9a41aa9624e5f6f9a004e1
                      └ directory: /etc/falco

Following as a Daemon

A great feature of falcoctl is its ability to run as a daemon to periodically check the artifacts’ repositories and automatically install new versions. - falcoctl

So this would be the key feature of falcoctl, because, again, it’s great to have the technology, but what does it do to make our lives easier?

When falcoctl tracks (follows) an artifact, it will automatically install new versions of that artifact without any human intervention, which I think is a good thing. Of course, we want to make sure that what we deploy works as expected, but that is a whole other can of worms.

So, with all that in mind, let’s create a service to run falcoctl as a daemon.

  • Create a service
cat << 'EOF' | sudo tee /etc/systemd/system/falcoctl.service
[Unit]
Description=Falcoctl
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/local/bin/falcoctl artifact follow
EOF
systemctl enable falcoctl
systemctl start falcoctl
  • Check the status now that it is running
root@falco:/etc/falcoctl# systemctl status falcoctl
● falcoctl.service - Falcoctl
     Loaded: loaded (/etc/systemd/system/falcoctl.service; static)
     Active: active (running) since Fri 2024-11-01 19:37:32 UTC; 11s ago
   Main PID: 5544 (falcoctl)
      Tasks: 7 (limit: 2296)
     Memory: 25.4M (peak: 25.6M)
        CPU: 674ms
     CGroup: /system.slice/falcoctl.service
             └─5544 /usr/local/bin/falcoctl artifact follow

Nov 01 19:37:33 falco falcoctl[5544]:                       └ artifact: ghcr.io/falcosecurity/r>
Nov 01 19:37:33 falco falcoctl[5544]: 2024-11-01 19:37:33 INFO  Found new artifact version
Nov 01 19:37:33 falco falcoctl[5544]:                       ├ followerName: ghcr.io/falcosecuri>
Nov 01 19:37:33 falco falcoctl[5544]:                       └ tag: 3
Nov 01 19:37:35 falco falcoctl[5544]: 2024-11-01 19:37:35 INFO  Artifact correctly installed
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ followerName: ghcr.io/falcosecuri>
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ artifactName: ghcr.io/falcosecuri>
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ type: rulesfile
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ digest: sha256:de2cd036fd7f9bb87d>
Nov 01 19:37:35 falco falcoctl[5544]:                       └ directory: /etc/falco
root@falco:/etc/falcoctl# systemctl status falcoctl --no-pager
● falcoctl.service - Falcoctl
     Loaded: loaded (/etc/systemd/system/falcoctl.service; static)
     Active: active (running) since Fri 2024-11-01 19:37:32 UTC; 15s ago
   Main PID: 5544 (falcoctl)
      Tasks: 7 (limit: 2296)
     Memory: 25.4M (peak: 25.6M)
        CPU: 674ms
     CGroup: /system.slice/falcoctl.service
             └─5544 /usr/local/bin/falcoctl artifact follow

Nov 01 19:37:33 falco falcoctl[5544]:                       └ artifact: ghcr.io/falcosecu…ules:3
Nov 01 19:37:33 falco falcoctl[5544]: 2024-11-01 19:37:33 INFO  Found new artifact version
Nov 01 19:37:33 falco falcoctl[5544]:                       ├ followerName: ghcr.io/falco…ules:3
Nov 01 19:37:33 falco falcoctl[5544]:                       └ tag: 3
Nov 01 19:37:35 falco falcoctl[5544]: 2024-11-01 19:37:35 INFO  Artifact correctly installed
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ followerName: ghcr.io/falco…ules:3
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ artifactName: ghcr.io/falco…ules:3
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ type: rulesfile
Nov 01 19:37:35 falco falcoctl[5544]:                       ├ digest: sha256:de2cd036fd7f…a004e1
Nov 01 19:37:35 falco falcoctl[5544]:                       └ directory: /etc/falco
Hint: Some lines were ellipsized, use -l to show in full.

Looks good.

Build Your Own Falcoctl Artifact

Let’s push a rules file into an OCI registry.

  • First create a rules file
cat << 'EOF' | sudo tee ./custom_rules.yaml
- list: falco_binaries
  items: [falcoctl]
EOF
  • Login to the registry
root@falco:/etc/falcoctl# falcoctl registry auth basic some_registry -u 'your_user' -p 'some_password'
2024-11-01 19:52:43 INFO  Login succeeded registry: some_registry user: your_user
  • Now push that single rules file to the registry

NOTE: We often think of “container images” as the only thing we can push to an OCI registry, but we can actually push any OCI-compliant artifact to an OCI registry, and over time we will see much more use of this.

export OCI_ARTIFACT_VERSION=latest
export OCI_REGISTRY=some_registry
export OCI_REPOSITORY=some_repo/falco-rules
export RULESET_FILE=custom_rules.yaml
falcoctl registry push \
    --config /dev/null \
    --type rulesfile \
    --version ${OCI_ARTIFACT_VERSION} \
    ${OCI_REGISTRY}/${OCI_REPOSITORY}:${OCI_ARTIFACT_VERSION} \
    ${RULESET_FILE}
  • Resulting output
root@falco:/etc/falcoctl# export OCI_ARTIFACT_VERSION=latest
export OCI_REGISTRY=some_registry
export OCI_REPOSITORY=some_repo/falco-rules
export RULESET_FILE=custom_rules.yaml
falcoctl registry push \
    --config /dev/null \
    --type rulesfile \
    --version ${OCI_ARTIFACT_VERSION} \
    ${OCI_REGISTRY}/${OCI_REPOSITORY}:${OCI_ARTIFACT_VERSION} \
    ${RULESET_FILE}
2024-11-01 20:15:19 INFO  Preparing to push artifact
                      ├ name: some_registry/some_repo/falco-rules:latest
                      └ type: rulesfile
2024-11-01 20:15:19 ERROR open custom_rules.yaml: no such file or directory 
root@falco:/etc/falcoctl# cd 
root@falco:~# falcoctl registry push     --config /dev/null     --type rulesfile     --version ${OCI_ARTIFACT_VERSION}     ${OCI_REGISTRY}/${OCI_REPOSITORY}:${OCI_ARTIFACT_VERSION}     ${RULESET_FILE}
2024-11-01 20:15:24 INFO  Preparing to push artifact
                      ├ name: some_registry/some_repo/falco-rules:latest
                      └ type: rulesfile
2024-11-01 20:15:24 INFO  Parsing dependencies from:  rulesfile: custom_rules.yaml
2024-11-01 20:15:24 WARN  No dependencies were provided by the user and none were found in the
                      │   rulesfile.
2024-11-01 20:15:24 INFO  Parsing requirements from:  rulesfile: custom_rules.yaml
2024-11-01 20:15:24 WARN  No requirements were provided by the user and none were found in the
                      │   rulesfile.
2024-11-01 20:15:24 INFO  Pushing layer d5c35695420a 
2024-11-01 20:15:26 INFO  97c38f4c17c8: layer already exists                                    
2024-11-01 20:15:26 INFO  Pushing layer c891d7815e0a 
2024-11-01 20:15:26 INFO  Artifact pushed                                                       
                      ├ name: some_registry/some_repo/falco-rules:latest
                      ├ type: rulesfile
                      └ digest: sha256:c891d7815e0a30a1a73e026aea4603503b0a12df9bc8b7efc38f61de2d77bd6b
  • Now follow the artifact by updating the falcoctl.yaml file

NOTE: Below I added some_registry/some_repo/falco-rules:latest to the refs section.

root@falco:/etc/falcoctl# cat falcoctl.yaml
artifact:
    follow:
        every: 6h0m0s
        falcoversions: http://localhost:8765/versions
        refs:
            - falco-rules:3
            - some_registry/some_repo/falco-rules:latest
driver:
    hostroot: /
    name: falco
    repos:
        - https://download.falco.org/driver
    type:
        - modern_ebpf
    version: 7.3.0+driver
indexes:
    - name: falcosecurity
      url: https://falcosecurity.github.io/falcoctl/index.yaml
      backend: ""
  • Restart the falcoctl service
systemctl restart falcoctl
  • Check the status of the falcoctl service
systemctl status falcoctl
  • And check if the new rules file was installed
root@falco:~# cat /etc/falco/custom_rules.yaml 
- list: falco_binaries
  items: [falcoctl]

It was!

  • There are falcoctl logs as well
root@falco:~# grep some_repo /var/log/syslog
2024-11-01T20:16:57.198277+00:00 falco falcoctl[5819]:                       ├ artifact: some_registry/some_repo/falco-rules:latest
2024-11-01T20:16:57.200084+00:00 falco falcoctl[5819]:                       └ artifact: some_registry/some_repo/falco-rules:latest
2024-11-01T20:16:57.433872+00:00 falco falcoctl[5819]:                       ├ followerName: some_registry/some_repo/falco-rules:latest
2024-11-01T20:16:59.154603+00:00 falco falcoctl[5819]:                       ├ followerName: some_registry/some_repo/falco-rules:latest
2024-11-01T20:16:59.154713+00:00 falco falcoctl[5819]:                       ├ artifactName: some_registry/some_repo/falco-rules:latest

Easy peasy. Well, at least getting it started. Now you need to automate the creation of the artifact, and of course write your own rules, which is the hard part once you have the technology.

Conclusion

In theory, you could deploy falcoctl as a daemon to every host you have, and configure it to check for your new rules basically on a cronjob-like schedule. Need every host to have a new rules file? No problem, just push the new rules file to the OCI registry and the falcoctl daemon will pull it down and install it.

Will that scale? I don’t know. Maybe…maybe not.

However, even once you have Falco installed, and then use falcoctl to have a distribution mechanism in place, you still need to customize your rules. Technology only gets us so far.

PS. Sysdig

I work at Sysdig, and while we use and support Falco, we’ve also built our enterprise product to have a much wider use case (see CNAPP which includes CSPM, CIEM, vulnerability management, and more), and to be considerably more scalable than vanilla open source Falco. Sysdig does all the heavy lifting, rule distribution, etc., etc. And most importantly, in the context of threat detection, we write the rules for you as well.

Further Reading