MuleSoft Anypoint Monitoring and Mule on a Raspberry Pi

MuleSoft Anypoint Monitoring is an amazing tool to monitor Mule apps. It works flawlessly as long as it runs within the supported configuration. However, it’s not so fun when you are experimenting with Mule 4 on a Raspberry Pi since Anypoint Monitoring does not support Raspbian from the get-go. Or perhaps I just need to pull the experimentation lever a little more and get it to work anyway.

Well, here’s a teaser of the end result. 🙂

<Disclaimer>
I’m currently an employee at MuleSoft, a Salesforce company at this point of posting this article. Also, note that this post is a result of my own experimentation and is NOT a supported configuration by MuleSoft.
</Disclaimer>

---

Getting Anypoint Monitoring to work with Mule on a Raspberry Pi

It took a little bit of reserve engineering to understand why it didn’t work. But more importantly, I also need to figure out how the service is initiated as well so I can create a custom startup script without breaking any of the default configurations.

I will not be describing it in detail here, but suffice to say, after understanding what the installation, setup and am-service script do, I found out that Anypoint Monitoring essentially uses a filebeat-god (go daemon) binary to run as a service to harvest the logs, and push it to the Anypoint Platform cloud.

But of course, there lies the problem. The bundled filebeat-god binary is only available for Linux on x86 architecture, not ARM.

Executing the binary on the Raspberry Pi (Exec format error) error.

Looking around the /am directory, you will also find quite a number of filebeat binary bundled along with it. But again, none for ARM. The reason is actually because Elastic today does not yet officially provide nor support ARM architecture builds. But there’s nothing to stop us from building it on our own anyway. 🙂

First, we need to figure out which version of filebeat we actually need. From my experimentation, it seems that the ELK stack is very sensitive to the version you use, so its best to build the same version for our ARM build as well. To do that, I just ran ‘{MULE_HOME}/am/filebeat/osx/filebeat version‘ to find out the version off my Mac. Take note that this is as of version 2.2.3 of the Anypoint Monitoring’s agent.

Building Filebeat for ARM

I will not take full credit here even if the build process is actually pretty straight forward. When searching on the web to see if there are already ARM builds of filebeat, I came across this site that describes at length the whole process. The only thing here is that it refers to older versions of Go and it checks out a much older version of filebeat to build. So I will describe the exact versions and steps I took to get my build working for Anypoint Monitoring here.

1. Installing the pre-requisites

According the site I mentioned above, the author states that python-pip and virtualenv are required for the build. I can’t really test this as I already have them installed on my Raspberry Pi. But to do so, just run the following apt commands

sudo apt-get install python-pip git
sudo pip install virtualenv

2. Installing Anypoint Monitoring

Install Anypoint Monitoring as per usual and run the am/bin/install and am/bin/setup scripts too. Do the necessary restart of Mule Server as well as how you would install it as per normal. There are a bunch of configurations that the script will do to prepare the Anypoint Monitoring agent to do what it needs to do. BTW, select ‘N’ to NOT install it as a service when you run the installation script.

Note that I’ve installed Mule at /opt/mule/mule-enterprise-standalone-4.3.0. Also, I’m assuming that your Mule server is already running.

cd /opt/mule/mule-enterprise-standalone-4.3.0
wget https://s3.amazonaws.com/cdn-anypoint-prod/artifacts/monitoring-center-ui/app/hybrid/am-2.2.3.0.zip
unzip am-2.2.3.0.zip
cd am
./bin/install
/opt/mule/mule-enterprise-standalone-4.3.0/bin/mule restart
./bin/setup

This should get all the necessary certificates and configuration between the Anypoint Monitoring agent running on the Raspberry Pi and the Anypoint Platform cloud to work. Just that, at this point, the service will fail to run.

2. Installing Go

Filebeat is a Go program. So you will need to install Go in order to build it from its source. I installed the latest stable version 1.14.4 as of this post. You can get the Go binaries at https://golang.org/dl/. Do note that there are two binaries for the ARM architecture, arm64 for ARMv8 and armv6l for ARMv6 and v7. Run uname -a to find out which version you need. My Raspberry Pi 4 with Raspbian Buster shoots out armv7l.

Installing it is as simple as untar-ing the Go binaries and adding the go/bin directory to your PATH environment variable.

What I prefer to do is to untar the go binaries to a ~/go/go-{version}/ directory and then create a symbolic link of the current version to /usr/lib/go. Then I add the /usr/lib/go/bin directory to the PATH environment variable.

cd ~/Downloads
wget https://golang.org/dl/go1.14.4.linux-armv6l.tar.gz
mkdir ~/gosdk
cd ~/gosdk
tar -xvf ~/Downloads/go1.14.4.linux-armv6l.tar.gz
mv go go-1.14.4
sudo ln -s $PWD/go-1.14.4 /usr/lib/go
echo "export PATH=$PATH:/usr/local/go/bin" >> .bashrc
source /etc/bash.bashrc

You can confirm that its properly installed using the go version command.

Building filebeat 6.2.4 for the Raspberry Pi

Again, credit goes to https://www.elasticice.net/posts/filebeat_on_raspberry_pi3/ for the guide. I’ve edited it here to build version 6.2.4 that is required to work for Anypoint Monitoring version 2.2.3.0. I’ve tried the latest version 7.8.0 and found that a fair number of editing of the filebeat configuration is required to get it to work.

cd ~
mkdir ~/go
export GOPATH=$HOME/go
mkdir -p ${GOPATH}/src/github.com/elastic
cd ${GOPATH}/src/github.com/elastic
git clone https://github.com/elastic/beats.git
cd beats/

At this point, you would have the beats source code on your Raspberry Pi. As of this post, the latest version of beats is 7.8.0. Remember, we need to build version 6.2.4. To do so, we need to check out the right release from the git repository. The instructions at the site I came across used the branch id as I found in the screenshot below. But the easier way is to just git checkout the v6.2.4 tag instead.

git checkout v6.2.4
cd filebeat
make

The first build will take a slightly longer time as the build process needs to download all the dependencies. Subsequent builds are a lot faster, though once you’ve built it, there is little reason to rebuild unless you need a different version of filebeat.

When done, you will find the filebeat binary in the same directory. Run filebeat version to confirm it is the correct version and that it is working as expected.

If you don’t want to build it yourself. Here’s the one that I’ve built on my Raspberry Pi. (filebeat-arm-6.2.4)

Move the file to somewhere at the Anypoint Monitoring directory within your Mule home directory. I’ve installed Mule at /opt/mule/mule-enterprise-standalone-4.3.0.

mkdir /opt/mule/mule-enterprise-standalone-4.3.0/am/filebeat/linux/arm6l/
mv filebeat /opt/mule/mule-enterprise-standalone-4.3.0/am/filebeat/linux/arm6l/filebeat

Creating a custom Anypoint Monitoring start script for the Raspberry Pi

The original bundle uses the daemonised filebeat-god version of filebeat. As of now, I’ve only tested the regular filebeat binary and have not tried to build filebeat-god for ARM. Instead, I was planning to just use nohup to keep the process running in the background after manually starting up the service. I personally think that this is a lot easier to manage and debug.

To do so, all we need now is to write a custom Anypoint Monitoring filebeat startup script. The following is the script I wrote. You can name the script with any name you want. I “creatively” named it start_am.sh.

#!/bin/bash
# Import variables
# Edit the MULE_HOME to your mule installation directory
MULE_HOME=/opt/mule/mule-enterprise-standalone-4.3.0
AM_HOME=${MULE_HOME}/am
source ${AM_HOME}/bin/tools/serviceHelper
source ${AM_HOME}/bin/tools/message
FILEBEAT_DATA=${MULE_HOME}/.mule/.am/filebeat/data
FILEBEAT_LOGS=${AM_HOME}/filebeat/logs
${AM_HOME}/filebeat/linux/arm6l/filebeat -strict.perms=false -c ${AM_HOME}/config/filebeat.yml -path.home ${AM_HOME} -path.data ${FILEBEAT_DATA} -path.logs ${FILEBEAT_LOGS}

Again, if you need to just download a copy, here is my start_am.sh script. You will need to allow the file to be executed.

sudo chmod +x start_am.sh

At this point, you’re ready to roll. As mentioned earlier, I just use nohup to keep the filebeat process running even when my terminal screen is killed.

nohup ./start_am.sh &

Don’t worry if you see the nohup: ignoring input…. message. Just press enter to get back to the prompt. Nohup returns the process ID (PID) if you want to know which one to kill. Or you can just use ps aux | grep <your_username> to find the process as well.

To confirm that Anypoint Monitoring is now running properly on your Raspberry Pi, head over to the am/filebeat/logs directory and check the content of the logs.

tail -f /opt/mule/mule-enterprise-standalone-4.3.0/am/filebeat/log/filebeat

You should see a bunch of log content as per the screenshot below. When you see the [monitoring] logs coming through the harvester, all is good!

Anypoint Monitoring does play well with Mule on Raspberry Pi!

When it works, it is really cool. You can see the series of screenshots below of the Hello-world Mule 4 API that is running on my Raspberry Pi.

In Conclusion

Never let unsupported configurations hold you back. The whole point of an open platform like Elastic is that you can always find a way to get it working. 😆 Plus, a little bit of vi <installation/setup file> can be insightful most of the time.

Again. Do note that this is not a supported configuration by MuleSoft. Therefore, do this as your own “risk”. Having said that, if you are really using Mule on a Raspberry Pi 4 as an actual real-life use-case, do reach out to me if it’s something you can share. I would be really keen to understand what you are doing.

Recommended Reading

Wondering if you also get Anypoint Monitoring to run on your Mac? TL;DR: Yes you can! Find out how I did that here.

You can also check out more MuleSoft related posts here on my MuleSoft Cookbook section.