Stars
文章
24
分类
1
评论
0

Malcolm 一款功能强大易于部署的网络流量分析工具套件

| 默认分类 •
字数总计 18448 | 阅读时长 46分钟 | 阅读量 538

Malcolm 一款功能强大易于部署的网络流量分析工具套件

项目地址:

GitHub

https://github.com/cisagov/Malcolm

malcolm下载地址:

https://github.com/cisagov/Malcolm/releases/download/v23.02.0/install.py

https://github.com/cisagov/Malcolm/releases/download/v23.02.0/malcolm_common.py

https://github.com/cisagov/Malcolm/releases/download/v23.02.0/malcolm_20230203_093307_360fbd60.tar.gz

Malcolm简介

Malcolm 是一个功能强大、易于部署的网络流量分析工具套件，用于完整的数据包捕获工件（PCAP 文件）和 Zeek 日志。

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs

Malcolm设计理念

  • 易于使用– Malcolm 接受以完整数据包捕获 (PCAP) 文件和 Zeek（以前称为 Bro）日志形式的网络流量数据。这些工件可以通过简单的基于浏览器的界面上传，也可以实时捕获并使用轻量级转发器转发给 Malcolm。在任何一种情况下，数据都会自动归一化、丰富和关联以进行分析。
  • 强大的流量分析——通过两个直观的界面提供对网络通信的可见性： OpenSearch Dashboards，一个灵活的数据可视化插件，具有数十个预建仪面板，提供网络协议概览；和 Arkime（以前称为 Moloch），这是一个强大的工具，用于查找和识别包含可疑安全事件的网络会话。
  • 简化部署——Malcolm 作为 Docker 容器集群运行，隔离沙箱，每个沙箱都服务于系统的专用功能。这种基于 Docker 的部署模型，结合一些用于设置和运行时管理的简单脚本，使 Malcolm 适合在各种平台和用例中快速部署，无论是在 Linux 服务器上长期部署安全运营中心 (SOC) 或用于个人参与的 Macbook 上的事件响应。
  • 安全通信——与 Malcolm 的所有通信，无论是来自用户界面还是来自远程日志转发器，都使用行业标准加密协议进行保护。
  • 许可许可– Malcolm 由几个广泛使用的开源工具组成，使其成为需要付费许可的安全解决方案的有吸引力的替代方案。
  • 扩展控制系统的可见性——虽然 Malcolm 非常适合通用网络流量分析，但其创建者认为社区特别需要能够深入了解工业控制系统 (ICS) 环境中使用的协议的工具。正在进行的 Malcolm 开发旨在为常见的 ICS 协议提供额外的解析器。

尽管构成 Malcolm 的所有开源工具都已经可用并普遍使用，但 Malcolm 提供了一个互连框架，使其大于各个部分的总和。虽然还有许多其他网络流量分析解决方案，从完整的 Linux 发行版（如 Security Onion）到许可产品（如 Splunk Enterprise Security），但 Malcolm 的创建者认为其易于部署和强大的工具组合填补了网络安全领域的空白,这将使公共和私营部门的许多人以及个人爱好者都可以进行网络流量分析。

简而言之，Malcolm 为完整的数据包捕获工件（PCAP 文件）和 Zeek 日志提供了一个易于部署的网络分析工具套件。虽然构建它需要 Internet 访问，但在运行时不需要访问网络。

安装部署Malcolm

先决条件

  • 运行 Ubuntu 22.04 的服务器。

  • 至少 16 GB RAM 和 4 个 CPU 内核。

  • 在服务器上配置了 root 密码。

创建 Malcolm 系统用户

首先，您需要创建一个专用用户帐户来运行 Malcolm，您可以使用以下命令创建它：

useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolm

接下来，使用以下命令设置用户密码：

passwd malcolm

接下来，使用以下命令检查 Malcolm 用户的 UID：

id malcolm

您应该看到以下输出：

uid=1000(malcolm) gid=1000(malcolm) groups=1000(malcolm),27(sudo)

安装 Malcolm Ubuntu 22.04

参考 ： Installation example using Ubuntu 22.04 LTS https://github.com/cisagov/Malcolm/blob/main/docs/ubuntu-install-example.md#InstallationExample

首先，将用户切换到 Malcolm 并使用以下命令下载最新版本的 Malcolm：

su - malcolm
git clone https://github.com/idaholab/Malcolm

下载完成后，将目录更改为下载的目录并使用以下命令开始安装 Malcolm：

cd Malcolm 
sudo ./scripts/install.py

在安装过程中，你会被问到几个问题，如下：

Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog']

"docker info" failed, attempt to install Docker? (Y/n): Y

Attempt to install Docker using official repositories? (Y/n): Y
Installing required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common']
Installing docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io']
Installation of docker packages apparently succeeded

Add a non-root user to the "docker" group?: Y

Enter user account: malcolm

Add another non-root user to the "docker" group?: n

"docker-compose version" failed, attempt to install docker-compose? (Y/n): Y

Install docker-compose directly from docker github? (Y/n): Y
Download and installation of docker-compose apparently succeeded
fs.file-max increases allowed maximum for file handles
fs.file-max= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y
fs.inotify.max_user_watches increases allowed maximum for monitored files
fs.inotify.max_user_watches= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y
fs.inotify.max_queued_events increases queue size for monitored files
fs.inotify.max_queued_events= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y
fs.inotify.max_user_instances increases allowed maximum monitor file watchers
fs.inotify.max_user_instances= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.max_map_count increases allowed maximum for memory segments
vm.max_map_count= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y
net.core.somaxconn increases allowed maximum for socket connections
net.core.somaxconn= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.dirty_background_ratio defines the percentage of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything
vm.dirty_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

/etc/security/limits.d/limits.conf increases the allowed maximums for file handles and memlocked segments
/etc/security/limits.d/limits.conf does not exist, create it? (Y/n): Y

安装 Malcolm 后，您可以继续下一步。

创建 Malcolm 管理员帐户

接下来，您需要创建一个管理帐户来访问 Malcolm Web 界面。

首先，将用户切换到 Malcolm 并使用以下命令导航到 Malcolm 目录,运行以下命令来创建一个管理员帐户：

su - malcolm
cd ~/Malcolm
./scripts/auth_setup

拉取 Malcolm Docker 镜像

接下来，您需要从 Docker Hub 注册表下载所有必需的 Docker 映像，您可以使用以下命令下载所有内容：

docker-compose pull

下载所有图像后，使用以下命令验证所有图像,您应该得到以下输出：

$ docker images
REPOSITORY                                                     TAG               IMAGE ID       CREATED      SIZE
malcolmnetsec/api                                              23.03.0           xxxxxxxxxxxx   3 days ago   158MB
malcolmnetsec/arkime                                           23.03.0           xxxxxxxxxxxx   3 days ago   816MB
malcolmnetsec/dashboards                                       23.03.0           xxxxxxxxxxxx   3 days ago   1.02GB
malcolmnetsec/dashboards-helper                                23.03.0           xxxxxxxxxxxx   3 days ago   184MB
malcolmnetsec/file-monitor                                     23.03.0           xxxxxxxxxxxx   3 days ago   588MB
malcolmnetsec/file-upload                                      23.03.0           xxxxxxxxxxxx   3 days ago   259MB
malcolmnetsec/filebeat-oss                                     23.03.0           xxxxxxxxxxxx   3 days ago   624MB
malcolmnetsec/freq                                             23.03.0           xxxxxxxxxxxx   3 days ago   132MB
malcolmnetsec/htadmin                                          23.03.0           xxxxxxxxxxxx   3 days ago   242MB
malcolmnetsec/logstash-oss                                     23.03.0           xxxxxxxxxxxx   3 days ago   1.35GB
malcolmnetsec/name-map-ui                                      23.03.0           xxxxxxxxxxxx   3 days ago   143MB
malcolmnetsec/netbox                                           23.03.0           xxxxxxxxxxxx   3 days ago   1.01GB
malcolmnetsec/nginx-proxy                                      23.03.0           xxxxxxxxxxxx   3 days ago   121MB
malcolmnetsec/opensearch                                       23.03.0           xxxxxxxxxxxx   3 days ago   1.17GB
malcolmnetsec/pcap-capture                                     23.03.0           xxxxxxxxxxxx   3 days ago   121MB
malcolmnetsec/pcap-monitor                                     23.03.0           xxxxxxxxxxxx   3 days ago   213MB
malcolmnetsec/postgresql                                       23.03.0           xxxxxxxxxxxx   3 days ago   268MB
malcolmnetsec/redis                                            23.03.0           xxxxxxxxxxxx   3 days ago   34.2MB
malcolmnetsec/suricata                                         23.03.0           xxxxxxxxxxxx   3 days ago   278MB
malcolmnetsec/zeek                                             23.03.0           xxxxxxxxxxxx   3 days ago   1GB

启动 Malcolm 服务

至此，Malcolm 所需的所有组件都已准备就绪，您现在可以使用以下命令启动 Malcolm 服务,等待一段时间以启动所有服务，启动所有服务后，使用以下命令验证所有正在运行的服务,您应该在以下输出中看到所有正在运行的容器：

./scripts/start

docker ps -a

CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS                             PORTS                                                                                            NAMES
840ea2b0e9ad   malcolmnetsec/nginx-proxy:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          0.0.0.0:443->443/tcp, 127.0.0.1:5601->5601/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:9200->9200/tcp   malcolm-nginx-proxy-1
dd5c8c63816c   malcolmnetsec/suricata:6.2.0            "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)                                                                                                           malcolm-suricata-1
3112e1bd8f73   malcolmnetsec/filebeat-oss:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          127.0.0.1:5045->5045/tcp                                                                         malcolm-filebeat-1
c93cfe93ad7e   malcolmnetsec/file-upload:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          80/tcp, 127.0.0.1:8022->22/tcp                                                                   malcolm-upload-1
18ee20b46f3c   malcolmnetsec/dashboards:6.2.0          "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          5601/tcp                                                                                         malcolm-dashboards-1
2c34206c06e4   malcolmnetsec/zeek:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)                                                                                                           malcolm-zeek-1
41103ef99ce1   malcolmnetsec/logstash-oss:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp                                                     malcolm-logstash-1
0408f42a76c3   malcolmnetsec/dashboards-helper:6.2.0   "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          28991/tcp                                                                                        malcolm-dashboards-helper-1
3e78024620de   malcolmnetsec/arkime:6.2.0              "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          8000/tcp, 8005/tcp, 8081/tcp                                                                     malcolm-arkime-1
58cd869beced   malcolmnetsec/pcap-monitor:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          30441/tcp                                                                                        malcolm-pcap-monitor-1
1040fa8bd6df   malcolmnetsec/file-monitor:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          3310/tcp, 8440/tcp                                                                               malcolm-file-monitor-1
25c83f14413d   malcolmnetsec/zeek:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-zeek-live-1
b321a96c0362   malcolmnetsec/api:6.2.0                 "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          5000/tcp                                                                                         malcolm-api-1
0f1f4ac023f9   malcolmnetsec/name-map-ui:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          8080/tcp                                                                                         malcolm-name-map-ui-1
ba4d553cf6b5   malcolmnetsec/suricata:6.2.0            "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-suricata-live-1
e4637d0ec04d   malcolmnetsec/opensearch:6.2.0          "/usr/local/bin/dock…"   17 minutes ago   Up 13 minutes (health: starting)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp                                                           malcolm-opensearch-1
ac002e31d9be   malcolmnetsec/htadmin:6.2.0             "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          80/tcp                                                                                           malcolm-htadmin-1
7223d5244a7b   malcolmnetsec/pcap-capture:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-pcap-capture-1
971931b21788   malcolmnetsec/freq:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          10004/tcp                                                                                        malcolm-freq-1
您还可以使用以下命令验证所有侦听端口：

您还可以使用以下命令验证所有侦听端口,应该看到以下输出：

ss -atlnp | grep -i docker


LISTEN 0      65535      127.0.0.1:5601       0.0.0.0:*    users:(("docker-proxy",pid=7480,fd=4))   
LISTEN 0      65535        0.0.0.0:488        0.0.0.0:*    users:(("docker-proxy",pid=7519,fd=4))   
LISTEN 0      65535      127.0.0.1:9200       0.0.0.0:*    users:(("docker-proxy",pid=7443,fd=4))   
LISTEN 0      65535      127.0.0.1:5044       0.0.0.0:*    users:(("docker-proxy",pid=6247,fd=4))   
LISTEN 0      65535      127.0.0.1:5045       0.0.0.0:*    users:(("docker-proxy",pid=7063,fd=4))   
LISTEN 0      65535      127.0.0.1:8022       0.0.0.0:*    users:(("docker-proxy",pid=6826,fd=4))   
LISTEN 0      65535        0.0.0.0:443        0.0.0.0:*    users:(("docker-proxy",pid=7567,fd=4))

访问 Malcolm

现在可以使用 URL https://your-server-ip/dashboards/ 访问 Malcolm OpenSearch 仪表板，您将被要求提供您的管理员用户名和密码：

提供您的管理员用户名、密码，然后单击登录按钮，您应该在以下屏幕上看到 OpenSearch 仪表板：

要访问 Malcolm 捕获文件和日志存档上传屏幕，请键入 URL https://your-server-ip/upload/。

要访问主机和子网名称映射编辑器，请键入 URL https://your-server-ip/name-map-ui/。

要访问帐户管理屏幕，请使用 URL https://your-server-ip:488/

中文版教程参考文档 https://www.ddosi.org/malcolm/

通过Malcolm抓取本地网卡流量

在本地网络接口上捕获流量

Malcolm 的pcap-capture容器可以捕获一个或多个本地网络接口上的流量，并定期轮换这些文件进行处理。pcap-captureDocker 容器以附加权限（、IPC_LOCK、NET_ADMIN和NET_RAW）启动SYS_ADMIN，以便它能够以混杂模式打开网络接口以进行捕获。

PCAP_文件中带有前缀的环境变量docker-compose.yml决定了本地数据包捕获行为。也可以通过运行./scripts/install.py --configure并对“ Should Malcolm capture network traffic to PCAP files?.”回答“是”来配置本地捕获。

配置完毕后重启服务器

Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): y

Setting 10g for OpenSearch and 3g for Logstash. Is this OK? (Y/n): y

Setting 3 workers for Logstash pipelines. Is this OK? (Y/n): y

Restart Malcolm upon system or Docker daemon restart? (y/N): y
1: no
2: on-failure
3: always
4: unless-stopped
Select Malcolm restart behavior (unless-stopped): 4

Require encrypted HTTPS connections? (Y/n): y

Will Malcolm be running behind another reverse proxy (Traefik, Caddy, etc.)? (y/N): n

Specify external Docker network name (or leave blank for default networking) (): 

Authenticate against Lightweight Directory Access Protocol (LDAP) server? (y/N): n

Configure OpenSearch index state management? (y/N): n

Automatically analyze all PCAP files with Zeek? (Y/n): y

Perform reverse DNS lookup locally for source and destination IP addresses in Zeek logs? (y/N): n

Perform hardware vendor OUI lookups for MAC addresses? (Y/n): y

Perform string randomness scoring on some fields? (Y/n): y

Expose OpenSearch port to external hosts? (y/N): n

Expose Logstash port to external hosts? (y/N): n

Forward Logstash logs to external OpenSearch instance? (y/N): n

Enable file extraction with Zeek? (y/N): y
1: none
2: known
3: mapped
4: all
5: interesting
Select file extraction behavior (none): 5
1: quarantined
2: all
3: none
Select file preservation behavior (quarantined): 1

Scan extracted files with ClamAV? (y/N): y

Scan extracted files with Yara? (y/N): y

Scan extracted PE files with Capa? (y/N): y

Lookup extracted file hashes with VirusTotal? (y/N): n

Download updated scanner signatures periodically? (Y/n): y

Should Malcolm capture network traffic to PCAP files? (y/N): y

Specify capture interface(s) (comma-separated): eth0   #在此输入需要被抓包的接口名称

Capture packets using netsniff-ng? (Y/n): y

Capture packets using tcpdump? (y/N): n

Malcolm has been installed to /home/user/Malcolm. See README.md for more information.
Scripts for starting and stopping Malcolm and changing authentication-related settings can be found
in /home/user/Malcolm/scripts.
文章作者： Stars
本文链接：
版权声明： 本站所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 Stars！
默认分类 network monitor
喜欢就支持一下吧