Purpose of Project

Create an easy-install honey pot that any Security Engineer or Systems Administrator can easily write custom plug-ins for applications or services running on their network. The honey pot needs to display critical data to the user in an easy to read and navigate format. This tool is useful to show what kind of attacks are going on and verify that the actual services are guarded appropriately.

Features and Libraries Used

Designed to Run on RaspberryPi Zero

Application built around running on a RaspberryPi Zero to make it easy to deploy in any physical environment as well as being cost efficient. Packages for both Debian and CentOS available to increase flexibility for users.

Displaying Reporting Data with Grafana

HoneyPotR will come packaged with some basic graphs and output configured by default. This allows any user to get up and running with HoneyPotR quickly by providing pre-built graphs so that information can be displayed for real-time attacks immediately. Grafana also makes it very easy to build your own graphs in order to display any metrics that fit your needs.

Using Docker for Builds and CI Management

Docker makes it easy to develop, test, build and manage a Continuous Integration development cycle. By using Docker we have given our clients the ability to run HoneyPotR on any platform. If running a Raspberry Pi does no fit your needs, Docker allows you to run our Debian and CentOS packages on any system such as Microsoft and Apple OS X. Docker can also be used to deploy Grafana on a machine other than the Raspberry Pi.

Passive OS Fingerprinting with P0f

By passively listening to the TCP Handshake between our honey pot and the attacker we are able to pull the following data:

Geoip

Using a simple cURL call to ipinfo.io we are able to gather useful information regarding the location of the attacker and the type of service they may be using. We are then mapping the information from this call to a world map that is able to display the location of all attacks made on HoneyPotR. This world map is accessible via an endpoint in HoneyPotR's REST API so that a user is able to pass a days GET parameter to narrow down the amount of points showing on the map.

REST API

Users will have access to reporting data through a RESTful API. Attack data resources are represented as JSON Arrays which makes it easy to narrow down and display exactly what information a user may want to access using HTTP GET calls. This API allows flexibility for our clients to get reporting data however fits best for their environment. For example if a client already had a GUI for reporting traffic on their network it would be easy for them to make calls to HoneyPotR via the REST API without the need to use Grafana.

How It Works

Framework

The framework class is responsible for managing the flow of information throughout the system. Once the application is started the framework will initiate a NetworkListener that will attempt to complete the TCP handshake of any SYN packets that are addressed to the machine HoneyPotR is running on. Once this handshake is complete, the framework is able to determine which plug-in is being called based on the port number set in the configuration. As the plug-in is running and recording data, it is being passed back down to the framework which then passes it on to the DataManager.

The framework is also responsible for handling the multi-threaded nature of our application, the safe and graceful shutdown of the application and all current connections. Multiple threads are needed to ensure that each running plug-in is able to handle connections from multiple attackers at once while being able to simultaneously write the data that is being collected from the malicious actor. It is also imperative that HoneyPotR is able to gracefully close all running connections so that no data is lost before it is written. In the case that the application is compromised we need to shutdown gracefully so that we can end the threat without losing data collected up to that point.

Plug-ins

Each plug-in should be developed to match services running in production as closely as possible. HoneyPotR comes with Telnet and HTTP servers by default. Both include interfaces in which the attacker can interact with that will record all of the attackers commands. By including these plug-ins we give our users an example of how to develop their own services, as well as the ability to collect data while developing their own.

The default plug-ins are also able to handle scans from popular network scanners such as Nmap. It is common for attacks to start with scans so that the hacker will have an understanding of what services are running on the network. The ability to capture and record these scans will give us an idea of what kind of information the attacker is looking to gain before going active with his attack.

Data Management

HoneyPotR uses Sqlite3 which comes embedded in Python3. This made creating a persistant database, that we store locally, easy and lightweight. All of the data management logic resides in the database directory which contains modules and classes used by the framework to create, modify and insert data into the database. The resulting database is then used by the report server to query data for reporting.

There are two types of tables defined within the system. User defined tables are those tables that users may create columns in for the storing of specific plug-in data. Non-User defined tables are created by various SQL scripts and should not to be altered by plug-in authors. These Non-User defined tables are the ipInfo, p0f and sessions tables.

The DataManager is instantiated by the Framework and has access to the global configuration defined in common/config/globals.cfg. The DataManager makes use of this information to both create non-existent tables and to update table definitions in the database by checking the difference between the current schema and the database configuration defined by the plug-in author. The DataManager also has access to a DataValidator class which checks that the data to be inserted is well formed. The DataQueue class contains a Python queue (FIFO) along with some other methods for standard queue operations. The DataManager gets items from this queue and inserts them to the database. This method of writing data allows us the ability to write data from the plug-ins without the need to keep those connections open waiting for the database to unlock from other write operations. To ensure that we do not lose in information in the queue the Framework ensures the queue is empty before shutting down the data management system.

DataManager Class

The DataManager Class is called by the framework within its start method. The DataManager creates an instance of Database so that it can call the create_default_database method and create the initial database or modify the existing database with updates to the user defined configurations between this run and the previous run of the program. The DataManager then creates the DataQueue object that will be used to store database transactions until the thread can process them by acquiring a condition variable.

When the Framework calls the DataManager's start method the DataManager's run method will be invoked. This run method will check to see if the queue is empty and then the thread will wait, this will release the lock on the condition variable and block until it is called by notify on that same condition variable.

When the Framework calls DataManager insert_data method this method will acquire the condition variable and call the Queue's insert_into_data_queue method and will then notify the condition variable telling the run method to continue checking the queue for data to insert. Once the queue is empty the run method will again wait for another notification from the insert_data method. The DataManager also contains a shutdown method that will shut down the thread.

Database Class and Table Init Module

The Database class will create the directory for storing the database itself as well as creating the database file. It will also execute any scripts for creating Non-User defined tables within that database. Finally it will execute the update_schema method that will in turn create and/or update any user defined definitions based on plug-in configuration. This class interacts with the Table_Init module to:

Currently when a user-defined plug-in table is redefined in the plug-in configuration that table will be effectively truncated in order to add or remove the columns defined. In the future it would be good to implement the redefinition in a way that keeps what data is possible to persist between redefinitions.

DataValidator Class

The DataValidator class is responsible for checking upon insert into the DataQueue that the data that is being inserted is well formed for the database. The DataValidator has access to the schema currently stored in the database. By pulling this schema out of the database it can be used to assist with data validation. Currently the DataValidator has check methods for the following:

If all checks in the validator do not pass then the value will not be inserted into the DataQueue. Any check errors will be logged to the recce7.log file explaining why that data was not allowed into the data queue. These errors can be useful for plug-in authors when designing a table for their plug-ins.

DataQueue Class

The DataQueue class simply creates a python Queue and instantiates the DataValidator class. The method insert_into_data_queue is called by the DataManager to insert data into the python Queue. This insert_into_data_queue method runs all the checks provided in the data validator prior to inserting the dictionary into the Queue. The DataQueue also provides helper methods for get_next_item and check_emtpy.

Table Insert Module

The Table_Insert module provides methods to insert the data from the DataQueue into the database itself. Because the name-value pairs in the insert dictionary represent the column names and their values it is necessary to prepare the data for insertion. The following methods are available in the Table Insert Module:

Util Module

The util module contains some common methods used throughout the database code and the configuration for the default columns that are standard for all plug-in tables. These default columns are pre-pended to all plug-in tables regardless of the extended definitions that the plug-in authors provide therefore are not defined in the plug-ins' configurations file. The default columns:

Data Reporting

As mentioned in the features section we provide a REST API to allow users to integrate data from our reporting server to their own user interface systems.

Setup RESTful Report Server

[Database]
path = honeyDB/honeyDB.sqlite
datetime.name = eventDateTime
peerAddress.name = peerAddress
localAddress.name = localAddress
[ReportServer]
reportserver.logName = reportserver.log
reportserver.logLevel = DEBUG
reportserver.host = 192.168.1.16
reportserver.port = 8080

Using REST API

All responses will return content-type: application/json All examples below assume report server configured to respond on localhost and port 8080

Analytics Endpoint

GET http://{REPORT SERVER HOST}/v1/analytics

Response HeadersContent-Type --application/json
Status Codes200 OK - Request completed successfully
Requesthttp://localhost:8080/v1/analytics
Response { "links": [ "rel: ports, href: http://127.0.0.1:8080/v1/analytics/ports", "rel: ipaddresses, href:http://127.0.0.1:8080/v1/analytics/ipaddresses" ] }

Ports Endpoint

GET http://{REPORT SERVER HOST}/v1/analytics/ports

Returns a list of enabled ports and their data. If the port was marked disabled in the configuration, it will not show up in this response. Default of the data in the response is timespan of 1 day, unless otherwise specified.

Response HeadersContent-Type --application/json
Status Codes200 OK - Request completed successfully
Requesthttp://localhost:8080/v1/analytics/ports
Optional Parameters
?minutes=xx

?hours=xx

?days=xx

?weeks=xx

Example: http://localhost:8080/v1/analytics/ports/8023?days=100
Response [ { "timespan": "days=1", "rel_link": "http://127.0.0.1:8080/v1/analytics/ports/8082?days=1", "total_attacks": "0", "port": "8082", "unique_ipaddresses": "0" }, { "timespan": "days=1", "rel_link": "http://127.0.0.1:8080/v1/analytics/ports/8083?days=1", "total_attacks": "0", "port": "8083", "unique_ipaddresses": "0" }, { "timespan": "days=1", "rel_link": "http://127.0.0.1:8080/v1/analytics/ports/8023?days=1", "total_attacks": "0", "port": "8023", "unique_ipaddresses": "0" } ]

Ports Endpoint : Specify Port Number

GET http://{REPORT SERVER HOST}/v1/analytics/ports/{PORT NUMBER}

Returns details of attacks for given port. Default of the data in the response is timespan of 1 day, unless otherwise specified.

Response HeadersContent-Type --application/json
Status Codes200 OK - Request completed successfully
Requesthttp://localhost:8080/v1/analytics/ports/8023
Optional Parameters
?minutes=xx

?hours=xx

?days=xx

?weeks=xx

Example: http://localhost:8080/v1/analytics/ports/8023?days=100
Response { "timespan": "days=100", "port": "8082", "items": [ { "duration": "22:02:33.477883", "end_time": "2016-04-24T14:14:18.991806", "session_items": [ { "peerAddress": "127.0.0.1", "localAddress": "127.0.0.1", "path": "/login", "ID": 1, "headers": "Host: localhost:8082\nConnection: keep-alive\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\nUpgrade-Insecure-Requests: 1\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36\nDNT: 1\nAccept-Encoding: gzip, deflate, sdch\nAccept-Language: en-US,en;q=0.8\nCookie: SESSION=d60718da-e011-4d80-88a2-c745b7af8fd2\n\n", "eventDateTime": "2016-04-23T16:11:45.513923", "command": "GET", "session": "d60718da-e011-4d80-88a2-c745b7af8fd2", "body": "" }, ] } ] }

IP Addresses Endpoint

GET http://{REPORT SERVER HOST}/v1/ipaddresses

Future release

IP Addresses Endpoint : Specify IP Address

GET http://{REPORT SERVER HOST}/v1/ipaddresses/{VALID IP ADDRESS}

Returns details of attacks for each enabled port that the IP address attempted to access. Default of the data in the response is timespan of 1 day, unless otherwise specified.

Response HeadersContent-Type --application/json
Status Codes200 OK - Request completed successfully
Requesthttp://localhost:8080/v1/ipaddress/127.0.0.1
Optional Parameters
?minutes=xx

?hours=xx

?days=xx

?weeks=xx

Example: http://localhost:8080/v1/analytics/ipaddresses/127.0.0.1?days=100
Response

Example of no attacks

{ "timespan": "days=1", "ipaddress": "127.0.0.1", "ports": [ { "8082": [] }, { "8083": [] }, { "8023": [] } ] }
Response

Example with attacks

{ "timespan": "days=10", "ipaddress": "127.0.0.1", "ports": [ { "8082": [ { "duration": "22:02:33.477883", "end_time": "2016-04-24T14:14:18.991806", "session_items": [ { "peerAddress": "127.0.0.1", "localAddress": "127.0.0.1", "path": "/login", "ID": 1, "headers": "Host: localhost:8082\nConnection: keep-alive\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\nUpgrade-Insecure-Requests: 1\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36\nDNT: 1\nAccept-Encoding: gzip, deflate, sdch\nAccept-Language: en-US,en;q=0.8\nCookie: SESSION=d60718da-e011-4d80-88a2-c745b7af8fd2\n\n", "eventDateTime": "2016-04-23T16:11:45.513923", "command": "GET", "session": "d60718da-e011-4d80-88a2-c745b7af8fd2", "body": "" }, ], "peer_address": "127.0.0.1", "session": "d60718da-e011-4d80-88a2-c745b7af8fd2", "local_address": "127.0.0.1", "begin_time": "2016-04-23T16:11:45.513923" }, }, { "8083": [ { "duration": "22:03:37.946130", "end_time": "2016-04-24T14:15:11.759311", "session_items": [ { "peerAddress": "127.0.0.1", "localAddress": "127.0.0.1", "path": "/", "ID": 7, "headers": "Host: localhost:8083\nConnection: keep-alive\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\nUpgrade-Insecure-Requests: 1\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36\nDNT: 1\nAccept-Encoding: gzip, deflate, sdch\nAccept-Language: en-US,en;q=0.8\nCookie: SESSION=d60718da-e011-4d80-88a2-c745b7af8fd2\n\n", "eventDateTime": "2016-04-23T16:11:33.813181", "command": "GET", "session": "d60718da-e011-4d80-88a2-c745b7af8fd2", "body": "" }, ], "peer_address": "127.0.0.1", "session": "d60718da-e011-4d80-88a2-c745b7af8fd2", "local_address": "127.0.0.1", "begin_time": "2016-04-23T16:11:33.813181" }, }, { "8023": [ { "duration": "0:00:06.704076", "end_time": "2016-04-23T16:12:27.201369", "session_items": [ { "peerAddress": "127.0.0.1", "localAddress": "127.0.0.1", "user_input": "hellow", "ID": 1, "eventDateTime": "2016-04-23T16:12:20.497293", "session": "f6cae8f0-5c61-4319-b357-1dcc20ab6fe4", "input_type": "username" }, ], "peer_address": "127.0.0.1", "session": "6c83a4ae-f32c-442d-a545-f0accd735d99", "local_address": "127.0.0.1", "begin_time": "2016-04-23T16:12:42.440673" } ] } ] }

WorldMap Endpoint

GET http://{REPORT SERVER HOST}/v1/worldmap

Returns an image map of the location of the ipaddresses. Can scope data by minutes, hours, days, weeks. If no data scope is given, it defaults to days=1.

Response HeadersContent-Type --application/json
Status Codes200 OK - Request completed successfully
Requesthttp://{REPORT SERVER HOST}/v1/worldmap?days=10
Optional Parameters
?days=xx

Example: http://localhost:8080/v1/analytics/ports/8023?days=100
Response

A global map with plots of attacks will displayed in browser when this endpoint is called.

Setup Grafana Reporting Graphs and Boards

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

How to Install and Run HoneyPotR

These instructions will walk you through installing and running HoneyPotR/Recce7 on a Raspberry Pi Zero. It is strongly recommended that your Raspberry Pi have at least 4GB of available storage, particularly if you would like to generate attacker worldmaps (see "Installing Basemap" below).

These instructions are intended for the Raspbian OS as this is currently the only modern Linux OS available for Raspberry Pi hardware. Similar instructions will work on most Debian-derived Linux distributions on x86, ARM or other platforms.

These directions assume that you are starting with a fresh installation of Raspbian and that you are working from a shell in the default user's home directory (/home/pi/).

Before beginning, ensure your Raspbian package repositories are up-to-date:

$ sudo apt-get update

Download and install the latest release .deb package from the project repository:

$ wget https://github.com/RECCE7/recce7/releases/download/1.0/python3-recce7_1.0-1_all.deb
$ sudo dpkg -i python3-recce7_1.0-1_all.deb

While this will install the package, it will not automatically install the package's dependencies. To do this, run

$ sudo apt-get install -f

Note: During installation, the following error has been known to prevent the installation from completing:

(Py3) Installing p0f API module...
Downloading/unpacking p0f
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not find any downloads that satisfy the requirement p0f
Cleaning up...
No distributions at all found for p0f
Storing debug log for failure in /home/ubuntu/.pip/pip.log
dpkg: error processing package python3-recce7 (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up blt (2.4z-7ubuntu2) ...
Setting up python3-tk (3.4.3-1~14.04.2) ...
Processing triggers for ca-certificates (20160104ubuntu0.14.04.1) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Processing triggers for libc-bin (2.19-0ubuntu6) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
 python3-recce7
E: Sub-process /usr/bin/dpkg returned an error code (1)

This appears to be due to an upstream bug; to remedy the problem, rerun the following command and the installation should complete successfully:

$ sudo apt-get install -f

After installation you should edit your configuration files. An example configuration is installed by default but you'll most likely want to edit the default settings to suit your needs. There are two configuration files located in the /etc/recce7/configs directory after installation: global.cfg and plugins.cfg.

In the global.cfg file, you should set the following line to the main address of the network interface you would like the honeypot to listen on (the address 192.168.0.2 is shown in the example):

[Framework]
  .
  .
  .
listeningAddress = 192.168.0.2

If you leave this line blank (after the '=') the honeypot will listen on all interfaces by default.

Similarly, you can configure the reporting server to listen for connections on the desired address and port by editing these lines:

[ReportServer]
reportserver.host = 127.0.0.1
reportserver.port = 8080

In the plugins.cfg file you can configure the plug-ins you'd like to run. Each section in this file describes one instance of a plug-in. An example section looks like this:

[MyHTTPInstance]
port = 80
table = http
module = http
moduleClass = HTTPPlugin
enabled = Yes
rawSocket = no
tableColumns = [[1,'command','TEXT'], [2,'path','TEXT'], [3, 'headers', 'TEXT'], [4, 'body', 'TEXT']]

This creates a new webserver plug-in instance. The section header [HTTP] defines the instance's name; this can be anything you'd like. The port line dictates which port the plug-in will listen on. The module and moduleClass lines can specify the type of plug-in you wish to run. To create a webserver plug-in instance, set the lines as follows:

module = http
moduleClass = HTTPPlugin

To create a telnet plug-in instance, set the lines as follows:

module = telnet
moduleClass = TelnetPlugin

The table line will dictate the name of the table that this plug-in instance will use.

A particular plug-in type (e.g. http or telnet) may have many running instances so long as they have unique names, unique table names and unique ports; this may be useful for running an HTTP plug-in on, for example, port 80, port 443, port 8080 or any other port you wish to use.

The other lines should not need to be modified unless you plan to extend the functionality of the plug-ins (see below).

Note: Plugins that listen on ports below 1024 will need to be enabled with authbind. When HoneyPotR/Recce7 is started, it drops its permissions to the user nobody (and group nogroup) but retains the ability to listen on low-numbered ports using the authbind command. Consult the authbind manual pages and follow the process to enable the desired ports. Port numbers greater than 1024 do not need to be enabled through authbind. The ports used by the default plug-in configuration that ships with HoneyPotR/Recce7 will be enabled with authbind during installation.

Installing Basemap

If you wish to use the /worldmap reporting server endpoint (see above) you will need to install the Python 3 Basemap module, which is a part of the Matplotlib Python 3 kit. This is an involved process on a Raspberry Pi Zero as the Basemap plug-in is quite hefty and requires a large quantity of RAM.

You should ensure that you have at least 4GB of storage available on your Raspberry Pi Zero before beginning this process.

It is recommended that you start by creating a swapfile that's at least 1500 MB in size. Consult the dphys-swapfile manpage on the Raspberry Pi for instructions on increasing the default swap file size on your system.

Next, download, build and install Basemap (Note: this process can take up to 2 hours to complete):

$ cd ~
$ wget https://github.com/matplotlib/basemap/archive/v1.0.7rel.tar.gz
$ tar zxvf v1.0.7rel.tar.gz
$ cd basemap-1.0.7rel/geos-3.3.3
$ export GEOS_DIR=/usr/local
$ ./configure --prefix=$GEOS_DIR
$ make
$ sudo make install
$ cd ../
$ sudo python3 setup.py install
$ sudo ldconfig

Instructions for Developing plug-ins

Config File Entries

The following fields must be filled in for each plug-in configurations.

The configuration entry nameConfig file entry must start with the plug-in name (does not need to match class name)[(string)]
The port numberThe value must lie in the range of 0 to 65535port = (number)
The database table nameThis name must adhere to the SQL syntax for table namestable = (string)
The module nameThis will be the name of the python module containing the plug-in classmodule = (string)
The class nameThis is the name of the plug-in class, this class must exists within the module specified by the module namemoduleClass = (string)
Specifies if the plug-in is enabledIf this field is set to no the framework will not spawn the listener for the port and no database table will be createdenabled = (Yes or No)
Specifies if the socket is receiving raw packetsThis functionality is currently not implemented in the softwarerawSocket = (Yes or No)
The column indexes, headings, and data types to be interpreted as a Python arrayThis row provides the parameters for building the dictionary to write plug-in data to the databasetableColumns = ([[column index, column heading, data type], ...]])

Example Configuration

[HTTP]
port = 8083
table = HTTP
module = http
moduleClass = HTTPPlugin
enabled = Yes
rawSocket = no
tableColumns = [[1,'command','TEXT'], [2,'path','TEXT'], [3, 'headers', 'TEXT'], [4, 'body', 'TEXT']]

API

Plug-ins will need to interact with our Baseplug-in so that the data can be inserted properly into the database. There are two methods that plug-ins must implement.

do_track(self):

Do track will hold all of the logic in the plug-in that tracks what a user is doing and stores that information in local data structures that follow the data formatting that is described above. This will differ for each service that is made into a plug-in. For example we would not expect to see the same kind of attacks hitting a MySQL server as we would for an Apache Web Server.

get_session(self):

This method allows HoneyPotR to group attacks into sessions based on when an attacker connects until they disconnect. The reason that this is not handled in the Baseplug-in is that some services natively allow session tracking. An HTTP server may use cookies or HTTP Headers for tracking sessions while for Telnet we needed to implement a UUID for this.

close_descriptors(self):

This method will handle closing any file descriptors the plug-in has opened. This method should be called at the end of the plug-in lifecycle. This includes the natural end of the plug-in or premature end by exceptions.

do_save(self):

This method handles calling the framework to save data to the database. A dictionary is constructed using the columns defined in the configuration. The dictionary values are populated by matching the column names with attributes of the same name in the plug-in. This populated dictionary is sent to the framework to be written to the database.

Authors and Contributors

Matt @howard-roark, Dina @dsteeve, Charlie @belmontrevenge, Ben @benphillips989, Jesse @jnels124, Randy @rsoren514, Zach @zkuhns

Support or Contact

Have questions or need support? Please contact any of the above contributors at their GitHub pages.

Our GitHub page.