Every now and then we all run into situation where software is not behaving as intended leaving us with the most battle tested troubleshooting approach ever discovered. Rebooting the whole system, crossing fingers and hoping that whatever went wrong inside the blackbox starts up again correctly and automagically recovers from its failed state. Does it work? Yes – sometimes just rebooting a system resolves issues at hand but most of the times a more graceful approach might be the better choice before resorting to a complete system halt. In this post we will look at managing Firepower processes with pmtool, a convinient command-line utility for interacting with Firepowers process manager.
Meet Firepower Process Manager#
Process Manager (pm) is responsible for managing and monitoring all Firepower related processes on your system. It takes care of starting up all components on startup and restart failed processes during runtime. To interact with Process Manager the CLI utiltiy pmtool is available. Pmtool can be used to disable, enable, (re-)start services or just check the status of the various services PM is managing.
Which processes can I manage with pmtool?#
Before diving into where we find pmtool and the specific syntax of managing the different firepower components let’s take a look at some important processes on both FMC and FTD
Firepower Management Center#
| Process | Description |
|---|---|
| adi | User Identity integration. ADI polls Active Directory for user:group mappings and integrates into ISE using pxGRID to download and parse session information (which are then handed over to SFDataCorrelator). Additionally adi is responsible for sending remediation requests to ISE |
| sftunnel | Encrypted communication tunnel between FMC and FTD. Used to push configuration and exchange state information between FMC and FTD |
| SFDataCorrelator | Processes various (event) data streams and correlates received data with other datasets. SFDC stitches together sensor events with vulnerability data available on FMC to enrich events but also processes things like user identity mappings and a lot of data correlation related tasks. |
| diskmanager | Cleanup of old files like url databases, user identity dumps and various cache files |
| sla | Smart Licensing agent |
| CloudAgent | Reaches out to AMP and Talos Cloud for ClamAV, URL Filtering and Security Intelligence updates |
| ui_archiver | Monitors the mysql database that the (depcrecated) user agent inserts user:ip mappings into and creates ui.bin files that are then processed by SFDataCorrelator |
| SSEConnector | Agent that connects to Security Services Exchange for registering devices to SSE and shipping security events to the Cloud |
| Prometheus | Backend for the new Health monitoring dashboard introduced with Firepower 6.7.0 |
| tid | Threat Intelligence Director backend |
| mysqld | The old mysql database backend (which is being replaced by sybase) |
| vmsDbEngine | The new database backend which slowly but steadily replaces the old mysql database backend |
| syncd | FMC High Availability synchronisation |
| Tomcat | Java Webserver that hosts FMC REST API and is required by the UI to work |
Firepower Threat Defense#
| Process | Description |
|---|---|
| adi | User identity integration. Directly connects to directory servers defined in realm configuration for authenticating captive portal users |
| sftunnel | Encrypted communication tunnel between FMC and FTD. Used to push configuration and exchange state information between FMC and FTD |
| EventHandler | Processes UnifiedEvents (Connection/AMP/IPS Events) generated by snort |
| SFDataCorrelator | Processes various (event) data streams and correlates received data with other datasets. SFDC stitches together sensor events with vulnerability data available on FMC to enrich events but also processes things like user identity mappings and a lot of data correlation related tasks. |
| DetectionEngine | Snort Detection Engine (NGFW portion of FTD) handling TLS Decryption, AVC, IPS, AMP, URL Filtering, Security Intelligence, etc. As for Firepower 6.7.0 (managed by FMC) Snort2 is being used which will be replaced with Snort3 soon |
| ngfwManager | Handles configuration deployment to lina and manages High Availability on FTD devices (responsible for synchronising data between peers) |
| lina | Stateful inspection engine. Lina stands for Linux over ASA and represents the good ol’ ASA code that still lives on in Firepower Threat Defense. Do not touch this process or FTD will reboot! |
| diskmanager | Cleanup of old files like url databases, user identity dumps and various cache files |
| mysqld | Mysql database backend |
| tomcat | Java Webserver that hosts Firepower Device Manager |
| bltd | Process that handles traffic destined to captive portal (exposes a UNIX socket which is used by snort to redirect traffic to bltd which in turn translates the source ip to 169.254.x.x and forwards the traffic to idhttpsd) |
| idhttpsd | Webserver hosting captive portal |
Now that we are aware of the different important processes running behind the scenes let’s take a look where pmtool is located and how we can use it to troubleshoot communication problems between FMC and FTD.
Where do I find pmtool?#
On FTD pmtool is available from SFCLI (>) while on FMC you will need a bash root shell (root@ftd01:~#)
to access pmtool.
Before executing pmtool on FMC we will need to access the bash shell and change users to root. This can be done by executing the expert command from SFCLI followed by sudo -i
| |
What commands does pmtool provide?#
While there a lot of different commands that might be handy for very specific troubleshooting scenario we will focusing on the following:
| |
| |
Displaying process status#
To get a quick overview of the running processes we can execute the status command and grep the important information
| |
| |
When looking over the list of processes you might see some of them in the Waiting state. Processes in waiting state might not be required in your current setup (e.g. tomcat might not be running if your device is managed by FMC, since it is only required by FDM)
Ok – so we got a list of all the processes, let’s dig a bit deeper and check out the sftunnel process
root@ftd01:/home/admin# pmtool status | grep sftunnel
| |
Looks like sftunnel is up and running. The process was initiated using the command /ngfw/usr/local/sf/bin/sftunnel -d -f /etc/sf/sftunnel.conf and has a process ID (PID) of 12858
To verify the information obtained through pmtool we can use ps (process status) which ships with most Linux distributions
| |
Great – now that we know how to verify process status using pmtool let’s go ahead and try restarting sftunnel
Tip
Always restart sftunnel from the sensor and not from FMC if you are having problem with a single device, otherwise all sftunnel connections to your sensor will need to be re-established
Restarting processes#
| |
It will take a few seconds to restart a process, just wait a bit and your shell should greet you again with a prompt.
To verify that the process has been restarted successfully we will check the status once more and verify that the PID has indeed changed
| |
If you are interested in to what is happening behind the scene go ahead and tail the messages logfile for any sftunnel related entries:
sftunnel log output
| |
| |
Restarting Processes by Type#
Another way to restart processes is by specifying a type. The only usecase I have come across for the RestartByType subcommand is for restarting the snort engine. Restarting snort can be a neccessary procedure for activating certain changes (e.g. after a manual cleanup) or could be a troubleshooting step – I sometimes resort to this step if unexplainable behavior occurs within the layer-7 engine.
pmtool restartByType DetectionEngine
Warning
Restarting the DetectionEngine may lead to a brief (0.1-3.0sec in my experience) dataplane outage so proceed with caution. In case you have a high availability pair restart snort on the passive unit and execute a failover for a more graceful non-disruptive option
Disabling & Enabling Processes#
Under normal circumstances there should be no need to disable any system services. The only reason to do so is for debugging purposes. Some processes like adi, CloudAgent, sla or SSEConnector provide additional debug flags that can be utilized to gain more insight of what is happening behind the scenes.
For a brief demonstration we will disable the adi process on FMC and run it in debug mode
Step 01: Disable adi process
root@ftd01:/home/admin# pmtool disablebyid adi
Step 02: Verify that adi process is user disabled and not running
root@ftd01:/home/admin# pmtool status | grep adi
adi (normal) - User Disabled
Command: /ngfw/usr/local/sf/bin/adi
PID File: /ngfw/var/sf/run/adi.pid
Command: /ngfw/usr/local/sf/bin/beakerd --module-path=/ngfw/usr/local/sf/lib64 --log-file=/ngfw/var/log/beakerd.log --log-level=info --cfgdir=/ngfw/etc/sf/beakerd --pidfile=/ngfw/var/sf/run/beakerd.pid --enable-module=updater --enable-module=nikita --datadir=/ngfw/etc/sf/beakerd --listen=/ngfw/var/tmp/beaker_client.sock --listen=/ngfw/var/tmp/beaker_async_client.sock
Step 04: Run adi in debug mode and send output to screen and /var/tmp/adi.log
adi --debug 2>&1 | tee /var/tmp/adi.log
Debugging log output
May 02 11:55:44 ftd01 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:318:ProcessConfigGlobalSettings(): Parsing global settings May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:350:ProcessConfigGlobalSettings(): global_settings.max_active_auth_retries = 3 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:394:ProcessConfigGlobalSettings(): global_settings.linaProtocol = ipc May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:337:ProcessConfigGlobalSettings(): global_settings.cipher_key_file_path = /ngfw/etc/sf/ADI_conf_Encryption_Key.bin May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:363:ProcessConfigGlobalSettings(): global_settings.sendS2SVPNEvents = true May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:368:ProcessConfigGlobalSettings(): global_settings.sendRAVPNEvents = tunnel May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:501:ProcessConfigFPReplication(): fpreplication.mode = follower May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:263:ProcessKeyValuePair(): tenant.uid = 1 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:630:ProcessConfigTenant(): New tenant May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:666:ProcessConfigTenant(): tenant.realm.uid = 4 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:691:ProcessConfigRealm(): New realm May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:711:ProcessConfigRealm(): tenant.realm.uuid = 48916640-4e4a-11ea-bba2-9b7b2e356a46 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:723:ProcessConfigRealm(): tenant.realm.name = example.com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:751:ProcessConfigRealm(): tenant.realm.realm_type = AD May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:763:ProcessConfigRealm(): tenant.realm.directory_username = firepower@example.com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:775:ProcessConfigRealm(): tenant.realm.directory_password = d3p3nd3nc!h3!!0Nep59RwdvuJ6IA== May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:787:ProcessConfigRealm(): tenant.realm.ldap_user_naming_attribute = sAMAccountName May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:799:ProcessConfigRealm(): tenant.realm.ldap_user_search_base = ou=com,ou=example,dc=example,dc=com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:811:ProcessConfigRealm(): tenant.realm.ldap_group_attribute = member May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:883:ProcessConfigRealm(): tenant.realm.ldap_group_search_base = ou=groups,ou=com,ou=example,dc=example,dc=com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:859:ProcessConfigRealm(): tenant.realm.iv_ldap_value = d3p3nd3nc!h3!!0Nep59RwdvuJ6IA== May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:871:ProcessConfigRealm(): tenant.realm.iv_ad_value = d3p3nd3nc!h3!!0Nep59RwdvuJ6IA== May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:823:ProcessConfigRealm(): tenant.realm.active_directory_domain = example.com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:847:ProcessConfigRealm(): tenant.realm.active_directory_password = d3p3nd3nc!h3!!0Nep59RwdvuJ6IA== May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:998:ProcessConfigDirectory(): New directory May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1018:ProcessConfigDirectory(): tenant.realm.directory.host = dc01.example.com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1031:ProcessConfigDirectory(): tenant.realm.directory.port = 389 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1053:ProcessConfigDirectory(): tenant.realm.directory.encryption = none May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1072:ProcessConfigDirectory(): tenant.realm.directory.timeout_seconds = 60 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:998:ProcessConfigDirectory(): New directory May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1018:ProcessConfigDirectory(): tenant.realm.directory.host = dc02.example.com May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1031:ProcessConfigDirectory(): tenant.realm.directory.port = 389 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1053:ProcessConfigDirectory(): tenant.realm.directory.encryption = none May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:1072:ProcessConfigDirectory(): tenant.realm.directory.timeout_seconds = 60 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:config [DEBUG] config.cpp:212:ProcessConfiguration(): Configuration read May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ADIStateMachine [INFO] ADIStateMachine.cpp:54:moveTo(): ADI state moved to ADI_STOPPED May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f79367fc700 object: 0x5558b87a3930 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f79357fa700 object: 0x5558b877fa10 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f7934ff9700 object: 0x5558b8807d40 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f7927fff700 object: 0x5558b87a4300 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f79277fe700 object: 0x5558b8807cb0 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f7926ffd700 object: 0x5558b87a5380 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f79267fc700 object: 0x5558b87a7970 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f7925ffb700 object: 0x5558b87a71e0 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f79257fa700 object: 0x5558b876bae0 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [INFO] ZMQWorker.cpp:76:start(): Added ZMQ worker thread id:7f7924ff9700 object: 0x5558b876be80 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:adi.ZMQWorker [DEBUG] ZMQWorker.cpp:96:start(): Widened permissions for /tmp/adireq to 777. May 02 11:55:44 SF-IMS[10482]: [10482] ADI:ADI [DEBUG] adi.cpp:203:setVPNProcessing(): Starting S2S VPN event handler due to configuration change May 02 11:55:44 SF-IMS[10482]: [10482] ADI:ADI [DEBUG] adi.cpp:210:setVPNProcessing(): Starting S2S VPN event consumer due to configuration change May 02 11:55:44 SF-IMS[10482]: [10482] ADI:ADI [DEBUG] adi.cpp:239:setVPNProcessing(): Starting RA VPN event handler due to configuration change May 02 11:55:44 SF-IMS[10482]: [10482] ADI:DataMessaging_EventQueue [DEBUG] EventQueue_impl.cpp:1990:ParseEventQueues(): Event Queues not configured in /etc/sf/SFDataCorrelator.conf, using internal defaults! May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:598:getSFTunnelConfiguration(): Processing file /ngfw/etc/sf/sftunnel.conf May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UECHelloDR lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:511:ProcessTunnelKeyValuePair(): Setting VPN event device UUID to 457d932a-3dfb-11ea-9b36-8a42de410c5c May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:561:ProcessTunnelKeyValuePair(): Adding peer 773010ea-3df2-11ea-b8af-7f5efeab3d9e May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UECReplyDR lookup returned 7 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UECPriorityStatEvent lookup returned 10 May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:293:processEvents(): IP: 457d932a-3dfb-11ea-9b36-8a42de410c5c, manager List ‘773010ea-3df2-11ea-b8af-7f5efeab3d9e’ May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:300:processEvents(): Current event count: 0 May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:374:processEvents(): Did FMCRefresh May 02 11:55:44 SF-IMS[10482]: [10482] ADI:DataMessaging_EventQueue [DEBUG] EventQueue_impl.cpp:2318:ProcessLLConfigFile(): tid config: enable_obs=1, send_obs_to_sfdc=0 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:DataMessaging_EventQueue [DEBUG] EventQueue_impl.cpp:2318:ProcessLLConfigFile(): tid config: enable_obs=1, send_obs_to_sfdc=1 May 02 11:55:44 SF-IMS[10482]: [10500] ADI:VPN Event [DEBUG] vpnEventHandler.cpp:388:processEvents(): Sending events with function: Remote::SF::VPN::setFTDStatus({targets=>[‘773010ea-3df2-11ea-b8af-7f5efeab3d9e’]}, ‘457d932a-3dfb-11ea-9b36-8a42de410c5c’, [], 3) May 02 11:55:44 SF-IMS[10482]: [10500] ADI:Forker [DEBUG] forker.cpp:25:run(): Forker::run() calling cmd ‘/ngfw/usr/local/sf/bin/send_vpn_events.sh’ with args:[/ngfw/usr/local/sf/bin/send_vpn_events.sh,perl,Remote::SF::VPN::setFTDStatus({targets=>[‘773010ea-3df2-11ea-b8af-7f5efeab3d9e’]}, ‘457d932a-3dfb-11ea-9b36-8a42de410c5c’, [], 3)] May 02 11:55:44 SF-IMS[10482]: [10482] ADI:DataMessaging_EventQueue [INFO] EventQueue_impl.cpp:2626:DataMessaging_InitEventQueues(): Event QoS started successfully! May 02 11:55:44 SF-IMS[10482]: [10482] ADI:DataMessaging_EventQueue [DEBUG] EventQueue_impl.cpp:2627:DataMessaging_InitEventQueues(): Started Event thread Success May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 7 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Deprecated_IPMask lookup returned 24 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Deprecated_UserService lookup returned 26 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 7 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 7 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Deprecated_VulnAck lookup returned 22 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute AttributeListItem lookup returned 39 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute AttributeAddress lookup returned 146 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute MacSpec lookup returned 63 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UserClientApp lookup returned 138 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute MacSpec lookup returned 63 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UserService lookup returned 76 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute MacSpec lookup returned 63 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnAck lookup returned 124 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UserProtocol lookup returned 57 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostMAC lookup returned 95 May 2 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute ServiceInfo lookup returned 117 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Payload lookup returned 123 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullServiceInfo lookup returned 106 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullServiceInfo lookup returned 106 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullServiceInfo lookup returned 106 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullServiceInfo lookup returned 106 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Payload lookup returned 123 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullServiceSubtype lookup returned 51 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 7 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute StringInfo lookup returned 35 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Payload lookup returned 123 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute ServiceSubtype lookup returned 1 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Payload lookup returned 123 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute DeviceInfo lookup returned 131 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FixList lookup returned 67 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostService lookup returned 103 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostService lookup returned 103 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute PndProtocol lookup returned 4 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute PndProtocol lookup returned 4 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostMAC lookup returned 95 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostClientApp lookup returned 122 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute ScanVuln lookup returned 109 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute genericScanResults lookup returned 108 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute UserProduct lookup returned 134 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IPRangeSpec lookup returned 141 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostIP lookup returned 143 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute OSFP lookup returned 130 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullHostService lookup returned 104 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullHostService lookup returned 104 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute PndProtocol lookup returned 4 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute PndProtocol lookup returned 4 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute HostMAC lookup returned 95 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute FullHostClientApp lookup returned 112 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VulnList lookup returned 85 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute AttributeValue lookup returned 48 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute IOCState lookup returned 150 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute VPNSession lookup returned 166 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 171 for FlowStats May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 139 for HostTracker May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 103 for HostService May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 145 for RNADHCPInfo May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 122 for HostClientApp May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 130 for OSFP May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 53 for OS May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 95 for HostMAC May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 14 for VLAN May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 35 for StringInfo May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 37 for ServiceBanner May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 167 for UserLoginInfo May 02 11:55:44 SF-IMS[10482]: [10482] ADI:RNASerializeLib [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/sfrnacommon/serialize/serialize.c:3785:InitRNASerialize(): Got 96 for SecondaryHostUpdate May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute Unified2UserMapUpdate lookup returned 11 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute MACADDR lookup returned 17 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute String lookup returned 0 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:SFSerialize [DEBUG] sfserialize.c:847:SFSerialize_Init(): Attribute INT32 lookup returned 6 May 02 11:55:44 SF-IMS[10482]: [10482] ADI:ADI [DEBUG] adi.cpp:246:setVPNProcessing(): Starting RA VPN event consumer due to configuration change May 02 11:55:45 SF-IMS[10482]: [10500] ADI:Forker [DEBUG] forker.cpp:48:run(): normal termination? 1, exit status: 0 May 02 11:55:45 SF-IMS[10482]: [10500] ADI:VPN Event [INFO] vpnEventHandler.cpp:410:processEvents(): Sent 1 VPN events to manager May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ADIStateMachine [INFO] ADIStateMachine.cpp:54:moveTo(): ADI state moved to ADI_READY May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:208:DoInit(): ADI Process Processing 1 tenants from configuration. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:213:DoInit(): ADI Process Processing 1 realms from next tenant. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:217:DoInit(): ADI Process constructing next realm. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:134:populateModelRealmFromConfigRealm(): Decrypting active directory Password… May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:142:populateModelRealmFromConfigRealm(): Success! May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:181:populateModelRealmFromConfigRealm(): Decrypting directory Password… May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:189:populateModelRealmFromConfigRealm(): Success! May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:181:populateModelRealmFromConfigRealm(): Decrypting directory Password… May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:189:populateModelRealmFromConfigRealm(): Success! May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [DEBUG] AdRealm.cpp:600:onSendStatus(): onSendStatus: Sending ZMQ message for realm 4 : REALM_UNCONFIGURED May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:31:send(): Sent control message May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:352:processMessage(): Found data to broadcast result on subject: realm_status recv_result: 1 May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:359:processMessage(): Sending broadcast response May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:38:send(): Received control response May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Realm [DEBUG] Realm.cpp:102:validate(): Directory: ldap://dc01.example.com:389, Validating At: 1619956545 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Directory [INFO] Directory.cpp:120:changeStatus(): Directory server ldap://dc01.example.com:389 changed state to up May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Realm [DEBUG] Realm.cpp:102:validate(): Directory: ldap://dc01.example.com:389, Validating At: 1619956545 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Realm [DEBUG] Realm.cpp:102:validate(): Directory: ldap://dc02.example.com:389, Validating At: 1619956545 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Directory [INFO] Directory.cpp:120:changeStatus(): Directory server ldap://dc02.example.com:389 changed state to up May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Realm [DEBUG] Realm.cpp:102:validate(): Directory: ldap://dc02.example.com:389, Validating At: 1619956545 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [INFO] AdRealm.cpp:159:configure(): ad: computer dn=cn=FTD01,cn=Computers,dc=ONG,dc=LAB May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.LdapRealm [WARN] LdapRealm.cpp:610:ldapSearch(): ldapParsePageControl: Control not found May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.LdapRealm [DEBUG] LdapRealm.cpp:647:ldapSearch(): ldap: search returned (1) result pages. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [DEBUG] AdRealm.cpp:214:findNetBIOSName(): ad: Found naming context CN=Configuration,DC=example,DC=com from Realm4(AD)_ldap://dc01.example.com:389 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.LdapRealm [DEBUG] LdapRealm.cpp:647:ldapSearch(): ldap: search returned (1) result pages. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [DEBUG] AdRealm.cpp:224:findNetBIOSName(): Found dns attribute example.com in cn=example,cn=partitions,cn=configuration,dc=example,dc=com, matching against example.com May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.LdapRealm [DEBUG] LdapRealm.cpp:647:ldapSearch(): ldap: search returned (1) result pages. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [DEBUG] AdRealm.cpp:233:findNetBIOSName(): Found netbios name ONG in cn=example,cn=partitions,cn=configuration,dc=example,dc=com May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [DEBUG] AdRealm.cpp:600:onSendStatus(): onSendStatus: Sending ZMQ message for realm 4 : REALM_UNCONFIGURED May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:31:send(): Sent control message May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:352:processMessage(): Found data to broadcast result on subject: realm_status recv_result: 1 May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:359:processMessage(): Sending broadcast response May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:38:send(): Received control response May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.AdRealm [INFO] AdRealm.cpp:171:configure(): ad: successfully configured directory. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RealmContainer [DEBUG] RealmContainer.cpp:236:DoInit(): RealmContainer populated from configuration, now contains 1 realms. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.Adi [DEBUG] Adi.cpp:153:configure(): configure connector. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:IseConnector [INFO] ISEConnector.cpp:216:configure(): Initializing Unified2 serialization library May 02 11:55:45 SF-IMS[10482]: [10482] ADI:IseConnector [DEBUG] ISEConnector.cpp:220:configure(): SXP Channel Factory May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.SXPDataProvider [DEBUG] SXPDataProvider.cpp:132:LoadBindings(): Loading bindings from files. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.SXPDataProvider [DEBUG] SXPDataProvider.cpp:152:LoadBindings(): Loading snapshot /var/sf/user_enforcement/sxp.snapshot.1604419773 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:Unified2Iterator [DEBUG] Unified2Iterator.c:407:Unified2Record_GetFromFile(): Opened /var/sf/user_enforcement/sxp.snapshot.1604419773 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.SXPDataProvider [DEBUG] SXPDataProvider.cpp:193:LoadBindings(): Loading update /var/sf/user_enforcement/sxp.1604419773 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:Unified2Iterator [DEBUG] Unified2Iterator.c:407:Unified2Record_GetFromFile(): Opened /var/sf/user_enforcement/sxp.1604419773 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.SXPEntryAppender [DEBUG] SXPEntryAppender.cpp:66:CreateArchiveFile(): initing archive file for type 1 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.SXPEntryAppender [DEBUG] SXPEntryAppender.cpp:305:UpdateRotatedFilesCounter(): Rotated files counter = 1 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:IseConnector [INFO] ISEConnector.cpp:343:createSXPFPReplicationFollowers(): mode Follower 127.0.0.2 5672 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:FPReplicationCommunicationRabbitFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationCommunicationRabbitFollower.cpp:26:FPReplicationCommunicationRabbitFollower(): using b:sxp:457d932a-3dfb-11ea-9b36-8a42de410c5cas follower queue name May 02 11:55:45 SF-IMS[10482]: [10482] ADI:FPReplicationCommunicationRabbitFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationCommunicationRabbitFollower.cpp:40:FPReplicationCommunicationRabbitFollower(): Creating new AMQP connection for consuming user/ip messages, with max priority: 0 and heartbeat: 30 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:RMQClient [DEBUG] RabbitMQClient.cpp:315:DoConnect(): b:sxp:457d932a-3dfb-11ea-9b36-8a42de410c5c connected to server successfully May 02 11:55:45 SF-IMS[10482]: [10482] ADI:IseConnector [INFO] ISEConnector.cpp:355:createSXPFPReplicationFollowers(): ReLoad SXP Filter May 02 11:55:45 SF-IMS[10482]: [10482] ADI:Tokenizer [DEBUG] /tmp/mesos/slaves/IMS/Pre-commit/90923-2388088/FTD/src/libraries/sfutil/parser/tokenizer.c:97:TokenizerInit(): Unable to open ‘/ngfw/var/sf/user_enforcement/network_filter.sxp.cur.773010ea-3df2-11ea-b8af-7f5efeab3d9e’: No such file or directory May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.NetworkFilter [DEBUG] NetworkFilterUtils.cpp:192:ReLoadFilters(): read current filter May 02 11:55:45 SF-IMS[10482]: [10482] ADI:Tokenizer [DEBUG] /tmp/mesos/slaves/IMS/Pre-commit/90923-2388088/FTD/src/libraries/sfutil/parser/tokenizer.c:97:TokenizerInit(): Unable to open ‘/ngfw/var/sf/user_enforcement/network_filter.cfg’: No such file or directory May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.NetworkFilter [DEBUG] NetworkFilterUtils.cpp:198:ReLoadFilters(): read config filter May 02 11:55:45 SF-IMS[10482]: [10482] ADI:IseConnector [DEBUG] ISEConnector.cpp:386:createSXPFPReplicationFollowers(): set follower filter for sxp May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.FPReplicationFollowerSXP [DEBUG] FPReplicationFollowerSXP.cpp:62:FPReplicationFollowerSXP(): Init FPReplicationFollowerSXP May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.FPReplicationFollowerSXP [INFO] FPReplicationFollowerSXP.cpp:63:FPReplicationFollowerSXP(): initializing filter at seq_no = 0 May 02 11:55:45 SF-IMS[10482]: [10482] ADI:realm.join.retryer [DEBUG] RealmJoinRetryer.cpp:25:RealmJoinRetryer(): Starting realm join retryer worker thread. May 02 11:55:45 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:567:consume_messages(): fp replication follower consume messages-start May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:53:send(): Sent control message May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:366:processMessage(): Found lina control message on subject lina-control May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:165:createLinaChannel(): creating lina status channel May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:168:createLinaChannel(): lina protocol: ipc May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:184:createLinaChannel(): creating lina update channel May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:187:createLinaChannel(): lina protocol: ipc May 02 11:55:45 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [ERROR] ZMQServer.cpp:189:createLinaChannel(): Unable to subscribe to subject vpn-update May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ZMQController [DEBUG] Controller.cpp:60:send(): Received control response May 02 11:55:45 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:576:consume_messages(): start consume_messages. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ADIStateMachine [INFO] ADIStateMachine.cpp:54:moveTo(): ADI state moved to ADI_RUNNING May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.ISEConnectionSub [WARN] ISEConnectionSub.cpp:136:start(): Configuration doesn’t have enough Pxgrid information May 02 11:55:45 SF-IMS[10482]: [10482] ADI:adi.RpcServer [INFO] RpcServer.cpp:204:start(): starting rpc server May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:436:registerSubjectDispatcher(): Added subject authenticate for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:436:registerSubjectDispatcher(): Added subject directory test for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject ldap query for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject realm information feed for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject refresh request adi service session directory for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject refresh request ise security group feed for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject refresh request ise SXP feed for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject refresh request ise endpoint feed for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject ise Query Remediation Capabilities for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [DEBUG] server.cpp:424:registerSubjectCallback(): Added subject ise remediation action request for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:infra.ev-rpc [INFO] server.cpp:383:start(): Started server ADI, listening for clients. May 02 11:55:45 SF-IMS[10482]: [10524] ADI:infra.ev-rpc [DEBUG] server.cpp:305:channelManagerThread(): Starting channelManagerThread for server ADI. May 02 11:55:45 SF-IMS[10482]: [10482] ADI:ADI [INFO] adi.cpp:447:main(): Started May 02 11:55:45 SF-IMS[10482]: [10523] ADI:infra.ev-rpc [DEBUG] server.cpp:276:channelUpdateThread(): Starting channelUpdateThread for server ADI. May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:236:initializeLinaChannel(): Initializing lina channel by sending hello May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.ZMQServer [DEBUG] ZMQServer.cpp:251:requestLinaSync(): Requesting lina vpn event sync May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.LinaEventHandler [DEBUG] LinaEventHandler.cpp:201:handleMessage(): Received VPN Sync response containing 6 events May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.SubscriberChannel [DEBUG] SubscriberChannel.cpp:39:sendMessage(): Sending request to ADI subscriber May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.SubscriberChannel [DEBUG] SubscriberChannel.cpp:56:sendMessage(): Receiving response from DataCorrelator May 02 11:55:46 SF-IMS[10482]: [10487] ADI:adi.LinaEventHandler [DEBUG] LinaEventHandler.cpp:205:handleMessage(): Successfully sent bulksync to SFDataCorrelator May 02 11:55:46 SF-IMS[10482]: [10508] ADI:realm.join.retryer [DEBUG] RealmJoinRetryer.cpp:90:threadWorkerFunction(): No more realm join retries needed, worker thread done. May 02 11:55:46 SF-IMS[10482]: [10527] ADI:infra.ev-rpc [DEBUG] server.cpp:184:subscribe(): Subscribed client 0x5558b8710180 to subject adi service session directory. May 02 11:55:46 SF-IMS[10482]: [10527] ADI:infra.ev-rpc [DEBUG] server.cpp:184:subscribe(): Subscribed client 0x5558b8710180 to subject ise security group feed. May 02 11:55:46 SF-IMS[10482]: [10527] ADI:infra.ev-rpc [DEBUG] server.cpp:184:subscribe(): Subscribed client 0x5558b8710180 to subject ise endpoint feed. May 02 11:55:46 SF-IMS[10482]: [10527] ADI:adi.RpcServer [DEBUG] RpcServer.cpp:295:realm_feed_subject_handler(): adding realm Info to feedrealm id = 4 domain = example.com short domain = ONG May 02 11:55:46 SF-IMS[10482]: [10527] ADI:adi.RpcServer [INFO] RpcServer.cpp:344:triggerSecurityGroupBulkDownloadCallback(): fill SECURITY_GROUP_WIPE_ALL event May 02 11:55:46 SF-IMS[10482]: [10527] ADI:adi.ISEConnection [DEBUG] ISEConnection.cpp:981:performEndpointProfileBulkDownload(): Postponing Endpoint Profile download until future connect. May 02 11:55:46 SF-IMS[10482]: [10527] ADI:adi.ISEConnection [DEBUG] ISEConnection.cpp:1678:kickoffSessionsBulkDownload(): Postponing Sessions download until future connect. May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:578:consume_messages(): consume_messages rval:1 May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:1458:publish_message_timeout_request(): requesting msg_timeout catchup May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:757:request_catchup(): SXPFollower expecting: 16 requesting catchup, target seq_no: 0 ,req_id: 1 snapshot entry offset: 0 May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:792:request_catchup(): published catchup request May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:1515:change_follower_state(): entering transient state: 2, setting timer to 1619956560 May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:576:consume_messages(): start consume_messages. May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:578:consume_messages(): consume_messages rval:0 May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:648:get_follower_event(): session seq_no is: 16 next_seq_no is: 16 May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:1225:process_message(): follower event is: CATCHUP_UPTODATE May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:1213:process_catchup_uptodate_message(): received catchup entry up-to-date msg. May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:977:process_buffered_messages(): begin to process 0 buffered messages May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [INFO] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:1010:process_buffered_messages(): completely processed buffered broadcast messages. May 02 11:56:00 SF-IMS[10482]: [10507] ADI:FPReplicationFollower [DEBUG] /var/tmp/build/workspace/IMS/IMS_6_7_0/FTD/src/libraries/fpreplication/src/FPReplicationFollower.cpp:576:consume_messages(): start consume_messages.
Step 04: Terminate the process by sending ^C to your shell (CTRL +C on your keyboard)
Step 05: Enable adi process again
root@ftd01:/home/admin# pmtool enableById adi
Step 06: Verify that adi is managed by process manager, up and running
root@ftd01:/home/admin# pmtool status | grep adi
adi (normal) - Running 10710
Command: /ngfw/usr/local/sf/bin/adi
PID File: /ngfw/var/sf/run/adi.pid
Command: /ngfw/usr/local/sf/bin/beakerd --module-path=/ngfw/usr/local/sf/lib64 --log-file=/ngfw/var/log/beakerd.log --log-level=info --cfgdir=/ngfw/etc/sf/beakerd --pidfile=/ngfw/var/sf/run/beakerd.pid --enable-module=updater --enable-module=nikita --datadir=/ngfw/etc/sf/beakerd --listen=/ngfw/var/tmp/beaker_client.sock --listen=/ngfw/var/tmp/beaker_async_client.sock
Summary#
In this post we took a look at Firepowers pmtool utility and how we can use it control processes running on the system. You should now be able to verify the status of a process, restart it or disable it to perform additional system debugging. If you are interested in analyzing firepower logs checkout my post on pigtail, another useful troubleshooting utility for normalizing and analyzing firepower logfiles.



