Oliver Kaiser

I'm a Systems Engineer focusing on networking and automation. Opinions are my own.

Firepower Management Center HA

10 Jul 2017 »

Configuring HA for FMC is pretty straight forward but how exactly does it work and how can we troubleshoot HA if it is not working correctly? In this post I will show you what FMC HA is doing behind the scenes and tools we have available to take a deeper look into the system and uncover issues.

FMC High Availability

High Availability is available on physical Firepower Management Center appliances. It lets you create an active/standby HA solution which does not require layer 2 adjacency (making it possible to have real HA over multiple sites without extending layer 2 across sites).

FMC HA will create a second “manager” registration on your sensor resulting in two sftunnel connections. One to your primary FMC and one to your secondary FMC.

It is a cold-standby solution which forces you to promote the passive FMC to active in case of failure of the primary device, but dont worry… all your events will be logged to both FMCs so in case of device failure you should not lose any events sent from the sensor to FMC. Just promote the passive FMC and it will start up all the necessary processes and become the active unit.

Behind the scenes the HA procedure consists of a sybase database mirror and a transaction framework that will make sure data is being replicated from the active to the passive management center.

fmc-ha-status

Configuration

Before configuring FMC HA make sure that…

  • Hardware is identical (you cant build HA between different hardware appliances)
  • Software release is identical
  • There are no sensors registered to the secondary FMC. If so, remove registration
  • You have a working backup (I havent seen a case where HA initialization caused any issues that would require re-imaging… but you never know :)

To configure HA login to FMC UI, go to Integration > High Availability and define a secondary peer. Then you grab some coffee… because it will take some time and you will see some warnings that will make you think it didnt work, but just be patient.

After some minutes the status of HA Synchcronization should change to “OK” like this:

fmc-ha-status

At this point your sensors should be registered with your passive FMC and should be listed when you navigate to the devices page on the passive FMC.

Upgrade Procedure

To upgrade an FMC in HA you will have to follow the following instructions:

  • Manually stop HA synchronization
  • Upgrade the passive FMC
  • Wait for the upgrade to finish (HA state may change to degraded, which is normal)
  • Upgrade the active FMC
  • Grab a coffee and give it some time to boot up after the upgrade procedure
  • Promote the primary FMC to become active
  • Deploy configuration to your sensors to verify everything is working as expected

Make sure to always check the current release notes for additional information!

Troubleshooting

Device not registered to secondary FMC after HA configuration

In case the device registration failed you will have to remove the sensor from your active FMC and login into your sensor. You will need to use the configure manager delete command followed by the configure manager add command to add your sensor to FMC again. At this point I would advice you to open up pigtail on both your sensor and FMC and re-add the sensor on the active FMC.

Using pigtail you will log all necessary output to find any issues in case the registration fails again.

Synchronization stopped during FMC backup

This behavior is by design and is not an issue. If a backup of FMC is being performed the HA synchronization will be stopped. During this timeframe you can continue configuration on your active FMC.

Log Files

Events related to FMC HA are being logged to /var/log/syncd.log. In case you have any issues that cant be solved via the UI make sure to check this logfile for further details.

CLI Tools

FMC ships with two perl scripts that can be used to query for high availability information. I would recommend not using these tools to change any configuration parameters but only to verify the current state of FMC HA. In the past I have usedmanage_HADC.pl to switch roles, break HA etc. and did not have any issues but keep in mind that I was working on a non production environment.

manage_HADC.pl

manage_HADC.pl provides a cli interface to query the current HA state and do some management tasks that are also exposed using the UI (atleast to some degree)

****************  Configuration Utility  **************

 1   Show HA Status
 2   Register as Secondary HA FMC
 3   Register as Primary HA FMC
 4   Switch HA roles
 5   Pause Mirror
 6   Re-establish Mirror
 7   Set as Active for Full UI
 8   Force as Active to resolve split brain
 9   Break HA - keep devices
 10  Break HA - delete devices
 0   Exit

**************************************************************

Option 1 is probably the only option you want to select. It will display the current state of HA with some additional details

SYNC_ACTIVE: 1 at /usr/local/sf/lib/perl/5.10.1/SF/Synchronize.pm line 456, <STDIN> line 1.
Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.10.1/SF/Transaction/HADC.pm line 232.
Sybase state :  at /usr/local/sf/lib/perl/5.10.1/SF/Synchronize.pm line 654.
Check peer ftd01.example.com... at /usr/local/sf/lib/perl/5.10.1/SF/PeerManager/Peers.pm line 51.
Check peer ftd02.example.com... at /usr/local/sf/lib/perl/5.10.1/SF/PeerManager/Peers.pm line 51.
Check peer fmc-secondary... at /usr/local/sf/lib/perl/5.10.1/SF/PeerManager/Peers.pm line 51.
Sybase state :  at /usr/local/sf/lib/perl/5.10.1/SF/Synchronize.pm line 747.
Sync status :Synchronization Task In-progress  at /usr/local/sf/lib/perl/5.10.1/SF/Synchronize.pm line 748.
Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.10.1/SF/Transaction/HADC.pm line 232.
STATUS:
Active HA DC: 1 
HA_SYNC: SYNCHRONIZATION_HADC 
LastFailover: Success, Mon Jun 12 19:54:23 2017 
Sybase_State: 1 
blocking: 0 
connected: 1 
degraded: 0 
degraded_message: ARRAY(0xabd5e78) 
mirroring: 1 
peer_connected: 1 
status: ARRAY(0xacdb880) 

troubleshoot_HADC.pl

As the name of the script indicates you can use troubleshoot_HADC.pl to verify some additional information not available using manage_HADC.pl

****************  Troubleshooting Utility  **************
 1   Show HA Info Of FMC
 2   Execute Sybase DBPing
 3   Show Arbiter Status
 4   Check Peer Connectivity
 5   Print Messages of AQ Task
 0   Exit
**************************************************************

Option 1 will display the current status of the sybase database replication

Enter choice: 1

HA Enabled: Yes

This FMC Role In HA: Active - Primary
Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)
Sybase Database Connectivity: Accepting DB Connections.
Sybase Database Name: csm_primary
Sybase Role: Active

Option 2 will connect to the local sybase database to verify that connectivity is working

Enter choice: 2
Sybase Database Name: csm_primary
$VAR1 = [
          'Mirror Server => csm_primary',
          {
            'stderr' => undef,
            'stdout' => 'SQL Anywhere Server Ping Utility Version 16.0.0.2271
Type       Property                  Value
---------  ----------------          ------------------------------
Database   MirrorRole                primary
Database   MirrorState               synchronizing
Database   PartnerState              connected
Database   ArbiterState              connected
Server     ServerName                csm_primary
Ping database successful.
',
            'rcode' => 0
          }
        ];

Option 4 will display the status of the peer (other FMC). Use this option to verify software version, ip address, etc. are correctly set

Enter choice: 4
Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected

$VAR1 = {
          'vip_local' => '',
          'priority' => '0',
          'ip' => '10.1.1.2',
          'model_id' => 'F',
          'uuid' => 'f1fa29d2-7b4f-11e6-b5a3-ca59e03bf0e4',
          'sw_version' => '6.2.0.2',
          'upgrade_version' => '',
          'persistent' => '0',
          'mgmt_mac_address' => '00:62:EC:42:EE:F2',
          'vnet' => undef,
          'primary_mgr' => '0',
          'vip' => '',
          'model_number' => '66',
          'ipv6' => undef,
          'reg_state' => '0',
          'name' => 'fmc-secondary',
          'active' => '1',
          'uuid_gw' => '',
          'reg_key' => '',
          'last_changed' => '1491849682',
          'role' => '0'
        };