Firepower Management Center HA
Configuring HA for FMC is pretty straight forward but how exactly does it work and how can we troubleshoot HA if it is not working correctly? In this post I will show you what FMC HA is doing behind the scenes and tools we have available to take a deeper look into the system and uncover issues.
FMC High Availability
High Availability is available on physical Firepower Management Center appliances. It lets you create an active/standby HA solution which does not require layer 2 adjacency (making it possible to have real HA over multiple sites without extending layer 2 across sites).
FMC HA will create a second “manager” registration on your sensor resulting in two sftunnel connections. One to your primary FMC and one to your secondary FMC.
It is a cold-standby solution which forces you to promote the passive FMC to active in case of failure of the primary device, but dont worry… all your events will be logged to both FMCs so in case of device failure you should not lose any events sent from the sensor to FMC. Just promote the passive FMC and it will start up all the necessary processes and become the active unit.
Behind the scenes the HA procedure consists of a sybase database mirror and a transaction framework that will make sure data is being replicated from the active to the passive management center.
Before configuring FMC HA make sure that…
- Hardware is identical (you cant build HA between different hardware appliances)
- Software release is identical
- There are no sensors registered to the secondary FMC. If so, remove registration
- You have a working backup (I havent seen a case where HA initialization caused any issues that would require re-imaging… but you never know :)
To configure HA login to FMC UI, go to
Integration > High Availability and define a secondary peer. Then you grab some coffee… because it will take some time and you will see some warnings that will make you think it didnt work, but just be patient.
After some minutes the status of HA Synchcronization should change to “OK” like this:
At this point your sensors should be registered with your passive FMC and should be listed when you navigate to the devices page on the passive FMC.
To upgrade an FMC in HA you will have to follow the following instructions:
- Manually stop HA synchronization
- Upgrade the passive FMC
- Wait for the upgrade to finish (HA state may change to degraded, which is normal)
- Upgrade the active FMC
- Grab a coffee and give it some time to boot up after the upgrade procedure
- Promote the primary FMC to become active
- Deploy configuration to your sensors to verify everything is working as expected
Make sure to always check the current release notes for additional information!
Device not registered to secondary FMC after HA configuration
In case the device registration failed you will have to remove the sensor from your active FMC and login into your sensor. You will need to use the
configure manager delete command followed by the
configure manager add command to add your sensor to FMC again. At this point I would advice you to open up
pigtail on both your sensor and FMC and re-add the sensor on the active FMC.
Using pigtail you will log all necessary output to find any issues in case the registration fails again.
Synchronization stopped during FMC backup
This behavior is by design and is not an issue. If a backup of FMC is being performed the HA synchronization will be stopped. During this timeframe you can continue configuration on your active FMC.
Events related to FMC HA are being logged to
/var/log/syncd.log. In case you have any issues that cant be solved via the UI make sure to check this logfile for further details.
FMC ships with two perl scripts that can be used to query for high availability information. I would recommend not using these tools to change any configuration parameters but only to verify the current state of FMC HA. In the past I have used
manage_HADC.pl to switch roles, break HA etc. and did not have any issues but keep in mind that I was working on a non production environment.
manage_HADC.pl provides a cli interface to query the current HA state and do some management tasks that are also exposed using the UI (atleast to some degree)
Option 1 is probably the only option you want to select. It will display the current state of HA with some additional details
As the name of the script indicates you can use troubleshoot_HADC.pl to verify some additional information not available using manage_HADC.pl
Option 1 will display the current status of the sybase database replication
Option 2 will connect to the local sybase database to verify that connectivity is working
Option 4 will display the status of the peer (other FMC). Use this option to verify software version, ip address, etc. are correctly set