FMC Task stuck? Deleting broken tasks from database

by Oliver Kaiser
Thumbnail

Every now and then tasks get stuck on Firepower Management Center. Even though there are mechanisms in place that should cancel long running tasks after a timeout is reached this problem has persisted for a quite some time (source: I’ve seen it happening for more than 5 years).

Engaging TAC to resolve the problem manually is the only officially supported solution, but doing it yourself is relatively safe in my opinion. I have executed the CLI procedure I am listing here for years without any issues, just keep in mind that you are deleting data directly from the systems database, which might not be a very smart move if you do not understand the inner working of the software you are trying to beat into submission.

Make sure you only use this procedure as a last resort. Various tasks have different timeout settings. If a deployment is running for 15 minutes it’s not a smart move to delete the tasks from the FMC database, since this will not stop the running deployment, but only makes FMC unaware of it!

Possible reasonable usecases for following this guide are configuration deployment that ran for multiple hours, backup jobs that already failed but are still stuck in the task view for multiple days…

Note

The procedure for deleting hanging tasks differs between software releases since Cisco changed the database backend from Mysql/MariaDB to Sybase

Step 01: Switch to bash (expert) shell and change to root user

> expert
 admin@fmc01:~$ sudo su - 

Step 02: Execute OmniQuery.pl to search for running tasks

root@fmc01:~#  OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"
+--------+-------------------+----------------------------------+
| status | category          | hex(uuid)                        |
| 7      | task:category.150 | bb0bba970b4c4423927b8f7d237edd0b |
+--------+-------------------+----------------------------------+
1 rows in set

Step 03: Copy the uuid of your task and delete it from Sybase using OmniQuery.pl:

root@fmc01:~#  OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("bb0bba970b4c4423927b8f7d237edd0b ");'
Query OK, 1 rows affected (0.000 sec) 

Step 04: Double-check to make sure the entry was successfully deleted from notification table

OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"

Empty set (0.001 sec) 

Step 05: Restart FMC Processes for changes to take effect

root@fmc01:~#  /etc/rc.d/init.d/console restart

Step 01: Switch to bash (expert) shell and change to root user

> expert
 admin@fmc01:~$ sudo su - 

Step 02: Execute mysql query to search for running tasks

root@fmc01:~# mysql -padmin -uroot sfsnort -e "select status,category,hex(uuid) from notification where status=7;"
+--------+-------------------+----------------------------------+
| status | category          | hex(uuid)                        |
| 7      | task:category.150 | bb0bba970b4c4423927b8f7d237edd0b |
+--------+-------------------+----------------------------------+
1 rows in set

Step 03: Copy the uuid of your task and delete it from the database:

root@fmc01:~# mysql -padmin -uroot sfsnort -e "delete from notification where uuid=unhex("bb0bba970b4c4423927b8f7d237edd0b ");"

 Query OK, 1 rows affected (0.000 sec) 

Step 04: Double-check to make sure entry was successfully deleted from notification table

root@fmc01:~# mysql -padmin -uroot sfsnort -e "select status,category,hex(uuid) from notification where status=7;"

Empty set (0.001 sec) 

Step 05: Restart FMC Processes for changes to take effect

root@fmc01:~#  /etc/rc.d/init.d/console restart

After your FMC has successfully restarted, login to the UI and make sure the task / deployment is no longer listed:

Screenshot showing no running deployment in the background

1 comment
2

Leave a Comment

* By using this form you agree with the storage and handling of your data by this website.

1 comment

rnRetBScdh June 15, 2021 - 23:05

XNLhoewjnIt

Reply

Related Posts