One of the most mysterious things in Firebird is definitely its optimizer. There were several teams worked at InterBase Optimizer during Ashton-Tate, InterBase Corp and Borland InterBase stages of history, and Firebird team in the beginning of 200x faced the full set of optimizer techniques you can imagine: histograms, ad hoc hacks, different cost algorithms and so on.
Arno Brinkman was one of the the first optimizer warriors: he performed a lot of improvements in optimization approach and made Firebird 1.5 much faster and predictable than previous versions, and it helped Firebird 2.0 won against InterBase 7.5 in tpc-r like test (we have performed it during IBDeveloper times).
Since version 2.1 Dmitry Yemanov is in charge for optimizer development in Firebird. Below is his short presentation regarding basics of optimizer in Firebird.
How optimizer works, how it decides to use this or that index, why sometimes it fails and what you can do to improve performance?
Definitely this presentation has no answers for all these questions but it gives you a basic knowledge of Firebird optimizer internals.
This is not for everyone and requires some qualification, definitely, but it should be interesting for real Firebird geeks.