I am wondering if there can be more options in format.txt. For example, restricting the number of cards in each set, or restricting no identical cards can be selected in a deck.
No. Those filters only select which sets are shown.
I see, but is there a way to do things I have mentioned above?
No. People need to know what they are allowed to put in a deck, the same as they would if they were constructing the deck with physical cards.