Additional Study Information: This trial studies the side effects and best dose of nivolumab when given together with radiosurgery and to see how well they work with or without ipilimumab in treating participants with grade II-III meningioma that has come back. Nivolumab and ipilimumab (two different types of immunotherapy drugs), may interfere with the ability of tumor cells to grow and spread. Stereotactic radiosurgery is a type of non-surgical radiation therapy. It is not yet known whether giving nivolumab and multi-fraction stereotactic radiosurgery with or without ipilimumab may work better in treating participants with grade II-III meningioma.