Mathematical methods in niche theory study