This paper reports on the cooling performance of a vegetable oil based magnetic nanofluid with different volume fractions of magnetic nanoparticles (MNPs) resulting from magnetoconvection effects in presence of an external magnetic field. Mineral-based oils are typically used in power transformers because of their high dielectric strength and good cooling performance under operating conditions. Owing to their environmental impact, the eco-friendly vegetable transformer oils have been lately replacing the mineral-based insulating oils. Until now, several studies reported only the dielectric breakdown characteristics of vegetable oil based magnetic nanofluids without considering any real electromagnetic system. Besides, the thermal characteristics of vegetable oil based magnetic nanofluids have not been fully discussed yet. The aim of our study, therefore, is to examine the cooling performance by adding magnetic nanoparticles to the vegetable transformer oil. To quantitatively analyze this effect, a multiphysics technique coupled with magnetic-thermal-fluidic fields was developed by using the quasi-static magnetic field approximation and conjugate heat transfer. To validate our numerical setup, some experiments were successfully conducted in a simple electromagnetic system with different insulating liquids.