Here comes a correction: At first I thought it worked, because started to show 200 OK headers randomly, but not related the hotfix above, not sure if it was browser cache or something else.
I found the source of 404's though, it's Batcache plugin. I added a line to disable it for site map URL (just for parent site map (sitemap_index.xml) was enough) and it worked. Like this:
if( strpos( $_SERVER['REQUEST_URI'], 'sitemap_index.xml' ) )
return;