Implicit conversion from class to std::ofstream not working as expected

I have a problem with getting an implicit conversion from my class "File" to an std::ofstream.

Here is the interesting stuff from the File.h:

class File
{
    public:
        File(const std::string path);
        operator std::ofstream&();

    private:
        std::ofstream _ofs;
        std::string _path;
}

And here is the File.cpp:

File::File(const std::string path)
    : _path(path)
{}

File::operator std::ofstream&()
{
    if (!_ofs.is_open()) _ofs.open(_path);
    return _ofs;
}

When I try to use the file with the << operator like this:

File file("test.txt");
file << "test";

g++ gives me the compile time error "Invalid operands to binary expression ('File' and 'const char *')". However, if instead of the "const char *" I use a class I've written my own << operator for (with operands "std::ostream&" and the class, it works as expected. E.g:

class Color;
std::ostream& operator<<(std::ostream&, const Color&);

Color color;
File file("file.file");
file << color;

This works as expected. So the conversion to std::ofstream seems to work, but somehow the << operator for "std::ostream" and "const char *" is not.