ESTA is valid but the Visa was denied- can I travel to the USA?

From the State's Department Visa Waiver Program webpage:

The Visa Waiver Program (VWP) allows citizens of participating countries* to travel to the United States without a visa for stays of 90 days or less, when they meet all requirements explained below. Travelers must be eligible to use the VWP and have a valid Electronic System for Travel Authorization (ESTA) approval prior to travel.

(Note that it says “and”, still having a valid ESTA does not exempt you from fulfilling the other requirements for eligibility at the time of entry.)

According to the same page, one requirement for eligibility is:

Previous Compliance and No Prior Visa Ineligibilities

If you have had a U.S. visa before or previously traveled to the United States under the VWP or another status, you must have complied with the conditions of previous admissions to the United States, and you must not have previously been found ineligible for a U.S. visa.

There are some subtleties regarding what counts as “being found ineligible” and you haven't provided any details on the reason cited to refuse your O-1 visa application but it seems you might not be eligible for the Visa Waiver Program anymore.

If you have in fact been found ineligible, this means that not only is there “a chance” to be denied entry but in fact that you should be denied entry if someone finds out about your earlier visa denial. You probably also would not be able to get a new ESTA without lying on the application form (which would create a new sets of liabilities for you, up to a potential ban from the US).

Also note that the rule, like other rules of this kind, apparently refers to you as a person, not to your passport. Using another passport/citizenship to try to get around checks does not make you eligible for the Visa Waiver Program.

Your latest comment suggests it might in fact still be possible to travel on an ESTA in your case. As explained on the Previously denied a visa or immigration benefit page from the CBP website, the best course of action would be to ask a US consulate about that before attempting to enter the US.

The answer is yes, there is a chance that you could be denied entry, and there is a chance that the denial could be based in part on your recent denied visa application. There is always a chance that you can be denied entry; the goal is to minimize the probability.

Based on the information in Relaxed's answer, and in Tor-Einar Jarnbjo's comment on that answer, I suspect that the probability is small, but do you want to run that risk? You're probably best off by taking Tor-Einar Jarnbjo's suggestion of asking at the consulate. That way you can remove the uncertainty before you travel, not when you arrive at the border (in addition to avoiding the potential cost of wasting your trip, you are also more likely to handle the situation better if you haven't just spent half a day in sleep deprivation on an airplane -- if you're anything like me, that puts you in exactly the wrong mood for dealing with government bureaucracy.)

Explain that you want to travel to the US as a tourist (or whatever visa-waiver-legitimate purpose you actually have for travelling to the US). Tell them that you would like to do this with your European passport, on the visa waiver program with ESTA. Tell them that you applied for an O-1 visa using your Israeli passport, and ask whether the denial of those applications disqualifies you from using the VWP and ESTA. If they say that it does, then apply for a B visa, using whichever passport you prefer.

Note that the visa waiver program is only valid for travel that would be permitted under a B visa (see http://travel.state.gov/content/visas/english/visit/visa-waiver-program.html). You may therefore face questions related to your recent applications for a different category of visa. A successful response to these questions would presumably take one of two forms:

1. Show that the purpose of your travel is permissible under a B visa, even though you applied earlier for an O-1 visa.
2. Show that your plans have changed since you applied for the O-1 visa, and that you no longer intend to pursue the activity for which you applied for that visa.

I say "show" rather than "say" because the more evidence you can present to support your assertions, the more likely you are to succeed in your application.

In general, a current ESTA holder is required to apply for a new ESTA if they want to use the Visa Waiver Program once a visa application has been denied (see below). The phrasing "your current or previous passport" in the ESTA question is a little unclear as to whether it technically applies to an alternate passport. I imagine it likely does.

8 CFR § 217.5 - Electronic System for Travel Authorization

(e) New travel authorization required. A new travel authorization is required if any of the following occur: ....

(5) The circumstances underlying the alien's previous responses to any of the ESTA application questions requiring a “yes” or “no” response (eligibility questions) have changed.

ESTA question:

Have you ever been denied a U.S. visa you applied for with your current or previous passport, or have you ever been refused admission to the United States or withdrawn your application for admission at a U.S. port of entry?