I don't know my nationality. How can I visit Denmark?

From your description it appears that you are stateless -- that is, no country recognizes you as its citizen. That makes international travel difficult, but what's generally supposed to happen is that the country where you're legally resident (that is, the United States) can issue you a passport-like document which can serve as an identification document while traveling but does not declare you to be a citizen of anywhere.

In most of the world this would be known as a "1954 convention travel document", but the US is not party to the 1954 Convention. Instead the USCIS can issue something called a U.S. Re-entry Permit, which a number of countries, including Denmark, recognize as equivalent to a 1954 convention document.

[Beware: As of 2017-04-09, the official lists of travel documents accepted by Schengen member states indicate that Denmark does not recognize US "permits to re-enter" but does recognize US "refugee travel documents". They are the same kind of booklet, but have different heading above the photo. There are different boxes you need to check on form I-131 when applying.]

After getting a re-entry permit you will still need to apply for a Danish visa, since you're not a U.S. citizen (who are visa-free in the Schengen area). On the visa application form, you would put "stateless" as your nationality and check "other travel document" rather than "ordinary passport". You would need to support your visa application with extra good evidence of ties with the US that motivate you to leave the Schengen area after your proposed trip (a green card is a good start here, but by no means sufficient), and having spent your childhood here is not going to make it easier. But it's not categorically impossible.

In the long run it would probably be in your interest to pursue naturalization as a U.S. citizen.

Henning Makholm's answer addresses the issue of your possible statelessness, and suggests a way of obtaining a travel document.

The other alternative is to acquire a citizenship. Your case is complicated and I would recommend consulting an immigration lawyer. Below is some relevant information, mostly extracted from Wikipedia. Note that it is possible that you are not in fact stateless (e.g. if you are considered a citizen of Thailand or Vietnam), but merely lack documents to prove your citizenship.

Thailand

Thailand's nationality law is based on the principles of both Jus sanguinis (citizenship through parents) and Jus soli (citizenship through birth). Your parents were not Thai citizens, so that leaves birth. This paragraph provides a summary.

Vietnam

Wikipedia doesn't have an entry for Vietnam's nationality law, but searching Google produced hits for me that suggest that the principle of Jus sanguinis (citizenship by parents) applies, so you may be considered a Vietnam citizen regardless of the fact that you were not born there.

Denmark

There was a real missed opportunity here as you and your family would have been entitled to acquire citizenship after being resident here for 9 years (known as naturalization) but as you have permanently become non-resident since then, that probably doesn't count for anything anymore. On the other hand, you say you have a travel document jointly with your mother. What kind of travel document is it? Presumably you used this document to enter the USA. How did you address the issue of your citizenship at that time?

USA

You didn't specify how long you have been resident in the USA for, but you should be able to naturalize as a citizen if you have been a permanent resident for 5 years. If this is the case, this is probably the easiest option as you can easily establish your case.

Dual / Multiple Citizenship

As it is possible that you may be eligible for two or more of the above citizenships, you may want to carefully consider your options. The laws regarding multiple citizenship can be complicated and vary for each country. Many countries, and particularly Asian countries, forbid multiple citizenship. So e.g. by taking up USA citizenship you may cut yourself off from the others. Whether or not this matters to you depends on what your long term plans are and where you intend to eventually settle. One downside of USA citizenship (and almost uniquely in the world) is the requirement to file US tax returns for the rest of your life (while you remain a citizen), even if you stop residing there (you may or may not have to actually pay US taxes, depending on your income and other circumstances).

I don't know my nationality. How can I visit Denmark?

Simple! talk to USCIS (could be as simple as using Ask a Question link on their website) and ask them that. Since they issued you a green card they will know your exact status in their records. They can best tell if you are a citizen of any country according to their records or not

Depending upon their answer you can then chose the best course of action later

It sounds like you are mainly concerned with stating what your nationality is on the Danish visa application form.

Since you have a US green card and you do not hold a passport, you would likely be traveling with a US re-entry permit (Form I-327), which is a passport-like booklet issued by USCIS that says “Travel Document” on the cover and is used in lieu of a passport. You would be submitting the re-entry permit to apply for your Danish visa, instead of a passport, and the visa sticker would be placed in your re-entry permit.

The re-entry permit lists a nationality, which would be the “country of citizenship” that is listed in field 5 of the Form I-131 you use to apply for the re-entry permit. Whatever nationality USCIS prints in your re-entry permit (which, based on other answers, may be Thai, Vietnamese or stateless) is the nationality you should fill out on the Danish visa forms. The Danish authorities are unlikely to question the nationality stated by USCIS or ask for further evidence.

(The nationality you provide on Form I-131 should match what you have previously told USCIS when you applied for the green card, or you might be asked to provide further evidence.)

The short, pragmatic answer to your question is: use whatever nationality is printed in your re-entry permit, unless it is clearly wrong.

The “joint Danish travel document” you are referring to is most likely a “fremmedpas” or “alien’s passport”, which is like the US re-entry permit and is issued to Danish residents who are stateless or unable to apply for a passport from their home countries, usually refugees. The Danish alien’s passport will also list a nationality, and ideally that would also match the nationality in your US re-entry permit; if not, you might be asked to explain the discrepancy, which might actually be relatively easy due to the objective confusion about your nationality.

The easiest way to address this would be via a birth certificate and the easiest route will likely be Thailand. Thailand extends citizenship to anyone born there.

If you were issued a birth certificate in Thailand, then it's quite likely you can approach the Thai Embassy in the US to confirm and issue citizenship documentation/passport.

If you weren't issued a birth certificate, contact the Thai Embassy anyway. With sufficient documentation surrounding your time there they should be able to confer citizenship.