Do European train bookings include reserved seats on a specific train departure?

It really depends on the country/train, IC is a code used in different countries for long-distance non-high-speed trains with slightly different rules. Generally speaking, those trains have either no seat reservations or optional seat reservations (but no mandatory seat reservations).

Therefore I would assume you do have a ticket that's valid on your chosen trains. In some cases, seat reservations are available but not mandatory and open tickets do not mention the train for that reason. For the one specific route you mentioned, seat reservations are simply not available on IC trains at all. If you're really concerned about finding a seat, you could book a first class ticket. It doesn't guarantee anything either but there is usually more space in first class.

Where available, it is certainly possible to book a seat reservation separately but that's usually not possible online (the German railways do sell them online).

From what you said, you can expect that you do not have a reservation.

On TGV/Thalys you are (usually) required to have a seat reservation, so whenever you buy a ticket, you kinda automatically have one (otherwise you can't get a ticket). The seat number is printed on the ticket.

On ICE (note the "E") which is the better, faster "express" version of IC with approx. 1/3 as many stops and running at approx. 2x cruise speed, you (usually) get reservation for free and automatically in first class, but in second class you do not get it automatically. You have to book it explicitly, and it will cost you some 4-5€ extra, but I would absolutely recommend it because some ICEs are full to the last seat, and nothing sucks more than having to stand, or stay in the restaurant wagon (if there's one) for 4 hours.
I wouldn't know for sure about seat reservations for first class in IC (without "E"), the tree or four times I've travelled IC in my life, it was 2nd class, and seat reservation was (of course) extra.

In any case, the wagon and seat number is printed on the ticket if you have a reservation (if there's none, you don't have one).

Further steps to take would be to reserve a seat, which can be done online. Note that depending on what country you do it in, and depending on what alternative carriers are involved, it can be adventurous (for example, two years ago I learned that booking with Bahn and going to Italy would much to my surprise work just fine with the Italians, but the Austrians would give me trouble for the short segment in between. Same physical train, different carrier. Hey, if anything, you'd think it wouldn't work on the Italian end, but that was actually no issue at all.).

It is generally possible to buy tickets without seat, and funnily enough to reserve seats without a ticket. This seems nonsensical, but since there are tickets that are valid on several days (virtually unlimited, even) and subscriptions that allow 100% free of charge travel in at least some countries, this actually makes sense. Those people having e.g. one of the 100% reduced fare subscriptions would just book a seat when they wish to travel.

About "tight schedule", I'd like to warn you. This is not the best possible approach in combination with "train", in particular IC. There's countries which are arguably somewhat better, and some which are definitively not among the better ones, and it's not always obvious. IC per se is on the more unreliable side due to having many stops and being a lower-priority train. If you have a tight schedule, ICE/TGV may be a better choice (although I've had a 2 hours delay on a 3 hour trip on German ICE already, too).
You mentioned Belgium-Luxemburg being the first segment. My personal experience with Belgium is that you stand at the station and no train is to be seen. As you get more and more nervous and ask staff about it, they'll say "Yeah sure, that train leaves in 2 minutes. But not today.". When you ask when the next train will leave, they'll tell you: "Tomorrow morning at 6:14". Yup, Brussels, capital of Europe.

So... tight schedule may work, but it may not be the best plan.

Specifically for the IC from Brussels to Luxembourg City. This is the IC 21xx (with xx a sequence number for the nth train of the day starting at 04 for that train line). You can look up the exact schedule on the Belgian railway website (select the train radiobutton and enter your train line, f.e. IC2110). Date is optional, but recommended as some trains have different schedules in weekends and on holidays.

This is a standard Belgian IC train and does not have seat reservation or specific time requirements. Your ticket is very likely counted as a standard ticket (not 100% because booked through SNCF and not SNCB). The general conditions for this type of ticket are: valid on the date (you can get on any train that day) printed on the ticket and you are allowed to get off at any stop on that line and get on a later train on the same line (this is a recent change in conditions). So you can visit any cities in between, should you want to. You can find the conditions at the bottom of the page I linked to for standard ticket, but they are only available in french and dutch.

Optional information: For future reference, when travelling through Belgium, it can be more interesting financially to use several tickets with linked destinations to reduce your travel costs. In this case, I would have suggested to take a ticket from Brussels to Arlon (Arlon being the last stop in Belgium) and a ticket from Arlon to Luxembourg. This comes out slightly cheaper in general (few euros usually). However, if you are under 26 or travel frequently in Belgium, you can take advantages of significantly cheaper rides using GoPass 1 or GoPass 10 (under 26) or RailPass (over 26, at least 10 trips). Return trips are also cheaper in the weekend (if the return happens in the same weekend).