I think you can simplify it down to this:
function setQuoteFor() { var lookup = Xrm.Page.getAttribute("customerid").getValue(); if (lookup != null) { if (lookup[0].entityType == "contact") { Xrm.Page.getAttribute("new_quoteforentity").setValue(100000000); } if (lookup[0].entityType == "account") { Xrm.Page.getAttribute("new_quoteforentity").setValue(100000001); } } }
A couple things to note:
JavaScript is case sensitive so XRM is not the same as Xrm - Xrm is correct
Field names - are all lower case - on the screen showing the field data under customizations - use the Name field
Option set fields can be set with just the integer value - since it is a number there won't be any commas
You started out referencing "lookupItem" and then went to "lookup"
But you are on the right track :)