If somebody I didn't know that well said 'You are silly' then I would find it ambiguous and would have to think about what they meant, and depending on the context I might well think they did mean 'stupid' or 'foolish', or they might mean your intended meaning.
Not knowing what they meant I would probably expect them to qualify it (which, of course, you did!), or be looking for non-verbal signs.
While we're off topic
