Hi Ka, I feel your pain, I'm an introvert too! My advice would be to pick just one coworker to befriend. Sometimes it's overwhelming to think about befriending the whole office. Maybe there's one person who seems more approachable than the others... every day, challenge yourself to chat with this person at least once.
Either (1) give a brief compliment ("I love your shoes," or "That was a great point you brought up..."), or (2) ask a question (about their weekend, their pet, or their partner / kids). The goal is to find something you have in common - a hobby, a love of dogs, favorite lunch spot, or a shared frustration.
And there's no need to enter into a lengthy conversation - start small, and you'll find that it gets easier the more you interact with this person. In my experience, it only takes one connection to get "in" with the larger group. Thanks for the honest question, I hope this helps! :)