Dodging the oncoming bikes, I climb a low mound a few feet above water level and look across at the city. It is a clear evening: the horizon holds a delicate tang of orange that ascends, becoming royal blue, then indigo. In front of me is a quay where a few boats are moored; beyond that, an expanse of water, and then the city itself, a smudge of bare trees, steep gables and church spires. It’s pleasant, but scarcely dramatic, and yet I think that this view changed the way my brain is wired. In about 1660 an artist came down here and painted what he saw. Johannes Vermeer was not a famous man then, nor would he be for more than 200 years, but that picture, View of Delft, would prove pivotal in art history.

Delft is a small Dutch city of about 100,000 people, just 40 miles south-west of Amsterdam. It was carved out of low-lying land in the 13th century, then grew into a centre for printing, pottery and, by the early 17th century, fine art. Perhaps it was the light that inspired the painters: those big northern skies reflected in the canals and the contrasting shady interiors filled with men in black hats and women with pale, enigmatic faces.

I cross a bridge and enter the old town, a maze of narrow canals and arched metal bridges that thunder under the tyres of cargo bikes. Lights are on in the old wood-panelled bars where office workers are taking borrel, a Dutch post-work tradition of beer and snacks. The shops and restaurants here all fit inside the dimensions of the 17th-century houses and all seem to be named after what they sell – I love the directness – guess what Bloemen, Voetbal and Hummus can offer. In the centre of town, in the marketplace, they are putting up stalls for tomorrow’s market. To one side is the spot where the family house of Vermeer once stood. Nearby is the guildhall where the artist presided for a few years (now a centre devoted to the man and his work).

Not much is known about Vermeer’s artistic life. You could jot all the important facts on the back of a lace collar. No one knows, for example, who taught him to paint, or how many canvases he actually produced. No sketches, letters or diaries survive. We know he lived in reasonable comfort, died in poverty in 1675, and a century later was almost completely forgotten.

Art shows are a huge draw these days. As well as fixtures like the Biennale and Frieze, you have the epic one-off moments in a lifetime. Vermeer at the Rijksmuseum in Amsterdam (an hour by train from Delft) will be one of those. Opened last week and running until 4 June, it’s the first time ever that 28 of the man’s 37 known works are in one place. Even Vermeer himself probably never had that pleasure. And the show itself – book now because it will soon be a sellout – is a stunner. But for me, it’s the backstory in the surrounding area that makes it compelling.

In the old city quarter of Delft, the cardinal points of Vermeer’s life are clustered around the marketplace. Behind his grave inside the Oude Kerk (Old Church) is the Museum Prinsenhof Delft, the perfect hors d’oeuvre for a Rijksmuseum visit with an exhibition that throws light on the place from which this phenomenal artist sprang. And what a tumultuous world of wonders and catastrophes it was. On the staircase inside the museum are two bullet holes. In 1584 this was the spot where an assassin sent by Spain murdered the Dutch head of state, William the Silent (the first ever political assassination of a nation’s leader by a firearm).

Elsewhere is a painting that shows the devastating gunpowder explosion that destroyed an entire Delft suburb in 1654, killing over 100 people, including one young artist in his studio, a man who might have been able to explain where his contemporary, Johannes Vermeer, got that sublime painting style. When rescuers came stumbling through the ruins, they found Carel Fabritius dying in his studio ruins. Any paperwork that might have proved what many suspect, that he was Vermeer’s tutor, was destroyed. Next to him was an oil painting of a small bird chained to a tin box. Somebody had the good sense to take it.

In those excitable times, the richly allegorical, colourfully adorned world of catholic taste was being challenged by a new austere aesthetic, one the new Dutch Republic could call its own. Walking into the magnificent Nieuwe Kerk (New Church) in Delft where Vermeer was baptised and William the Silent buried, you see starkness all around. In the iconoclasm of the 16th century, the ornate decorations were stripped back to suit new religious views and a few artists embraced a cool plain aesthetic peopled by ordinary folk. At the inn called The Flying Fox (now a private house), down on Voldersgracht, Vermeer was, presumably, imbibing all of this change. His father was an art dealer and innkeeper – we do know that. We also know where Johannes painted: his studio would have looked out into one of those narrow streets with a canal. These days he would be looking at signs for a fish and chip restaurant. At sunset he might have knocked off and gone for borrel nearby, ordering some bitterballen, deep-fried meat balls, or kaasstengels, deep-fried cheese.

I could stroll these streets for days and never get tired, but I want to head across to The Hague and find more traces of Vermeer. (There are trains and trams for what is about a six-mile journey.) The Mauritshuis museum is in the heart of the Dutch government, right next to their prime minister’s office, a few minutes stroll from the grave of another great philosopher, Spinoza. The collection was opened in 1822, soon after adding a painting titled View of Delft, largely for its local interest – cityscapes from the 17th century were rare. Twenty years later, a young Frenchman called Théophile Thoré came to visit. He was a radical, exiled for his political opinions, a self-confessed bohemian whose ambition was to change public taste.

Thoré had come to the Mauritshuis for the Rembrandts, but he liked that View of Delft painting and began an obsessive search to find its forgotten creator, scouring the area and buying up works for what he called his collection of bric-a-brac, rarely spending more than a few guilders. I wonder what it was that grabbed him so powerfully. Was there something bohemian in the mix of the austere and luxurious: bare walls and oriental carpets? Or was it the engaging look that Vermeer’s subjects often have, pulling the viewer into intensely personal moments.

In the attic of a fellow collector, Thoré found a beautiful little portrait of a bird chained to a tin box against a luminous wall painted with delicate brushstrokes that reminded him of “Van der Meer”. It became his favourite possession. These days that obscure little painting is a treasure of the Mauritshuis. The Goldfinch by Carel Fabritius was later made famous by Donna Tartt’s novel and the film starring Nicole Kidman, but it was Thoré who rescued it and drove it from jumble-sale image to treasured masterpiece. “When we restored it,” says Mauritshuis senior curator, Quentin Buvelot, “we found minuscule indentations, probably from the debris of the 1654 gunpowder explosion.”

With Vermeer, Thoré excelled himself, buying several works and extolling their virtues. Others soon grasped Vermeer’s genius too. Girl with a Pearl Earring was bought for two guilders in 1881 by Hague collector Andries des Tombe, then bequeathed to the Mauritshuis. (It will return from the Rijksmuseum on 1 April and in the meantime there is an entertaining show of alternative versions of the painting sent in by the public.) Like The Goldfinch, its reputation was polished by a novel, then a film.

Back in Delft, I walk along the Wateringsevest canal that divides the old from new, a hi-tech modern city full of glass and chrome. What Vermeer would make of that aesthetic? It seems incredible that his “introvert, tranquil world”, as Rijksmuseum director Taco Dibbits calls it, is still such a powerful force.

Turning down a narrow alleyway, I pass a graffiti that nods to the Vermeer heritage, then emerge in the network of old canals and streets. At night when the lights come on inside the buildings, your eye is drawn to the little vignettes of life: the florist trimming stems, the baker covering leftover loaves, and the animated faces of friends enjoying borrel together – fleeting images that Vermeer would recognise. I check the map: there’s brewery called – you guessed it – Brouwhuis, and I’m told they do excellent beer and kaasstengels, so that is where I’m going.

Kevin Rushby was a guest of the Netherlands Board of Tourism. The Vermeer at the Rijksmuseum runs until 4 June. Extra tickets for late openings should be available soon. Vermeer’s Delft at the Museum Prinsenhof Delft also runs until 4 June. Tickets available. The installation My Girl with a Pearl runs at the Mauritshuis until 4 June. Vermeer’s Girl with a Pearl Earring returns there on 1 April