Just like starting over: when Britain (briefly) fell in love with New Towns

"Modern girls and modern boys: it's tremendous!" So goes the sunny reflection of the eponymous hero in Bill Forsyth's 1981 film Gregory's Girl, as he surveys the playing fields, comprehensive schools and spaghetti plate of dual carriageways in Cumbernauld, a mid-20th-century Scottish 'New Town'. Gregory and his friends playfully mock the town, but their youthful affection for Cumbernauld shines through; it neatly encapsulates the optimism these places were all about: doing things differently, doing them better.

New Towns were sometimes sublime and surely strange; but more of a success than the popular consensus gave them credit for. These weren't just council estates, but whole functioning places with jobs, shops and services.

Perhaps now we're truly recognising some of that value because, as archetypal New Towns like Milton Keynes and Harlow celebrate milestone birthdays this year (fiftieth and seventieth respectively), the UK government has floated a new generation of New Towns that could once again change the face of Britain.

Most cities we live in haven't been planned at all, they're the product of hundreds or thousands of years of architectural accretions. Most cities are ultimately exercises in speculative pissing in the wind: developers develop, architects design, but none of it is woven together and thought through from scratch. It's planning on the most piecemeal scale.

But not all. Mohenjo-daro might have been the first planned city, appearing 4,500 years ago in what is now Pakistan. Alexandria was planned. And Renaissance Italy boasted the star-shaped Palmanova. But these were the enlightened exceptions, and in Britain it was mainly the kind of hotchpotch best illustrated by the Shambles in York: quaint, but a bloody mess.

It was towards the end of the 19th century that modern and urban change came to Britain. Tenements and slums were the rule in most large towns of the era. A number of enlightened capitalists planned their own towns, toy communities almost; but such innovative plans were rare. Schoolchildren today are taught about Titus Salt's dry settlement of Saltaire and the model village that started it all, Bournville. But we make a show of these places and the characters who bequeathed them to make us feel better as a country – to play up our successes rather than our failures.

Today Bournville feels quaint, especially if you compare it to the later, more radical New Town of Redditch, a mere six stops down the Midlands' Cross-City Line. Bournville was the brainchild of the Cadburys, and its bucolic buildings and tree-lined streets led towards the garden cities movement at the start of the 20th century. With Bournville and the garden cities we see a key touchstone that would also be echoed in the later New Towns project: the idea that the city was broken and escape was the answer. That sentiment endured beyond the end of the "dark satanic mills" era. Arguably it's only really been in the last 20 years that the city, the British city at least – other European nations typically had a milder view towards their cities – has come to be seen as the answer rather the question.

However the garden cities like Letchworth were more of a dream than a reality, an exercise in placemaking reverie; and like Bournville as much of a fantasy as Middle Earth. Tolkein saw Bournville as a child. These towns were visions of an idealised Britain, a pre-industrial, anti-industrial one. This line of thinking continues in the oddball planned suburb of Poundbury, which appears as one of those miniature model villages (but one with a Waitrose, of course). Strangeness wasn't far from all these places. Jonathan Meades picked up on the multitude of cults that infected the garden cities: teetotallers, vegetarians, religious dissenters, political radicals.

It was only after the second world war ended that a gutsy modernism bloomed. The New Towns of this era sat alongside the radical municipal socialism exemplified by existing cities like Sheffield, London and Newcastle, which built swathes of housing and other civic amenities in the electric post-war period of progress. Around the globe, planners and architects were getting to make their mark, from Chorweiler to Chandigarh to Brasilia, new cities rose. Top of the list in Britain was providing working people with high quality, affordable housing in healthy surroundings. The 1946 New Towns Act was a way to make things happen by creating an all-powerful development corporation in each of the towns, allowing building to get going quickly.

"Amazing people were involved in Harlow, Cumbernauld and Peterlee," points out Catherine Croft of the Twentieth Century Society. Architects like John Madin at Telford, Frederick Gibberd at Harlow, Geoffrey Jellicoe at Hemel Hempstead deploying a complete vision. This was about top-down, total design; men smoking pipes in committee rooms and deciding what was best for women and children. There's no better depiction of this than in Catherine O'Flynn's bravura novel The News Where You Are, where the harassed architect (that she's very careful to point out isn't Madin) pores over his beautiful scale model of a Midlands New Town populated with miniature plastic people lacking faces.

"I love the high-profile public art," says Croft, "especially the murals, and would like to see more of that today. As well as the main set pieces, some of the low-key housing developments deserve to be more cherished."

Surrounded by the highest quality council housing and landscaping, Victor Pasmore's Apollo Pavilion in Peterlee, for instance, has every right to be as high up on a visitor's itinerary as Durham Cathedral.

In the public consciousness, everything from the edge estate to the expanded town to the full New Town has become conflated: we see council houses surrounded by trees and are not always sure if it's an estate or a New Town. Frequently these associations are negative.

The sprawling exurban council estates, like Chelmsley Wood on Birmingham's outskirts, faced challenges with a lack of infrastructure, jobs, amenities and transport. There was also psychological isolation from the geographic and social communities that previously bound together urban working-class life. In her book Estates, Lynsey Hanley paints pictures of estates like this as if they were flawed works of cubism.

The expanded towns like King's Lynn, Haverhill and Thetford were never fully comfortable with their double lives as market towns and an overspill zone for Cockneys. But the fully planned New Towns were attempts to make a whole place with all the facilities, factories, shopping and bus links so essential to any functioning city – even if it did sometimes take too long for these to arrive. Milton Keynes didn't get a hospital for 13 years.

In John Grindrod's groundbreaking (pardon the pun) book Concretopia, he says New Towns "sit alongside the creation of the welfare state, the NHS and the post-war revolution in education as monuments to a nation's desire to move on, not just from the destruction of the war years, but from the inequalities and squalor inherited from the Industrial Revolution."

Mike Althorpe of Karakusevic Carson Architects, agrees: "I think the New Towns project in the UK was much more successful than people give it credit for… It's one of the greatest modern movements of people and the biggest built project in our history; and its legacy is one of architectural bravery, optimism and a sincere belief in the idea and the qualities of 'place'. These were not mere housing estates, they were intentional communities with great thought given over to what makes a town."

It could be a challenge. Aside from the sheer effort of planning a whole new town there was occasional dissent from those who feared the concreting over of the countryside. And some councils – notably Glasgow – wanted to keep their population (in this case a Labour-voting population) within city limits. Occasionally residents and businesses needed a little gentle convincing to relocate: witness the bonkers space pop 7" single, Energy in Northampton, which Northampton Development Agency commissioned to sell the town; and the proto-Gregory's Girl social realism of Living at Thamesmead. Milton Keynes had the charming red balloon TV ad and, more bizarrely, Cliff Richard rollerskating through the shopping centre.

Yet what's remarkable is that all this got done, all this got built, and often very quickly. The timescales compare with the ridiculously quick builds we see in China and the Arabian Gulf today. Opposition was won over and people did move in – and they often liked New Towns, and the modernist architecture that underpinned them. Mike Althorpe grew up surrounded by Scots in Corby who came south for steel jobs. "The structure that impacted me most was the 1972 town centre and bus station," he says now. "As a kid I loved running up and down the cantilevered stairs onto balconies to wind my mum up! It had the town's only (broken) escalator, which took you deep into a dark underworld where the smell of diesel bus fumes and chip fat was intoxicating; and a big National Express sign announced 'Book here for Scotland'. It had a fantastically urban quality."

JG Ballard said he wrote about the future because he believed it would be better than the past. This is the very essence of town planning: that creating something new, something that works better than what went before, can mould superior worlds. But in an infamous section of Robert Hughes's masterful BBC art series The Shock of the New, this fierce Aussie decried Brasilia as "a ceremonial slum" and Paris's Peripherique New Towns as dead ends. He urged urban planners to shut up because we all need a bit of (his words) "shit" around us in the cities artists and the rest of us live in: like Paris, New York and London.

Each UK New Town has its own character. Cumbernauld’s infamous town centre megastructure has been called Britain's ugliest building, but it was intended as a radical and revolutionary attempt to get all of the town's services – library, shops, bookies, hotel, car park, bus station and penthouse flats – into one space station-like building. "I tried to take some American friends to Cumbernauld [town centre] and they refused to get out of the car!" says Catherine Croft. "That's unusually urban and intimidating; in general there is a calm softness to our New Town design."

Harlow, with its gardens and Moore sculptures, embodies this softness in its 70th year. But Ballard called the low rise suburbs with house, garden and car in the drive – so typical of New Towns – "the death of the soul". And he lived in a suburb.

It could all have been more dramatic: Geoffrey Jellicoe's Motopia in Slough envisaged a city with roads on the roof, while unbuilt proposals for Hook in Hampshire look like a jet-propelled version of quasi-New Town Thamesmead. Hubert de Cronin Hastings, longtime honcho of the Architectural Review, dreamt up Civilia in the 1960s. He wanted to stack Moshe Safdie-esque residential superblocks, Tuscan piazzas and boating lakes (all New Town plans had their marina) on top of an old quarry outside Nuneaton and stick a million people in a kind of retro-futurist Arezzo on the Anker.

Civilia didn't make it and what did at that exact time was completely antagonistic to it: low-rise, low density Milton Keynes. This "Los Angeles in Buckinghamshire", according to John Grindrod, is filled with Mies van der Rohe-apeing minimalism and houses by a welter of starchitects like Norman Foster and Ralph Erskine. It continues to look forward, with trials of driverless cars on its ample roads.

Katy Lock, the Town and Country Planning Association's New Towns expert, talks eloquently about her own upbringing in Milton Keynes. Crucially, she mentions "people being consciously part of the story. People had chosen to move [to New Towns]. Like with Stevenage earlier, where people had bought into the story of an inside bathroom and a new job."

Christopher Smith's forthcoming film, New Town Utopia, focuses on Basildon. "New Towns were a grand ambition that could still work," he says. "But for the first wave of new towns, the execution was flawed. These were places created for the working classes, but designed by the middle and upper classes. They also faced a number of negative external forces, including globalisation, Thatcher's Right to Buy policy, and a lack of care and attention."

The current UK government recently put its weight behind more New Towns in places like Essex and Cheshire. "We've been campaigning for a new generation of garden cities," says Lock. "It's one of the solutions of the housing crisis – but the renewal of existing cities is too. We need to learn the lessons from garden cities and post-war New Towns."

The question will be: can we fully commit to building a concrete future? The 20th-century New Towns embraced innovation in housing, public realm and transport design. The New Towns of today can do that too – look at Vauban, the ecologically-rigorous New Town on the outskirts of Freiburg in Germany with all kinds of green innovations. The danger with Britain's potential new New Towns is that they simply become overblown dormitory suburbs for the middle managers of Cambridge, Manchester and London: commuter towns with cut-price architecture and planning, rather than truly viable and thriving towns. However, with architects and planners at the tiller instead of just property developers, and with technical innovations such as communications connectivity, futuristic transportation and that all-elusive sense of 'place' front and centre, the new New Towns could offer the 21st century something truly unique.

And as the 20th-century New Towns around the world hit middle age, they've often settled into being quietly successful: just look at Australia's spirited capital, Canberra, or the way Milton Keynes has matured to nurture a sense of pride in its inhabitants. Architecture is our gift to future generations; building whole cities supersizes this impulse. It's an urge that will, in various forms, forever linger.